Upload folder using huggingface_hub

README.md

@@ -0,0 +1,187 @@
+---
+base_model: Qwen/Qwen3.5-4B
+license: apache-2.0
+library_name: transformers
+tags:
+  - qwen3.5
+  - merge
+  - omnimerge
+  - task-arithmetic
+  - code
+---
+
+# Qwen3.5-4B-MicroCoder
+
+A 4B-parameter code-leaning merge of Qwen3.5-4B that beats every individual
+source on LCB-medium-55, while preserving full GSM8K parity with the strongest
+reasoning fine-tune in the pool.
+
+This card documents `Qwen3.5-4B-MicroCoder` (internally `v2i-jv-base-task-arith`),
+the chosen frontier point of a 19-variant ablation that swept merge methods,
+density, importance signals, AIME-protection masks, and skip-layer surgery.
+
+Built with [**OmniMergeKit**](https://github.com/mann1x/omnimergekit) — the
+open-source merge engine developed for this work.
+
+## Headline numbers (Q6_K, greedy)
+
+| Benchmark | base Qwen3.5-4B | jackrong-v2 (best source) | **MicroCoder** | Δ vs source |
+|---|---:|---:|---:|---:|
+| HumanEval (164q) | 60.37 | 60.37 | **57.32** | −3.05 |
+| MBPP (500q) | 46.00 | 45.00 | **52.00** | **+7.00** |
+| LiveCodeBench-30 (medium, post-2024-10-01) | 3.33 | 23.33 | **26.67** | **+3.34** |
+| LiveCodeBench-55 (full medium pool) | — | 25.45 | **27.27** | **+1.82** |
+| HumanEvalPlus (164q) | — | 54.88 | 50.00 | −4.88 |
+| GSM8K (100q) | — | 83.00 | **83.00** | 0.00 |
+| MMLU-Pro (200q) | — | 56.81 | 52.46 | −4.35 |
+| AIME (30q) | — | 26.67 | 3.33 | −23.34 |
+
+**Net:** +7pp MBPP, +3.3pp LCB-30, +1.8pp LCB-55, GSM8K parity. Trade-offs are
+HumanEval (−3pp), MMLU-Pro (−4.4pp), and the AIME math-reasoning floor
+(see "Why no AIME?" below).
+
+## Recipe
+
+```bash
+python omnimergekit.py \
+    --base       Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2 \
+    --task-base  Qwen/Qwen3.5-4B \
+    --source     coder_eval/continuum-code-forged \
+    --source     coder_eval/jackrong-python \
+    --method omnimerge_v2 --v2-features fisher,darex \
+    --weights 0.55,0.45 --density 0.53 --darex-q 0.85 \
+    --fisher continuum-forged.safetensors,jackrong-python.safetensors \
+    --pr682-turbo \
+    --seed 42 --device cuda
+```
+
+This is a **task-arithmetic** merge:
+
+```
+MicroCoder = jackrong-v2 + 0.55·DARE(continuum-code-forged − base) + 0.45·DARE(jackrong-python − base)
+```
+
+- **`jackrong-v2` is the merge base** — its full output style and reasoning
+  policy survive intact at zero deltas. The two coding teachers contribute
+  only their *delta from the official Qwen3.5-4B base*, not their absolute
+  representations. This isolates "what the coder fine-tunes added on top of
+  the public base" and grafts that onto the reasoning-distilled model.
+- **DAREx-q 0.85** drops the bottom 85% of cf/jp deltas by magnitude
+  (per-tensor quantile) before random pruning, then rescales by 1/density.
+  This kills low-magnitude noise while preserving the high-amplitude
+  code-skill structure.
+- **Fisher importance** from forward-pass gradient maps over the coder
+  fine-tunes' own training-style data weights the EMR election so dominant
+  per-element directions win when the two coding teachers disagree.
+- **PR682-turbo** protects critical layers (norms, embeddings, lm_head,
+  biases) at density 1.0 and falls back gracefully on shape mismatch.
+
+## Sources
+
+| Model | Role | Weight |
+|---|---|---:|
+| [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B) | task base (delta reference) | — |
+| [`Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2`](https://huggingface.co/Jackrong/Qwen3.5-4B-Claude-4.6-Opus-Reasoning-Distilled-v2) | merge base | 1.0 (passthrough at δ=0) |
+| `continuum-code-forged` | code teacher (delta) | 0.55 |
+| `jackrong-python` | code teacher (delta) | 0.45 |
+
+## Evaluation methodology
+
+All evaluations: [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness),
+llama.cpp `llama-server` with the published Q6_K quantization,
+`/v1/completions` raw endpoint, greedy decoding (`temperature=0.0,
+top_p=1.0`), `max_gen_toks=2048` for HE/MBPP, `max_gen_toks=8192` for LCB,
+`--parallel 2 --cache-type-k q8_0 --cache-type-v q8_0`.
+
+LiveCodeBench: medium-difficulty functional problems with
+`min_date=2024-10-01` (post-Qwen3.5 training cutoff to avoid contamination).
+LCB-30 = first 30 problems of that pool, LCB-55 = full pool of 55.
+
+## Experiment trail (why this recipe?)
+
+19 variants were ablated over a multi-week sweep. Summary table for the
+informative subset:
+
+| variant | merge form | AIME | HE | MBPP | LCB-30 | verdict |
+|---|---|---:|---:|---:|---:|---|
+| base | Qwen3.5-4B | 0.00 | 60.4 | 46.0 | 3.33 | floor |
+| jackrong-v2 | source | **26.67** | 60.4 | 45.0 | 23.3 | strong reasoning, weak LCB |
+| v2g | 3-src DARE-TIES, fisher+darex | 0.00 | 56.1 | **54.0** | 26.7 | code champion (no AIME) |
+| **v2i = MicroCoder** | task-arith on jv-base | 3.33 | **57.3** | 52.0 | **26.7** | **balanced — picked** |
+| v2j | v2i + skip mlp.gate_proj 18-25, darex 0.92 | 10.00 | — | — | — | first AIME signal |
+| v2k | v2j + wider skip 14-27 | 0.00 | — | — | — | over-blocked, collapsed |
+| v2l | v2j + full MLP skip 18-25 | 3.33 | — | — | — | up/down_proj carry code skill |
+| v2m | v2j + density 0.45 | 3.33 | — | — | — | lower density hits jv harder |
+| v2n | v2j + darex 0.95 | **13.33** | 55.5 | 50.8 | 20.0 | reasoning ceiling |
+| v2o | v2n + darex 0.97 | 13.33 | 56.7 | 51.0 | 16.7 | saturated |
+| v2p | v2n + jv-AIME fisher mask α=1.0 | 13.33 | 55.5 | 50.8 | 20.0 | mask redundant |
+| v2q | v2n + jv-AIME mask α=0.5 | 13.33 | 54.9 | 51.2 | 20.0 | mask redundant |
+| v2r | mask α=1.0 alone, no skip | 3.33 | — | — | — | per-element scaling cannot replace layer skip |
+
+### Key findings (apply to future merge work)
+
+1. **Task-arithmetic with the strong source as merge_base wins over symmetric
+   DARE-TIES** when one source is much stronger on the target axis (here:
+   reasoning). v2g and v2i tie on LCB-55 (27.27%) but v2i wins HE/HE+/GSM8K
+   and retains a small AIME signal that pure DARE-TIES kills.
+
+2. **Skip mlp.gate_proj layers 18-25 is the load-bearing AIME-recovery knob**
+   (+6.7pp). This maps from Qwen3.6's think-policy band 27-52/64 → 32-layer
+   Qwen3.5 = 14-26 conservative narrow 18-25. Wider bands (v2k 14-27)
+   collapse; full-MLP skip (v2l) destroys code skill.
+
+3. **DAREx-q 0.92 → 0.95 adds 3.3pp AIME on top of the skip** by killing more
+   low-magnitude cf/jp deltas in the protected reasoning band. **0.95 → 0.97
+   saturates** (v2n=v2o on AIME).
+
+4. **The jv-AIME fisher suppression mask is fully redundant with skip-layers**
+   (v2n=v2p=v2q at AIME 13.33 *and* code metrics within noise). Per-element
+   scaling cannot substitute for layer-level passthrough — jv's reasoning
+   lives in the *coherent per-layer behavior* of mlp.gate_proj 18-25, not in
+   the highest-importance individual cells. Mask alone (v2r) gives nothing.
+
+5. **The 13.33% AIME ceiling is structural, not a tuning problem.** Three
+   different mechanisms (high darex, higher darex, mask) all converge at
+   the same number. Closing the remaining 13.34pp gap to jv source requires
+   SFT distillation, not more merge tuning.
+
+### Why no AIME on the chosen variant?
+
+MicroCoder (v2i) is the **code-leaning frontier point**. The skip-layer
+recipe (v2n) recovers AIME to 13.33% but at a 6.7pp LCB-30 regression.
+v2i preserves the better LCB; the trade is real and structural. A
+reasoning-leaning sibling exists internally (v2n) but is not published —
+LCB regression makes it strictly worse than `jackrong-v2` for math users
+who already have access to the original.
+
+## Files
+
+- Full-precision safetensors weights (BF16). Use [`ManniX-ITA/Qwen3.5-4B-MicroCoder-GGUF`](https://huggingface.co/ManniX-ITA/Qwen3.5-4B-MicroCoder) for the Q6_K quantization.
+
+## Use
+
+```bash
+llama-server -m Qwen3.5-4B-MicroCoder-Q6_K.gguf \
+    --port 8099 -c 32768 -t 12 -ngl 99 \
+    --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0
+```
+
+Greedy `temperature=0.0, top_p=1.0` recommended for code tasks.
+
+## Citation
+
+If you use this model or the OmniMergeKit recipes in your work:
+
+```
+@misc{mannix2026microcoder,
+  title  = {Qwen3.5-4B-MicroCoder: a task-arithmetic merge for code},
+  author = {Mannix, F.},
+  year   = {2026},
+  url    = {https://huggingface.co/ManniX-ITA/Qwen3.5-4B-MicroCoder},
+  note   = {Built with OmniMergeKit, https://github.com/mann1x/omnimergekit}
+}
+```
+
+## License
+
+Apache 2.0, inherited from Qwen3.5-4B and the source fine-tunes.