Gemma-4-26B-A4B agentic

google/gemma-4-26B-A4B-it, trained to stop looping on long agentic tasks. Same architecture, same tokenizer and chat template, same vision tower (bit-identical), same voice: a BF16 recovery SFT on the model's own regenerated answers, filtered so that no trained turn loops.

GGUF (F16, Q8_0, Q6_K, Q4_K_M, imatrix; built with omnimergekit): ManniX-ITA/gemma-4-26B-A4B-agentic-GGUF.

Ollama: mannix/gemma4-26b-agentic (Q4_K_M, Q6_K, Q8_0; plain and vision- tags, plus mtp- tags for xOllama with the Gemma-4 assistant drafter and a thinking budget; the head-to-head sampler is preset).

The problem it fixes

Stock Gemma-4-26B-A4B-it is a strong agent until a task gets long. Then it can fall into a loop — the same paragraph of reasoning, or the same tool call, again and again until the context or the wall clock runs out. This model was trained to keep working instead.

Measured head-to-head against the base model on a long agentic repair task, 20 runs each with identical settings: 12 of 20 runs finished within the 30-minute budget against 3 of 20 for the base model, with reasoning loops cut about 6× less often. Details below.

Head-to-head with the base model

base gemma-4-26B-A4B-it this model
runs finished within the 30-minute limit (20 each) 3 / 20 (15 %) 12 / 20 (60 %)
95 % interval (Wilson) 5 %–36 % 39 %–78 %
passed within 10 / 20 / 30 min 0 % / 15 % / 15 % 25 % / 50 % / 60 %
reasoning-loop cuts per run (median) 8.0 1.0
reasoning-loop cuts per model-hour 18.1 2.9

Pass/fail difference: Fisher exact two-sided p = 0.0079.

What the pass rate means. It is not an absolute success rate: it is the share of runs that finished the repair within a fixed 30-minute budget. The test is built to measure what the fine-tune changes in a definite amount of time — how much of the budget goes to loops, how quickly the model gets from a broken file to a working game. A timeout means "not done in 30 minutes", not "cannot do it". With more time both models would pass more often and the gap would likely narrow, but not by much: in extended manual testing of the base model and of its pruned 98-expert variants (the v7-coder and v9-agentic recipes), a run that does not converge quickly becomes steadily more likely to loop, so the base model either needs one to two hours or more to pass, or ends in massive loops. Anti-repetition sampling hides the loops but costs the model much of its effectiveness and its ability to converge. The fine-tune targets exactly this — less looping, faster convergence — and the data below shows it achieved. With only presence_penalty 0.025 (no other anti-repetition setting) it is virtually loop-free and completes the task in an acceptable time even on runs where it does not converge quickly. The pass curve below shows how the rate builds up minute by minute.

Setup. One task, run 20 times per model: repair a broken HTML game (manic_miner, a syntax error plus logic bugs). The runs are orchestrated and scored by automaticum, a test harness for coding agents whose oracle runs the game itself. The agent is the Cerebriline SDK 4.100.277 (VS Code host profile, LSP diagnostics after every edit and the check_file / ask_lsp tools on, browser off) until the game's own oracle reports that it loads, reaches play and passes 23 logic checks. 30 minutes wall clock per run, one run at a time per model. Both models as BF16 GGUF on the same llama.cpp build, each on its own RTX PRO 6000 Blackwell (96 GB), f16 KV cache, Gemma-4 assistant drafter for speculative decoding, thinking on (effort high), temperature 0.8 / top_p 0.95 / top_k 64 / min_p 0 / presence_penalty 0.025. The two configurations differ only in the model file: harness, task, sampler, server build, chat template and guards are identical (checked from each run's recorded provenance). The harness guards were on for both: a reasoning block that repeats itself is cut and the request retried, a repeated identical tool call draws warnings, then a stop. Arms, server launchers, analysis scripts, the full report and per-run data: omk/models/gemma-4-26b-a4b/sft11/eval/.

Outcome

base this model
passed 3 12
failed: game never starts (syntax / runtime error left in the file) 17 6
failed: game starts, some logic checks still failing 0 2 (17/23, 21/23 checks)
runs ended by the harness for repeating a tool call 1 1
runs that claimed success but failed 0 0
time to the first passing game check, median (passed runs) 16.9 min 11.5 min

Share of all runs passed by minute t:

5 10 15 20 25 30
base 0 % 0 % 0 % 15 % 15 % 15 %
this model 5 % 25 % 40 % 50 % 55 % 60 %

Loops and guards

Medians per run, all 20 + 20 runs; p = Mann–Whitney two-sided.

base this model p
reasoning blocks cut for repeating themselves 8.0 1.0 < 0.001
guard interventions, all kinds 9.5 4.0 0.001
model requests cut or failed 9.0 1.0 < 0.001
reasoning blocks with a phrase repeated ≥ 10× 2.0 1.0 0.03
worst repeat count in any reasoning block 12 10 0.05
repeated-tool-call warnings 0.5 1.5 0.12
runs with no reasoning-loop cut 0 / 20 9 / 20

Per model-hour (sum over all runs ÷ total model time): reasoning-loop cuts 18.1 → 2.9, all guard interventions 24.8 → 17.7, repeated-tool-call warnings 6.1 → 9.9.

Time, length and effort

base this model p
wall time per run (min, 30 = timeout) 30.2 21 0.05
longest single model turn (s) 213 108 0.001
median model turn (s) 22 9.1 0.006
reasoning per run (k characters) 226 230 0.95
longest reasoning block (k characters) 38 30 0.005
generated tokens per run (k) 121 128 0.56
tokens written into tool-call arguments (k) 12.1 22.6 0.17
generated tokens per tool call 3,866 2,100 < 0.001
peak context (k tokens) 84 110 0.26

Same restricted to failed runs (all or nearly all of the 30 minutes used): reasoning 232k vs 416k characters, tool calls 24 vs 69, reasoning-loop cuts 9.0 vs 1.0.

Behaviour

base this model p
tool calls per run 28 50 < 0.001
tool calls per model-minute 0.9 3.3 < 0.001
edits applied 5 10 < 0.001
edits refused by the tool 3 7 0.004
edits not followed by a check 4 6 0.08
game runs (run_game) 2 5 < 0.001
read_files calls 8 13 0.006
LSP: edits checked automatically 8 18 < 0.001
LSP: check_file calls 1 0 0.58
LSP: ask_lsp calls 0 0 0.35
runs that restored the original file and started over 7 / 20 12 / 20

What changes. Total reasoning per run is about the same (226k vs 230k characters); how it is spent is not. The base model spends 3,866 generated tokens per tool call against 2,100, its longest turn runs 213 s against 108 s, and its reasoning repeats itself until the guard cuts it (8.0 cuts per run against 1.0). It makes 0.9 tool calls per model-minute against 3.3 and runs the game 2 times per run against 5, so it sees far less often what is still broken. All 17 of its failed runs ended with the game not even loading; 2 of this model's 8 failures got the game running and passing 17/23, 21/23 logic checks.

Known weaknesses

  • Repeating a refused tool call. When an edit is refused, this model is more likely than the base model to send the same call again: more repeated-tool-call warnings in failed runs (median: base 0.0, this model 6.0; p = 0.010). Runs ended by the harness for repeating a call: base 1, this model 1.
  • Degenerate tool-call bursts. In 2 of 20 runs this model emitted the same refused editor call 310 and 320 times in a single response, until the output-token cap. Both began with an edit whose start_line argument was dropped, so the tool refused it (the base model hit that refusal in 7 runs, this model in 7); the base model never answered it with a burst. Worth capping tool calls per response when running it in an agent loop.
  • Serving. This model's server generated 168 tok/s against 176 for the base model's (4 % slower on the same build and GPU model), which if anything works against it under a wall-clock limit. Speculative-draft acceptance was the same for both (90 % vs 89 %), so it is not the drafter; this model also worked at longer contexts (median peak 110k vs 84k tokens), which slows decoding.
  • Scope. One task, 20 runs per model, one harness. The intervals above are wide; treat the size of the gap as approximate, its direction as solid.

Earlier gates

Before the head-to-head, the same task ran 10 times per checkpoint (automaticum, Cerebriline SDK 4.100.266) at temperature 1.0 (top_p 0.95, top_k 64, min_p 0), 30 minutes per run:

base step 400 step 800 final (this model)
runs finished within 30 min 1 / 10 4 / 10 5 / 10 5 / 10
loops per model-hour (guard cuts + loop blocks) 9.2 9.7 3.1 2.0

On SDK 4.100.276 with the recommended sampler, 7 of 8 finished runs passed. In a VS Code agent session on the same server it fixed the game in 3 of 3 sessions in 6–10 minutes, with no loops seen.

How it was trained

Built with omk-model-optimizer in its BF16 recovery mode (guide): the model's own bf16 weights are trained directly (no quantizer, no LoRA, no fp32 master copy), and the export writes the source checkpoint back with only the trained tensors replaced.

Data — mostly on-policy. Prompts, tool declarations and recorded tool outputs come from public SFT sets (9 NVIDIA Nemotron sets and interstellarninja/hermes_reasoning_tool_use). Every trained assistant turn was regenerated by the untouched Gemma-4-26B-A4B-it itself, turn by turn, in its own chat template with thinking on, and kept only when a validator passed it: no reasoning loop, no foreign tokens, a tool call equivalent to the recorded one (so the recorded tool output still answers it), the right final answer where a reference exists. One source is regenerated by the bigger sibling Gemma-4-31B-it (Kimi-K2.5 agent tasks from lambda/hermes-agent-reasoning-traces). The model learns to finish tasks in its own style rather than imitating another model's.

corpus
sources agentic, opencode, hermes, ifchat v1/v2, kimi31b (Gemma-4-31B), swe, multilingual, science, math, competitive programming
size 16,296 training conversations → 4,217 packed windows of 16,384 tokens (11.6 M scored tokens), one epoch
held out 522 conversations (per-source CE every 200 steps) + 27 agentic windows scored against Gemma-4-31B
published ManniX-ITA/gemma-4-26B-A4B-agentic-sft (conversations + the exact training windows) · ManniX-ITA/hermes-agent-kimi-traces-gemma4-31b

Training.

trained attention, per-layer dense MLP, router and all routed experts: 295 tensors, 24.5 B parameters (embedding, norms, vision tower frozen)
loss next-token cross-entropy on the regenerated assistant tokens only (reasoning, prose, tool calls, stop token); prompts, user turns and tool outputs masked; each packed conversation attends only to itself
schedule 2,108 steps × batch 2 × 16,384 tokens, peak LR 4.74e-6 (10-step warmup, cosine), momentum-free Adafactor with stochastic-rounding bf16 updates
hardware one RTX PRO 6000 Blackwell (96 GB), 74 GiB peak, 25.5 h
result held-out CE on the corpus 0.281 → 0.263; agreement with Gemma-4-31B on held-out agentic windows unchanged (top-1 0.930 → 0.929)

Full recipe, every script as run, corpus provenance (upstream revisions, sha256 of every file) and reproduce commands: omk/models/gemma-4-26b-a4b/ in omk-model-optimizer. The three training modes (QAD to GGUF, ternary recovery, BF16 recovery) are documented in omk/docs/.

Use

Exactly like google/gemma-4-26B-A4B-it (same chat_template.jinja, processor and generation config; thinking via enable_thinking).

Recommended sampler for agentic work (the setting used for the head-to-head): temperature 0.8, top_p 0.95, top_k 64, min_p 0, presence_penalty 0.025, no repetition penalty.

from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "ManniX-ITA/gemma-4-26B-A4B-agentic"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

vLLM: vllm serve ManniX-ITA/gemma-4-26B-A4B-agentic --dtype bfloat16 --enforce-eager --max-num-batched-tokens 8192 --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": true}' — the settings the training corpus was generated with on vLLM 0.20.2 (--enforce-eager avoids a Gemma-4 MoE routing fault under CUDA-graph capture seen on that version). Tool calls were parsed there with a hardened Gemma-4 parser plugin, published in the recipe as sft11/corpus/sft2/gemma4_safe_tool_parser.py (--enable-auto-tool-choice --tool-call-parser gemma4_safe --tool-parser-plugin gemma4_safe_tool_parser.py).

Limitations

  • Evaluated on agentic coding behaviour (the head-to-head above) and held-out CE; no general benchmark suite has been run on this checkpoint yet. 8 of 20 head-to-head runs still did not finish within 30 minutes: reasoning loops are much rarer, not gone, and repeated refused tool calls are a known failure mode (see Known weaknesses).
  • Trained on English-dominant agentic, coding and instruction-following data, with a small multilingual share; vision was not trained (the tower is the base model's, byte for byte).

License

Apache-2.0, as the base model. Training data licenses: see the corpus dataset card.

Downloads last month
53
Safetensors
Model size
26B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ManniX-ITA/gemma-4-26B-A4B-agentic

Finetuned
(249)
this model
Quantizations
3 models

Datasets used to train ManniX-ITA/gemma-4-26B-A4B-agentic

Collection including ManniX-ITA/gemma-4-26B-A4B-agentic