True2456/gemma-4-31b-it-8.6bit-frontierdistill

Gemma-4-31B — Agentic Coding (fused, 8-bit, vision-preserving)

Built for the MLX ecosystem on Apple Silicon. The goal: bring frontier-model distillation data -- real agentic coding traces from top-tier teacher models, not synthetic filler -- to native Apple Silicon deployment, without trimming sample count or curation quality to make the fit easier. MLX end to end: quantization, QLoRA training, and the fused deployment artifact all run natively on-device.

QLoRA adapter fused into google/gemma-4-31B-it (8-bit), toward autonomous coding-agent behavior: reproduce-before-fixing methodology, tool-call formatting, and response tone calibration, distilled from real agentic coding traces. This is the merged, ready-to-run model -- weights = base + LoRA delta, no separate adapter needed at inference time.

Vision-preserving build: an earlier fuse of this same adapter used mlx_lm.convert/mlx_lm.fuse (text-only tooling) against the source VLM checkpoint, which silently dropped the vision_tower/embed_vision weights instead of erroring. This build re-quantized via mlx_vlm.convert and fused via mlx_vlm.load(..., adapter_path=...) + per-layer .fuse() instead, so the 358 vision tensors ride through intact. The adapter itself only ever touched language_model.* linear layers, so no retraining was needed -- only re-quantization/re-fusion with the correct tooling.

Base model

• Base: google/gemma-4-31B-it (stock instruction-tuned release, not the QAT variant -- QAT calibration targets 4-bit robustness specifically, which buys nothing when the training/deployment target is 8-bit, so stock bf16 was quantized directly instead).
• Quantization: 8-bit, affine, group size 64, via mlx_vlm.convert -q --q-bits 8 (not mlx_lm.convert -- see vision note above). 8.6 bits/weight actual (embeddings/output layer kept slightly higher precision).

Training data

Combined trace dataset from three sources (all vetted for license/provenance before use):

Source	Role
greghavens/fable-5-coding-and-debugging-traces	primary agentic coding traces
greghavens/gpt-5.6-sol-coding-and-debugging-traces	additional coding/debugging traces
greghavens/kimi-k3-coding-and-debugging-traces	additional coding/debugging traces

Built via scripts/build_lora_data.py: tool-call arguments pre-parsed from JSON string to dict (the base HF chat template has no fromjson filter, so string-arguments rows fail to render otherwise), split by source_trajectory_id (not by row -- rows are cumulative-context slices of a trajectory, so a row-level split would leak near-duplicate context across train/valid), and length-filtered to <=6144 tokens against the actual gemma-4-31b-it-bf16 tokenizer (rows over the cap are dropped, not truncated -- avoids the head-truncation bug where a naive cutoff clips the target assistant response off the end of long sequences).

Split	Rows
Train	14,516
Valid	1,689
Test	801

No Reasoning: <level> system-prompt tag (that convention is Step-3.7-specific syntax, meaningless to Gemma-4).

Method

QLoRA via mlx_lm.lora (text-only training, since the adapter only targets language_model layers), hyperparameters matched to the published greghavens/fabletron-nemotron-3-super-120b precedent (rank 8 / alpha 16, grad-accum 16, lr 2e-5 cosine+warmup) rather than guessed from scratch:

Parameter	Value
LoRA rank	8
LoRA alpha (scale = alpha/rank)	16 (2.0)
Dropout	0.0
Target layers	all 60 (attention q/k/v/o + MLP gate/up/down)
Batch size	1
Gradient accumulation	16 (effective batch 16)
Learning rate	2e-5, cosine schedule
Max sequence length	6144
Loss	response-only (mask_prompt) -- system prompt, tool
definitions, and prior turns are fed as context but excluded from the loss
Gradient checkpointing	on
Total iterations	1000 (~1.1 epochs over the 14,516-row train set)

Training run

Training spanned four process launches across two machines (M3 Max 64GB, then M5 Max 128GB) due to two OOM-driven hard crashes on the 64GB machine, plus one Metal resource-limit crash near the end on the 128GB machine, before completing. Checkpointed every 15 iterations throughout specifically to survive exactly those failure modes.

Validation loss (25-batch val set, evaluated every 50 iterations):

Absolute iter	Val loss
1	2.249
200	1.206
585	0.618
685	0.611
955	0.578 (best -- this checkpoint)
1000 (final, not used)	0.729

This is the iter ~945-955 checkpoint, not the final iter-1000 weights -- val loss drifted back up over the last ~250 iterations while train loss kept falling, the standard mild-overfitting signature.

Evaluation

No formal benchmark run yet (e.g. HumanEval, BFCL) -- current validation is loss-based plus qualitative generation smoke tests at fixed intervals during training, which showed increasingly refined agentic response structure (reasoning about response tone/framing, concrete tool-invocation proposals, not just mechanical steps) without degeneration.

Known limitations

• Rank-8 adapter, single training campaign -- capacity for holding multiple distinct skills simultaneously (e.g. if later stacked with a second LoRA for a different task) is limited.
• No quantitative benchmark comparison against the base model or other public Gemma-4-31B fine-tunes yet.
• Trained at ~1.1 epochs; scale (14.5K rows) is roughly on par with the largest comparable public precedent found (TendieLabs/Capybara-31B, 10-20K rows), but that precedent has no published quantitative results either -- there is no public proof point that this scale outperforms a smaller, more heavily curated set for this model.
• Vision/image capability is present (weights intact) but was never exercised during training or evaluation -- the training data was text-only, so the fine-tune's effect on multimodal behavior specifically is unverified.

License

Derivative of Gemma 4, distributed under and subject to the Gemma Terms of Use and Gemma Prohibited Use Policy. Training data license terms per source dataset apply to the data itself.