mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A 4-bit mixed-precision MLX quant of mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16 produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. +1.6 pp on GSM8K over stock uniform 4-bit at the same per-weight precision target.

Nemotron 3 Nano is a hybrid Mamba2 + attention model. Of the 42 backbone blocks, 4 are full attention and the rest are Mamba2 SSM or MLP. OptiQ measures each linear's KL-divergence sensitivity against a bf16 reference forward pass and assigns 4-bit or 8-bit per-layer based on that signal. Sensitive layers go to 8-bit; robust ones stay at 4-bit.

Quantization details

Property	Value
Predominant precision	4-bit
Layers at 8-bit (sensitive)	46
Layers at 4-bit (robust)	47
Total quantized layers	93
Achieved BPW	5.43
Group size	64
Calibration mix	six-domain mix (40 samples)
Reference for sensitivity	bf16 (gold standard)
Bundled KV-cache recipe	kv_config.json, 3 attn layers @ 4-bit, 1 @ 8-bit (5.0 avg KV bits)

We follow the same naming convention llama.cpp uses for Q4_K_M-style mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average.

Usage

Load it with mlx-lm (the custom NemotronH modeling files ship in the repo and are picked up automatically):

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain why hybrid Mamba+attention models scale to long contexts.",
    max_tokens=300,
)

For mixed-precision KV-cache serving and sensitivity-aware LoRA fine-tuning, install mlx-optiq:

pip install mlx-optiq

# Serve with the bundled KV-cache recipe
optiq serve --model mlx-community/NVIDIA-Nemotron-3-Nano-4B-OptiQ-4bit \
            --kv-config kv_config.json

Benchmarks

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit:

Metric	OptiQ	Uniform 4-bit	Δ
MMLU (5-shot, 1000 samples)	64.0%	63.3%	+0.7
GSM8K (1000 samples, 3-shot CoT)	81.5%	79.9%	+1.6
IFEval (full set, strict)	56.2%	56.0%	+0.2
BFCL-V3 simple (200 calls)	94.0%	94.0%	+0.0
HumanEval (164 problems, pass@1)	77.4%	80.5%	-3.1
HashHop (long-context retrieval)	27.0%	25.0%	+2.0
Capability Score (mean of 6)	66.68	66.44	+0.24
KL vs bf16 reference (mean / p95)	0.0353 / 0.1136	,	,
On-disk size	2.94 GB	2.13 GB	+0.81

Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. On this Mamba-attention hybrid the mixed allocation runs a little richer than on a pure transformer, Mamba2 SSM blocks carry only two linears each, so the sensitivity probe upgrades more of them to 8-bit, which is why the disk delta is larger than on the Qwen / Gemma families. See the eval-framework writeup for the full methodology.

Links

• Project website: mlx-optiq.com
• PyPI: pypi.org/project/mlx-optiq
• Calibration mix: mlx-optiq.com/blog/calibration-mix
• Eval framework: mlx-optiq.com/blog/eval-framework
• Base model: mlx-community/NVIDIA-Nemotron-3-Nano-4B-BF16

Base model

This is a quantized derivative of NVIDIA Nemotron 3 Nano 4B. See the NVIDIA Nemotron Open Model License for terms, the quant is distributed under the same license as the base.

Quantize your own

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

pip install mlx-optiq
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8
optiq lab   # full local workbench: chat, compare, quantize, fine-tune