Raon-1-mini-preview-NVFP4-GPTQ

An NVFP4 quantization of KRAFTON/Raon-1-mini-preview, made by Sapid Labs with GPTQ instead of round-to-nearest. KRAFTON did not make or review this checkpoint.

It quantizes the same layers as KRAFTON's own Raon-1-mini-preview-nvfp4, plus lm_head, and loads with the same vLLM fork. On one DGX Spark it is closer to the original model, decodes 12% faster for a single request, and is 0.4 GB smaller.

Compared with KRAFTON's NVFP4

Quality is measured against the unquantized model (the MXFP8 release with its experts expanded to BF16), on 64 texts of 512 tokens: 16 each of English Wikipedia, Korean Wikipedia, Python code and chat-formatted maths. Each quantized checkpoint's weights are dequantized into KRAFTON's Transformers model, KRAFTON's MXFP8 lm_head included, so the numbers compare weights only.

	KRAFTON NVFP4	This checkpoint
KL divergence from the original, mean per token	0.0577	0.0513
Same top token as the original	89.11%	89.11%
Perplexity	7.85	7.78
KL on English / Korean / code / chat	0.125 / 0.039 / 0.023 / 0.023	0.109 / 0.035 / 0.022 / 0.019
Size	17.6 GB	17.2 GB

Speed and task scores come from one DGX Spark (GB10, 128 GB), KRAFTON's vLLM fork, the same settings for both, no speculative decoding. The throughput test runs closed-loop agents that share a system prompt and grow their context every turn, with 250-token replies.

	KRAFTON NVFP4	This checkpoint
Decode, 1 request	71.5 tok/s	80.0 tok/s
Total throughput at 16 / 32 / 64 requests	116 / 107 / 109 tok/s	121 / 120 / 115 tok/s
First token at 64 requests, median / p95	3.4 s / 23.2 s	5.1 s / 22.0 s
GSM8K, first 200, thinking off / on	95.0% / 96.5%	97.0% / 97.5%
Tool calls (20 cases)	20/20	20/20
JSON extraction (15 cases)	13/15	13/15
Memory for weights / KV cache at 0.80	16.41 GiB / 6.71M tokens	16.12 GiB / 6.90M tokens

What is different

KRAFTON's NVFP4 already uses error-compensating rounding (about a fifth of its 4-bit codes differ from round-to-nearest at its own scales). This checkpoint changes four things:

1. GPTQ on every quantized layer, routed experts included. Each layer's Hessian comes from its real inputs on 128 calibration samples of 512 tokens (English and Korean Wikipedia, Python code, chat-formatted maths), none of them in the evaluation set above. Each routed expert gets its own Hessian from the tokens routed to it. 33 of the 9,489 quantized layers (projections of experts that no calibration token reached) keep the scale-search rounding of point 2.
2. The FP8 scale of each 16-weight block is searched, not fixed at block max / 6: 21 candidates from 0.80 to 1.00 times that value, chosen on the error-updated weights during GPTQ.
3. Fused layers share one per-tensor scale: gate and up projections, the linear-attention input projections (q, k, v, g, f_a, b) and MLA's q_a and kv_a projections, because vLLM runs each group as one matrix.
4. lm_head is NVFP4 (KRAFTON's is MXFP8). It is 259,840 x 2,560 and read for every generated token.

Layers	Format
Routed and shared experts, dense MLP, attention projections, lm_head	NVFP4 W4A16
kv_b_proj, router gates, convolutions, hyper-connections, norms, embeddings	BF16

Every NVFP4 layer is weight-only (W4A16). The checkpoint uses ModelOpt's MIXED_PRECISION format with quant_method: modelopt_fp4, as KRAFTON's does, and carries the same placeholder input_scale of 1.0 that their fork expects on weight-only layers.

KRAFTON's routed experts are W4A4, with calibrated activation scales. On a DGX Spark both checkpoints run on the same Marlin weight-only kernels, so the comparison above is like for like. On B200/B300, where FP4 activations have hardware support, KRAFTON's experts may prefill faster; that was not measured.

Running it

It needs KRAFTON's vLLM fork, as their own checkpoints do:

uv venv --python 3.12 && source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install \
    "vllm @ git+https://github.com/krafton-ai/vllm.git@raon-1-mini-preview"

On B300, use KRAFTON's serve command from their model card with this repository's name. On a DGX Spark (GB10, sm_121) the fork's default kernels are built for sm_100 only, and these settings replace them:

uv pip install ninja                     # the fork compiles a kernel on first start
export PATH=/usr/local/cuda/bin:$PATH CUDA_HOME=/usr/local/cuda TORCH_CUDA_ARCH_LIST=12.1
export RAON_KDA_PREFILL_VSPLIT=0 RAON_KDA_PREFILL_FRONT_FUSED=0 RAON_KDA_SGL_FUSED=0
vllm serve sapidlabs/Raon-1-mini-preview-NVFP4-GPTQ \
  --max-model-len 32768 --max-num-seqs 64 --max-num-batched-tokens 8192 \
  --enable-chunked-prefill --enable-prefix-caching --gpu-memory-utilization 0.80 \
  --reasoning-parser raon_1 --enable-auto-tool-choice --tool-call-parser raon_1 \
  --attention-config '{"backend_per_kind": {"mla_attention": "TRITON_MLA"}, "mla_prefill_backend": "FLASH_ATTN"}'

Without the three RAON_ variables the server starts, then the first request fails with "no kernel image is available for execution on the device". scripts/serve_raon.sh is the command used for the measurements above.

Files

scripts/convert_nvfp4.py (--preset raon-full --search mse --quant-head) writes the starting checkpoint; scripts/gptq.py re-rounds it with GPTQ; scripts/kl.py is the KL measurement.

License

The base model is released by KRAFTON under the Apache License 2.0, and this checkpoint is distributed under the same license; LICENSE holds its text. Sapid Labs changed the weight files (quantized as described above), model.safetensors.index.json, config.json (a quantization_config was added) and this README.md, and added scripts/. KRAFTON's original model card is kept unchanged as ORIGINAL_MODEL_CARD.md.