3.0bpw experts-only EXL3 quant: 3.89M-token KV pool on 2× DGX Spark, EvalPlus 468/542
Sharing a 3.0-bit EXL3 quantization of GLM-5.3-Flash with measured numbers, in case it is useful to anyone running it locally:
HF: androidli/GLM-5.3-Flash-Uncensored-EXL3-3.0bpw-mcg-TensorFold
Serving recipe (GitHub): greatyingzi/GLM-5.3-Flash-EXL3-3bit-TensorFold
What it is: only the 288 routed experts per MoE layer are quantized (EXL3 3.0 bpw, mcg codebook); every non-routed tensor (attention, shared experts, dense MLP, embeddings, head, vision tower) is kept at source-native precision. Base: dealignai/GLM-5.3-Flash-UNCENSORED-FP8 (MIT), a derivative of zai-org/GLM-5.3-Flash. Tokenizer / chat template / processor config unchanged.
Measured on 2× DGX Spark (GB10, TP=2 over direct CX7), TensorFold 0.6.0:
| 3.0bpw (this pack) | reference 4bpw | |
|---|---|---|
KV pool (KV_POOL_GIB=26.5) |
3,891,200 tokens (~3.9 concurrent full-1M sessions) | 1.86M tokens |
| EvalPlus HumanEval+ / MBPP+ (542, thinking on) | 468/542 = 86.3% | 469/542 = 86.5% |
| Forward latency (spec-decode verify width 1 / 4 / 8) | −8% / −16% / −17.5% | baseline |
Full results incl. comparison against Mia-AiLab's official 4bpw: results/benchmarks.md.
Verified path is TensorFold (vision ✓, fp8 KV ✓, spill ✓, DFlash2 speculative decoding ✓); other EXL3 runtimes untested by us. Not affiliated with Z.ai or dealignai. Happy to share harness details if anyone wants to reproduce.
Pardon my question if its stupid Im jsut learning all of this and built my lab with 2x MSI Edgexperts using the GB10's. Does your model with the revisions run a little smoother on the split brain config ? I notice sometimes I lag a little no big deal but was curious if your model changes are more optimized for the dual spark play.
No stupid questions here — and yes, this pack was built and measured on exactly that setup: two GB10 boxes (TP=2) over a direct CX7 link, i.e. the same topology as a 2× MSI EdgeXpert pair.
What "smoother" means here, with numbers: only the 288 routed experts per MoE layer are quantized to 3-bit; attention, shared experts, dense MLP, embeddings, head and vision tower stay at source precision. On identical 2× GB10 hardware that measured −8 / −16 / −17.5% forward latency vs the 4bpw reference at spec-decode verify widths 1 / 4 / 8, and it leaves much more memory for KV — with KV_POOL_GIB=26.5 the pool is 3,891,200 tokens, so throughput doesn't degrade as context grows.
About the occasional lag on a two-box pair, three things worth checking (all from our own setup notes):
- Make sure both CX7 ports are actually in use. On our boxes the second 200G port was up but had no IPv4 / a mismatched GID, so NCCL silently ran single-rail. Getting both rails active took NCCL large-message throughput from 9.6 → 14.5 GiB/s and prefill +3–4%.
- Don't shrink the memory reserve to "give the model more room." On unified-memory GB10 machines a too-small reserve causes
NV_ERR_NO_MEMORY— and on DGX Spark that can take the whole box down, not just the job. Too large and your KV pool shrinks instead. The exact config behind each number is next to it inresults/benchmarks.mdin the model repo. - A little inter-box stall is inherent to TP across two machines. Every layer syncs twice, and in our profiling most of that sync time is the two machines waiting on each other's kernel jitter — not transfer. That part is normal on any 2-box TP setup; the weight-side savings above are what this pack actually removes.
Happy to share the exact launch config if you want to compare against yours.