docs: measure both builds under llama.cpp, including REAP152

README.md

@@ -243,28 +243,34 @@ step.** The converter sizes the head's experts from `n_routed_experts`, which is
 recipe that fixes it — or reuse a drafter converted from REAP128, whose
 `mtp.*` tensors are byte-identical to this build's.
 
-**No llama.cpp figure for this build has been measured** — every REAP152 number
-on this card is from vLLM. On REAP128, where both engines were measured, the
-same weights accept 63.1% of drafts under llama.cpp against 53.2% under vLLM.
-Full recipe in
-[Serving on DGX Spark](docs/SERVING.md#serving-with-llamacpp).
+Measured on one DGX Spark, warm, same four prompts and draft width as every
+other figure here: **20.9 tok/s with the shipped 128-expert head** (44.2%
+acceptance, 2.44 accepted drafts per step), 21.8 tok/s with the unpruned
+256-expert head, and 15.1 tok/s with no drafter at all. vLLM is slightly faster
+end to end on this build (23.7 tok/s) while accepting fewer drafts (2.03 per
+step) — the same trade it makes on REAP128. Full recipe and the complete
+cross-engine table in
+[Serving on DGX Spark](docs/SERVING.md#what-it-measures-and-where-it-differs-from-vllm).
 
 ## Measured performance
 
-One DGX Spark, greedy decoding, 0.85 utilization, 64K context, all three
-overlays mounted:
+One DGX Spark, greedy decoding, all three overlays mounted under vLLM, warm —
+the first request after a load runs up to 1.8x slower there and is discarded:
 
 | | this build |
 |---|---:|
-| weights loaded | 95.58 GiB |
-| decode with DSpark | 14.5 tok/s |
-| draft acceptance | 40.7%, 2.03 tokens per step |
-
-For scale, REAP128 measured 16.5 tok/s without speculation and 23.5 tok/s with
-it, and 53.2% acceptance on the same short-prompt check. **This build has no
-non-speculative throughput measurement**, and the two figures above come from one
-short run; throughput depends heavily on prompt shape, context and software
-build.
+| weights loaded (vLLM, 0.85) | 95.58 GiB |
+| decode with DSpark, vLLM, 8K | 23.7 tok/s |
+| decode with DSpark, llama.cpp, 8K | 20.9 tok/s |
+| decode without a drafter, llama.cpp, 8K | 15.1 tok/s |
+| draft acceptance | 40.7% / 2.03 per step (vLLM), 44.2% / 2.44 (llama.cpp) |
+
+The same configuration at 64K starts and answers correctly, with 3.32 GiB of
+cache and 212,831 KV tokens; its first-request throughput there was 14.5 tok/s,
+which is a cold number and should be read as such. **There is still no
+non-speculative vLLM measurement of this build** — the 15.1 tok/s plain figure is
+llama.cpp. For scale, REAP128 measures 28.2 tok/s under vLLM and 26.3 under
+llama.cpp with the same drafter and prompts.
 
 ### The 2-bit build speculates too, and it is faster — measured here
 
@@ -281,13 +287,19 @@ notes below — both cost more than the differences being measured):
 |---|---:|---:|---:|---:|
 | 2-bit IQ2_M, 256 experts | 128 | 62.9% | 3.22 | **36.1 tok/s** |
 | 2-bit IQ2_M, 256 experts | 256 | 60.6% | 3.05 | 35.4 tok/s |
-| REAP128 (sibling) | 128 (shipped) | 63.1% | 3.19 | 26.9 tok/s |
-| REAP128, unpruned head | 256 | 66.4% | 3.38 | 27.8 tok/s |
-
-REAP152 itself is absent from that table because it has not been served under
-llama.cpp; its rows elsewhere on this card are all vLLM measurements.
-
-The unpruned 2-bit build is **1.34x faster** with matched engine, harness and
+| REAP128 (sibling) | 128 (shipped) | 63.1% | 3.19 | 26.3 tok/s |
+| REAP128 (sibling), unpruned head | 256 | 66.4% | 3.38 | 27.8 tok/s |
+| **this build (REAP152)** | 128 (shipped) | 44.2% | 2.44 | 20.9 tok/s |
+| **this build**, unpruned head | 256 | 47.7% | 2.64 | 21.8 tok/s |
+
+**This build drafts materially worse than its sibling, on both engines.** 2.44
+accepted drafts per step against REAP128's 3.19 under llama.cpp, 2.03 against
+2.66 under vLLM — 24% less either way, from a byte-identical draft head. That
+head was calibrated against the 128-expert target; against 152 experts it is
+predicting a model it never saw. Choosing this build means choosing its quality
+edge on MMLU, not its speed.
+
+The unpruned 2-bit build is **1.37x faster** with matched engine, harness and
 draft width, at acceptance within a few points. It also scored 84.39% against
 this build's 58.05% on the generative MMLU sample. At this footprint, on this
 hardware, pruning is not buying speed either. What this checkpoint offers is
@@ -300,7 +312,7 @@ prompts and the same draft width, its drafter lands **53.2%** where llama.cpp
 lands 63.1% on weights converted from the same checkpoint — 2.66 accepted drafts
 per step against 3.19. The unpruned head shows the same shortfall (60.0% against
 66.4%). vLLM's individual steps are faster, so end-to-end throughput comes out
-similar (28.2 against 26.9 tok/s), but roughly a fifth of the drafting is being
+similar (28.2 against 26.3 tok/s), but roughly a fifth of the drafting is being
 left on the table. **The cause is not known.** KV quantization is ruled out:
 llama.cpp with `q8_0` KV reproduces its own f16 acceptance *bit for bit* (570 of
 903 either way), and vLLM's DSv4 path refuses anything but fp8, so the two cannot

SHA256SUMS

@@ -1,7 +1,7 @@
 11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361  .gitattributes
 f2c6c602815669d292889e5be8c802f2ed950653b77999b1584e8e6aed25d040  LICENSE
 51baf525dd29349e29e44bf04ca48a0632198bb28aab945431e3a31dc1d78acc  LICENSE-CODE
-f27a62515a204b106cac625ee7c0734c5f515925d6da9be63b82efa2d7e3b21d  README.md
+c41ad5c5dcf781862fe0c2196c79b0f8e9f7cdafb3d609ac68780cf231c307a1  README.md
 af6f2a8a05f4148cfde5020b0711aa13c816845c1884d27df91528b20d3d838c  calibration/README.md
 34a35532512eabeb90181b2b35fcd87b151373f5e0c2cf62059d73a693e1beac  calibration/calib.pt.sha256
 b0f7ec170f9599b439f8c277a9dceff4091a587786ca96a8ad6ad9545bb491c1  calibration/examples/mix-de-en.example.json
@@ -13,7 +13,7 @@ c0f488896e41c60a412e36a601cede420c2bffae2e419ee56121b24871c9dd88  calibration/ta
 64ef8f5c564799217d8f8442a2c400c8478d95ad048ecb170b9ebeb3d5ff52c7  config.json
 356f01a69634da18c0c9fc435fec7295548e6b0aeb3869c51feabf477653d09a  docs/PROVENANCE.md
 4484e4c66a386c7096787d30a327300939f52d094f6590d4f7b89de59b80ab60  docs/QUALITY.md
-ece5bdec36af4f92e7975c3b34bde18913c59a6247ca09981b750eed7b90c365  docs/SERVING.md
+60e16d3a6e77a2f6e9b5459b0fac08bb748705b63ccc6862c4c94d4664604438  docs/SERVING.md
 5fccff80f55a4d455bbe516bdd552edf3e9623df95e99fbf2a3c3389fdf91af0  generation_config.json
 cbc166fe80eef86de49506511bebb7812399fd9de0f38e27e27b76baeff87dbe  model-00001-of-00026.safetensors
 ec81df9003c081d0128e78fa35c67df5987ebe46b9f7ad3f99c5f4d6e476e4d8  model-00002-of-00026.safetensors

docs/SERVING.md

@@ -134,7 +134,8 @@ separately below. The output type is detected from the checkpoint (`bf16`) and
 does not need to be given: routed **FP4 experts are repacked to MXFP4**, FP8
 attention weights become **Q8_0**, and the remainder stays BF16 or F32. REAP128
 converts to 82.7 GB / 1,328 tensors in about four and a half minutes on the
-build machine. REAP152 converts identically and keeps `expert_count = 152`.
+build machine. REAP152 converts identically to 96.5 GB / 1,328 tensors, keeping
+`expert_count = 152`.
 
 ### Convert the draft head
 
@@ -221,26 +222,51 @@ Three settings are worth stating explicitly:
 
 ### What it measures, and where it differs from vLLM
 
-One DGX Spark, four prompts, greedy, 8,192-token context, draft width 5, warm:
+One DGX Spark, four prompts, greedy, 8,192-token context, draft width 5, warm —
+every row taken with the same script and the same harness:
 
 | target | draft head | engine | acceptance | accepted drafts / step | decode |
 |---|---|---|---:|---:|---:|
-| REAP128 | 128 (shipped) | llama.cpp | **63.1%** | **3.19** | 26.9 tok/s |
+| REAP128 | none | llama.cpp | — | — | 15.4 tok/s |
+| REAP128 | 128 (shipped) | llama.cpp | **63.1%** | **3.19** | 26.3 tok/s |
 | REAP128 | 256 (unpruned) | llama.cpp | 66.4% | 3.38 | 27.8 tok/s |
 | REAP128 | 128 (shipped) | vLLM | 53.2% | 2.66 | 28.2 tok/s |
 | REAP128 | 256 (unpruned) | vLLM | 60.0% | 3.00 | — |
+| REAP152 | none | llama.cpp | — | — | 15.1 tok/s |
+| REAP152 | 128 (shipped) | llama.cpp | 44.2% | 2.44 | 20.9 tok/s |
+| REAP152 | 256 (unpruned) | llama.cpp | 47.7% | 2.64 | 21.8 tok/s |
+| REAP152 | 128 (shipped) | vLLM | 40.7% | 2.03 | 23.7 tok/s |
+| 2-bit IQ2_M, 256 experts | none | llama.cpp | — | — | 19.0 tok/s |
 | 2-bit IQ2_M, 256 experts | 128 | llama.cpp | 62.9% | 3.22 | 36.1 tok/s |
 | 2-bit IQ2_M, 256 experts | 256 | llama.cpp | 60.6% | 3.05 | 35.4 tok/s |
 
-vLLM's steps are faster, so end-to-end throughput lands close, but it accepts
-about a fifth fewer drafts from the same weights. The cause is unknown and KV
-quantization is ruled out (above). Under llama.cpp, where the engine is held
-fixed, pruning the draft head costs only 4% of accepted drafts per step.
-
-**REAP152 has not been measured under llama.cpp.** The conversion path above was
-checked against it and produces a valid 152-expert GGUF, but no llama.cpp
-throughput or acceptance figure for that build exists in these documents; the
-REAP152 numbers published here are all from vLLM.
+Three things this table says.
+
+**vLLM's steps are faster; its drafting is not.** End-to-end throughput lands
+close, but it accepts about a fifth fewer drafts from the same weights — and the
+shortfall is a near-constant factor rather than a property of one build: 3.19
+against 2.66 accepted drafts per step on REAP128, 2.44 against 2.03 on REAP152,
+which is 1.20x both times. The cause is unknown and KV quantization is ruled out
+(above).
+
+**Pruning the draft head is cheap; mismatching it is not.** With the engine held
+fixed, the unpruned 256-expert head is worth 6% more accepted drafts per step on
+REAP128 and 8% on REAP152 — small either way. The gap *between builds* is much
+larger: REAP152 accepts 2.44 per step against REAP128's 3.19, 24% less, on both
+engines alike. Both builds ship the same byte-identical draft head, and it was
+calibrated against the 128-expert target; on the 152-expert target it is
+predicting a model it was not measured on.
+
+The REAP128 row was published earlier at 26.9 tok/s and re-measured here beside
+the REAP152 rows. The re-run reproduced its speculative counters **exactly** —
+570 of 903 drafts accepted, the same 3.19 per step — and its throughput within
+2%, so 26.3 against 26.9 is the run-to-run band on wall clock, not a change in
+behaviour. Greedy decoding fixes the tokens; only timing moves.
+
+**Speculation is where the throughput is.** Plain decode is 15.1-15.4 tok/s for
+either build — near-identical, because both activate 6 routed experts per token
+whatever their total — and the draft head takes REAP128 to 26.3 and REAP152 to
+20.9.
 
 ## Context length
 
@@ -333,13 +359,13 @@ DSpark (`--spec-type draft-dspark`), and `convert_hf_to_gguf.py --dspark`
 extracts the draft head from the official checkpoint. Measured here on the same Spark, same prompts, same harness, all warm: under
 **llama.cpp**, where both models can be compared on one engine, the 2-bit
 256-expert target runs 19.0 tok/s plain and **36.1 tok/s** with DSpark, against
-**26.9 tok/s** for REAP128 — 1.34x, at acceptance within a few points (62.9%
+**26.3 tok/s** for REAP128 — 1.37x, at acceptance within a few points (62.9%
 against 63.1%).
 
 **vLLM gets less out of the same weights.** REAP128 under vLLM 0.25.1 accepts
 **53.2%** of its drafts where llama.cpp accepts 63.1% from weights converted out
 of the same checkpoint, and 60.0% against 66.4% with the unpruned head. Its steps
-are faster, so throughput lands close (28.2 against 26.9 tok/s), but about a
+are faster, so throughput lands close (28.2 against 26.3 tok/s), but about a
 fifth of the drafting is lost. The cause is unknown; KV quantization is ruled out
 (llama.cpp reproduces its acceptance bit-for-bit with `q8_0` KV, and this vLLM
 path refuses anything but fp8). Treat vLLM acceptance figures as a floor.