Post
689
🚀 opencoti-llamafile 0.10.3-c6 — multi-stream everywhere
New cut of the opencoti single-file inference engine (llamafile 0.10.3 / llama.cpp + 94 additive patches). One zero-dependency APE executable for Linux, Windows, macOS & BSD.
What's new vs c5:
⚡ Windows GPU, one file. New win-gpu .exe with the CUDA DLL embedded — download & run, no side-load. Plus a universal binary carrying every GPU payload. All CUDA payloads are nvcc-compressed (zero measured load/throughput cost) to fit Windows' 4 GiB image cap; five artifacts now ship per release, incl. aarch64 with embedded sbsa CUDA for DGX Spark (GB10).
🧩 Flat multi-stream KV. Split-KV multi-slot serving was 30–60× slower than unified (per-layer KV reassembly every step). Rebuilt as flat stream-contiguous layouts with zero-copy views — sliding-window layers and the pinned-host spill tail included: window + --parallel 2 went 1.2 → 25.3 tok/s aggregate, token-identical.
🔓 PolyKV no longer requires --kv-unified. Pools snapshot & share prefixes on split KV too — the c5 limitation is gone. /capacity is now stream-aware (kv_streams_* fields).
🤝 MTP × multi-stream. The assistant-MTP "force unified" guard is retired: the draft context mirrors the target's streams, split vs unified token-identical. Validated 7/7 Gemma-4 draft pairs on the shipped bytes, 1.39–1.97× decode.
⏱️ Deterministic GPU sharing. Feedback pacing replaced by weighted time-division on the wall clock: exact ratios by construction, zero solo tax, cross-process on Windows too. Peers now keyed by PCI bus multi-GPU safe.
🚛 r2 same-day re-cut: concurrent sessions with quantized spilled KV tails were collapsing to ~⅙ PCIe bandwidth; per-block bulk staging restores it — dual q4_0 4.3 → 19.2 tok/s aggregate (within 5% of solo), token-identical.
🔗 Repo (binaries, DSOs, full patch series, docs): ManniX-ITA/opencoti-llamafile
New cut of the opencoti single-file inference engine (llamafile 0.10.3 / llama.cpp + 94 additive patches). One zero-dependency APE executable for Linux, Windows, macOS & BSD.
What's new vs c5:
⚡ Windows GPU, one file. New win-gpu .exe with the CUDA DLL embedded — download & run, no side-load. Plus a universal binary carrying every GPU payload. All CUDA payloads are nvcc-compressed (zero measured load/throughput cost) to fit Windows' 4 GiB image cap; five artifacts now ship per release, incl. aarch64 with embedded sbsa CUDA for DGX Spark (GB10).
🧩 Flat multi-stream KV. Split-KV multi-slot serving was 30–60× slower than unified (per-layer KV reassembly every step). Rebuilt as flat stream-contiguous layouts with zero-copy views — sliding-window layers and the pinned-host spill tail included: window + --parallel 2 went 1.2 → 25.3 tok/s aggregate, token-identical.
🔓 PolyKV no longer requires --kv-unified. Pools snapshot & share prefixes on split KV too — the c5 limitation is gone. /capacity is now stream-aware (kv_streams_* fields).
🤝 MTP × multi-stream. The assistant-MTP "force unified" guard is retired: the draft context mirrors the target's streams, split vs unified token-identical. Validated 7/7 Gemma-4 draft pairs on the shipped bytes, 1.39–1.97× decode.
⏱️ Deterministic GPU sharing. Feedback pacing replaced by weighted time-division on the wall clock: exact ratios by construction, zero solo tax, cross-process on Windows too. Peers now keyed by PCI bus multi-GPU safe.
🚛 r2 same-day re-cut: concurrent sessions with quantized spilled KV tails were collapsing to ~⅙ PCIe bandwidth; per-block bulk staging restores it — dual q4_0 4.3 → 19.2 tok/s aggregate (within 5% of solo), token-identical.
🔗 Repo (binaries, DSOs, full patch series, docs): ManniX-ITA/opencoti-llamafile