LFM2.5-2.6B — LiteRT-LM

LiquidAI/LFM2.5-2.6B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14; the bundled executor-metadata section makes the same file run on 0.15's new state binding as well).

LFM2.5-2.6B is the flagship of Liquid AI's LFM2.5 on-device family: a hybrid architecture (22 gated short-convolution blocks + 8 grouped-query attention layers) and a thinking model — it reasons inside a <think>…</think> block before answering. The bundle declares the thought channel, so LiteRT-LM apps receive the reasoning and the answer as separate streams.

File	Recipe	Size	GSM8K (n=100)
LFM2.5-2.6B_int8.litertlm	int8 dynamic (linears + convs + embedding)	2.87 GB	88% (bf16 reference: 92%)
LFM2.5-2.6B_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	1.67 GB	83%

Context (KV cache)	4096 max (see --max-num-tokens note)
Backend	CPU, and GPU with litert-lm ≥ 0.16.0 (Android OpenCL and macOS — verified by generation, see Speed; iOS Metal still fails at engine creation, tracked upstream in LiteRT-LM#3129)
Template	bundled — ChatML-style with tool-list support; generation prompt pre-fills <think> (the model's trained convention); thought channel declared
Base model	LiquidAI/LFM2.5-2.6B (LFM Open License v1.0)

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 2048 (thinking model), n=100, same harness for all rows:

Configuration	GSM8K
PyTorch bf16 (reference, MPS)	92%
LiteRT int8 (this repo)	88%
LiteRT int4-b32 OCTAV (this repo)	83%

Both files pass an 8-question sanity gate 8/8 with zero degenerate outputs (CPU), a 42-length prefill sweep with zero corrupt first tokens (fresh engine per length), and a 3-turn conversation gate (fact recall + arithmetic).

Usage

pip install litert-lm
litert-lm run ./LFM2.5-2.6B_int4.litertlm --prompt "What is 17 + 25?"

The model thinks before answering — through the Python/C++ API the reasoning arrives on the thought channel and the reply stays clean; the CLI prints both.

--max-num-tokens tip: reasoning turns are long — give the model at least 2048 tokens of budget for math/complex questions, or the think block may not close. Decode speed drops as the token budget grows, so don't set it higher than you need.

Multi-turn note: on the current runtime the conversation context retains previous turns' reasoning, so long chats fill the context faster than the visible text suggests. For unrelated questions, prefer fresh conversations.

Speed

litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024 (prefill-1024 rows measured at max-num-tokens 2048):

Variant	Prefill (256)	Prefill (1024)	Decode	TTFT
int8	169 tok/s	434 tok/s	38.0 tok/s	1.54 s
int4	156 tok/s	193 tok/s	43.7 tok/s	1.66 s

GPU backend (litert-lm benchmark … --backend gpu --cache no, litert-lm 0.16.0, same machine; both variants verified to generate correct text on GPU before quoting):

Variant	Prefill (256)	Decode
int8	1893 tok/s	125.6 tok/s
int4	1972 tok/s	161.6 tok/s

On Android the same bundle runs GPU-accelerated with litert-lm ≥ 0.16.0. Pixel 8a (Tensor G3), litert_lm_main v0.16.0 built from the release tag, 261-token prompt with generation capped at 256, 3 runs per backend (int4 file):

Backend	Prefill (261 tok)	Decode	TTFT
GPU (OpenCL)	87.8–88.4 tok/s	11.0–11.2 tok/s	3.04–3.06 s
CPU	16.2–30.2 tok/s	7.4–10.2 tok/s	8.7–16.3 s

At this size the GPU wins across the board on-device — ~3–5× prefill, ~3–5× faster to the first token, and a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat to within 1%) — so prefer the GPU backend where the runtime offers it. A short prompt shifts the ratio further toward the GPU: with a 19-token prompt the same file prefills at 37.7–38.8 tok/s on GPU against 4.1 tok/s on CPU.

iPhone 17 Pro (CPU backend, cold first runs): the int4 file answers the 8-question quality gate 8/8 and decodes at ~20 tok/s warm (the first, cold run is slower while caches build). The int8 file also loads and runs on iPhone 17 Pro when the host app carries the extended-virtual-addressing / increased-memory entitlements (its 2.87 GB single weight section exceeds what a default-entitlement app will memory-map) — int4 is the recommended phone variant (smaller, faster on phone-class memory bandwidth).

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend: LiteRT takes the whole graph in each and the model generates.

file	GPU backend	delegation	peak
LFM2.5-2.6B_int4.litertlm	runs	11132 / 11132 ops across 12 subgraphs on LiteRT GPU	1231 MB
LFM2.5-2.6B_int8.litertlm	runs	11132 / 11132 ops across 12 subgraphs on LiteRT GPU	1189 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24 and 2026-08-25.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. GPU works from litert-lm ≥ 0.16.0 (litert_lm_main --backend=gpu, or the SDKs); the Gallery app bundles its own runtime which may lag behind that version — if its GPU toggle fails, use CPU there.

Conversion notes

Converted with released litert-torch 0.9.2 (upstream lfm2 hybrid support incl. the ShortConv prefill-pad fix) and packaged for litert-lm ≥ 0.14 (the executor-metadata section binds the 22 conv states + 16 KV caches; files exported without it do not run on 0.15).

• The generation prompt pre-fills <think> exactly as the vendor chat template does. This matters: with a bare assistant prompt, think-block emission becomes the model's choice, and the int4 quantization degrades that discipline first (unscaffolded rambling in place of answers on multi-turn). With the pre-fill, all variants answer tersely and the thought channel routes cleanly.
• The checkpoint is notably sparse: OCTAV int4-b32 produced 746k all-zero weight blocks whose zero scales the XNNPACK delegate rejects at load. The zero scales are patched to the tensor's smallest nonzero scale (dequantization unchanged — the blocks are all zeros).
• int8 quantizes convs at export time (safe); post-hoc conv int8 breaks generation. int4 keeps convs float.
• Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; a 42-length first-token sweep verifies no prefill-padding state corruption.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
LFM2.5-2.6B_int4.litertlm	28.5 (28.5–28.5)	4.9 (4.9–4.9)	9.2 s	2.9 GB
LFM2.5-2.6B_int8.litertlm	43.8 (43.6–43.9)	3.5 (3.5–3.5)	6.1 s	3.9 GB