litert-community/CLIPSeg-rd64-LiteRT

CLIPSeg rd64 — LiteRT on-device text-prompted segmentation

CLIPSeg (CVPR 2022, Apache-2.0) re-authored for LiteRT: type what you want to segment ("a cat", "the sky") and get a mask — no fixed class list. Three graphs — CLIP text and vision encoders on the CompiledModel GPU, the tiny 3-layer decoder on CPU (its 4-head/head_dim-16 attention fp16-miscomputes on the Mali delegate; the 12-head/head_dim-64 vision encoder survives at 0.998).

Input | prompt "a dog" | prompt "the grass" — the same image, two prompts, masks from the on-device model. Photo: "Lily the Golden Retriever in the grass" (Wikimedia Commons, Public Domain).

Verified on a Pixel 8a: text 761/761 GPU (8.7 ms) + vision 613/613 GPU (8.2 ms) + decoder CPU (exact); end-to-end device-vs-PyTorch logits corr 0.99998, mask IoU 0.9986.

Files

file	graph	delegate
clipseg_text_fp16.tflite	token-emb [1,77,512] → hidden [1,77,512]	GPU
clipseg_vision_fp16.tflite	image [1,3,352,352] → t3,t6,t9 [1,485,768]	GPU
clipseg_decoder.tflite (fp32)	t3,t6,t9,cond[512] → logits [1,352,352]	CPU
token_embedding_f16.bin, text_projection_f16.bin, vocab.json, merges.txt	host assets	—

Minimal usage (Python)

import numpy as np, torch
from PIL import Image
from transformers import CLIPSegProcessor
from ai_edge_litert.interpreter import Interpreter

proc = CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
img = proc(images=Image.open("photo.jpg"), return_tensors="pt")["pixel_values"].numpy()  # [1,3,352,352]

vis = Interpreter("clipseg_vision_fp16.tflite"); vis.allocate_tensors()
vis.set_tensor(vis.get_input_details()[0]["index"], img); vis.invoke()
t = [vis.get_tensor(o["index"]) for o in sorted(vis.get_output_details(), key=lambda o: o["index"])]  # t3,t6,t9

# cond[512] from the text encoder (token-emb lookup -> text graph -> EOT row @ text_projection)
dec = Interpreter("clipseg_decoder.tflite"); dec.allocate_tensors()   # CPU (exact)
ins = dec.get_input_details()
for d, arr in zip(ins, [t[0], t[1], t[2], cond]):   # cond: [1,512] float32
    dec.set_tensor(d["index"], arr.astype(np.float32))
dec.invoke()
mask = 1 / (1 + np.exp(-dec.get_tensor(dec.get_output_details()[0]["index"])[0]))   # sigmoid, [352,352]

Kotlin (Android)

// vision + text: Accelerator.GPU; decoder: Accelerator.CPU
val vis = CompiledModel.create(File(dir,"clipseg_vision_fp16.tflite").path, CompiledModel.Options(Accelerator.GPU), null)
val dec = CompiledModel.create(File(dir,"clipseg_decoder.tflite").path, CompiledModel.Options(Accelerator.CPU), null)
// vision: image[1,3,352,352] -> t3,t6,t9   decoder: (t3,t6,t9,cond[512]) -> logits[1,352,352]
// text graph + host BPE/emb-lookup/text_projection produce cond; see ClipSeg.kt in the LiteRT sample.
val logits = decOut[0].readFloat()   // sigmoid -> mask

Conversion

Re-authored with litert-torch: qkv-3D-BMM attention, quick-GELU, baked interpolated pos-embed (14²→22² @352), host-side token-embedding lookup, safe_ln_up (up-scaled LayerNorm keeping the eps fp16-normal), convT4x4 (exact non-overlapping ConvTranspose as 1×1-conv + 4-D interleave). The decoder ships on CPU because its small-head-dim attention fp16-miscomputes on the Mali GPU delegate (re-authoring is exact — desktop fp16 corr 0.999996).

Upstream

CIDAS/clipseg-rd64-refined (Apache-2.0). Please cite Lüddecke & Ecker, Image Segmentation Using Text and Image Prompts (CVPR 2022).

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
TFLite benchmark_model (TfLiteGpuDelegateV2) — clipseg_decoder.tflite	GPU (OpenCL)	44 / 196	96.2 ms
TFLite benchmark_model (TfLiteGpuDelegateV2) — clipseg_text_fp16.tflite	GPU (OpenCL)	32 / 761	154.4 ms
TFLite benchmark_model (TfLiteGpuDelegateV2) — clipseg_vision_fp16.tflite	GPU (OpenCL)	51 / 613	did not run
TFLite benchmark_model — clipseg_decoder.tflite	CPU (XNNPACK, 4 threads)	—	11.7 ms
TFLite benchmark_model — clipseg_text_fp16.tflite	CPU (XNNPACK, 4 threads)	—	53.8 ms
TFLite benchmark_model — clipseg_vision_fp16.tflite	CPU (XNNPACK, 4 threads)	—	791.8 ms

Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.

On this delegate the CPU is the faster choice for clipseg_decoder.tflite (11.7 ms on CPU against 96.2 ms on GPU), clipseg_text_fp16.tflite (53.8 ms on CPU against 154.4 ms on GPU) — worth knowing before you reach for the GPU on a mid-range phone.

Note that the GPU does not take the whole graph here (44 / 196 in clipseg_decoder.tflite, 32 / 761 in clipseg_text_fp16.tflite, 51 / 613 in clipseg_vision_fp16.tflite); the remainder runs on the CPU and the split costs a per-partition round trip.

Snapdragon NPU (Hexagon)

• clipseg_decoder.tflite — the NPU is 1.95x faster than the GPU (1.67 ms against 3.25 ms) and loads 7.72x faster (102 ms against 784 ms).
• clipseg_text_fp16.tflite — the NPU is 2.95x faster than the GPU (2.10 ms against 6.20 ms) and loads 7.55x faster (138 ms against 1041 ms).
• clipseg_vision_fp16.tflite — the NPU is 2.23x faster than the GPU (15.90 ms against 35.51 ms) and loads 8.17x faster (192 ms against 1569 ms).

file	backend	compiled	inference (median / min)	load
clipseg_decoder.tflite	NPU (Hexagon v81)	on-device JIT	1.67 ms / 1.63 ms	102 ms
clipseg_decoder.tflite	GPU (Adreno)	—	3.25 ms / 2.82 ms	784 ms
clipseg_text_fp16.tflite	NPU (Hexagon v81)	on-device JIT	2.10 ms / 2.07 ms	138 ms
clipseg_text_fp16.tflite	GPU (Adreno)	—	6.20 ms / 6.13 ms	1041 ms
clipseg_vision_fp16.tflite	NPU (Hexagon v81)	on-device JIT	15.90 ms / 15.28 ms	192 ms
clipseg_vision_fp16.tflite	GPU (Adreno)	—	35.51 ms / 32.45 ms	1569 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.71–0.74, where 1.0 is the throttling threshold.

The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. Those first compiles took 1.0 s to 18 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.

GPU wiring: GPU guide.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT benchmark_model tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).

File	Inference (median)	Spread (min–max)	Runs	Peak memory
clipseg_decoder.tflite	19.0 ms	18.8–21.1 ms	159	135 MB
clipseg_text_fp16.tflite	69.9 ms	68.5–86.7 ms	150	258 MB
clipseg_vision_fp16.tflite	856.7 ms	845.5–896.5 ms	150	504 MB