Glint-2.3

Glint-2 with a delete key. Thats about it.

parameters	1,710,145 (Glint-2's 1,710,049 plus one embedding row)
architecture	one shared block of 645,312, looped 8 times, plus one coda block
vocabulary	4097 (4096 BPE plus <bs>)
attention	causal sliding window of 256
context	4096
model file	7 MB
wiki byte ppl	2.993 (Glint-2: 3.127)
BLiMP	69.70 (Glint-2: 74.49)
ARC-Easy	37.00 (Glint-2: 36.20)

quick start

pip install -r requirements.txt

python infer.py --info
python infer.py "The capital of France is"
python infer.py "Once upon a time" --raw --stats
python infer.py --repair "text you want it to check"
echo "The sun is" | python infer.py --tokens 120

Everything needed to run the model is in this directory. infer.py imports torch and tokenizers, carries its own copy of the architecture, and reads its shape from config.json, so there is no project to clone.

from infer import Writer

writer = Writer()
stream = writer.generate("The capital of France is", tokens=60, seed=7)

print(writer.visible(stream))    # what a reader sees, deletions applied
print(writer.raw(stream))        # the stream as emitted, retractions bracketed
print(writer.deletions(stream))  # how many it used

the coda.

Same weights, same tokenizer, same 512 KiB of WikiText-2, at sequence length 1024:

	parameters	wiki byte ppl
Glint-2 as published (generate.py)	1,064,737	3.522
Glint-2 with its coda block	1,710,049	3.127

The card's own numbers (1.71M, 3.09) match the second row, so the checkpoint was always fine. If you have been running Glint-2 from its release script, you have been running a different and worse model than the one that was benchmarked.

how the deletion was taught

Glint-2 and Shard-2 share 275 of 4096 token ids. Nothing can be distilled through logits across that gap. What can cross is the teacher's judgement, carried as characters:

1. Shard-2 reads real fineweb text in its own tokenizer and draws its own next token at every position, restricted to the same top-40 the decoder here samples with. Draws it rated below 0.02 and that disagree with the corpus are candidate mistakes.
2. Those candidates are spliced in — runs of one to three tokens — and Shard-2 reads the spliced stream once more. At the end of each run it is asked one question: how much probability do you put on deleting what you just wrote? 350,305 of 583,766 candidates (60.0%) cleared 0.25 and were kept.
3. Each surviving repair is recorded as (character offset of the gold token, the text that was retracted) and replayed into Glint-2's vocabulary at the same characters — but only where Glint-2's tokenizer also starts a token there, so no splice ever cuts one of its tokens in half. One teacher token often re-encodes to several student tokens, so multi-token retractions appear in the corpus for free.

The result is 18.04M tokens in which 4.65% of positions are a retracted mistake, and in which the visible stream is byte-for-byte the original fineweb text. What it looks like:

made and played[ around][ animals][ander]⌫⌫⌫ by Neandertals
Now[ claim]⌫ it's time to stop the music
Here are some[ evidence]⌫ examples
Educate those around you on the importance of[ using][ environmental]⌫⌫ lifestyle decisions

The wrong tokens are context, never targets. Training to predict them would teach the model to make the mistakes rather than to notice them.

did distilling beat just using the model's own errors?

Yes, on the thing the distillation was supposed to move — and the control was run at a matched token budget and a matched repair density so the only difference is where the repairs are.

	wiki byte ppl	BLiMP	ARC-E	P(⌫) at a repair site	elsewhere	separation	AUC
Glint-2	3.127	74.49	36.20	—	—	—	—
taught by Shard-2	3.051	67.30	35.00	0.445	0.029	15.3x	0.874
control, own errors	3.078	66.39	36.00	0.342	0.036	9.6x	0.851

(both at learning rate 3e-4, 17,549 blocks, 4.65% vs 4.86% repair density; the probe is on held-out corpus the model never trained on.)

The distilled model is better at knowing where a retraction belongs, and it damages the base less doing it. A 1.71M model learning from a 49.4M model's error judgement beats it learning from its own. At this size is mostly noise.

does the deletion fire, and does it help

Once upon a time, there was a little b[oy][ named][ T][n][o]⌫⌫⌫⌫⌫ about the day.
happily stopped themselves[ at][ him]⌫⌫
The people who believe ... are[ also][ more]⌫⌫ among others for other social attitudes

So yeah I guess it does.

what SFT costs, and what DPO said about it

SFT can only say a deletion belongs here. It can never say this deletion was a mistake, because the corpus contains no deletions that were wrong. oof. yeah.

	wiki byte ppl	BLiMP	ARC-E	P(⌫) at a site	elsewhere	separation
Glint-2	3.127	74.49	36.20	—	—	—
SFT, lr 3e-4	3.051	67.30	35.00	0.4447	0.0290	15.3x
SFT, lr 1e-4	2.977	67.91	37.60	0.4323	0.0282	15.4x
SFT, lr 3e-5	2.977	69.85	36.20	0.2275	0.0079	28.9x
SFT, 4400 blocks @ 3e-5	2.987	71.46	35.80	0.0001	0.0000	4.2x
SFT, 8800 blocks @ 3e-5	2.979	71.04	36.80	0.0003	0.0000	6.8x

DPO; what it found. SFT moves the whole model; DPO moves only the decision. Wherever the policy emits <bs>, the same prefix is decoded again with the deletion blocked for exactly one step, so both branches are continuations the model would have written. Shard-2 reads the two visible texts and prefers one. Gradients flow only through that difference and a KL anchor to a frozen copy of the policy holds the rest still.

the real read

This is a 1.71M parameter model. It is a research artifact and it will drift within two sentences on anything encyclopedic. I hope that isnt a suprise to anybody. Specifically:

• The deletion token continues runs rather than starting them on evidence, per the correction above.
• Installing it costs BLiMP: 74.49 to 69.70. Measured separately, having the token is free -- a mean-row <bs> with no training leaves BLiMP at exactly 74.49 -- so the cost is the fine-tune, not the extra vocabulary entry. Perplexity and ARC-Easy both improved.
• The gains under the independent judge are small and the sample sizes behind them are small. Read +0.008 as "not negative", not as "better".
• The effort ladder from Glint-2 (effort.py, effort_probe.pt) is not ported. The probe is a linear layer on the hidden state of the coda-less model; with the coda running it is miscalibrated and would need retraining.

reproduce

hf download Glint-Research/Glint-2 --local-dir runs/glint23/glint-2

python3 glint23.py build --chars 65000000 --rate 0.26 --endorse 0.25 --out runs/glint23/teacher
python3 glint23.py train runs/glint23/teacher --learning-rate 3e-5 --out runs/glint23/sft.pt
python3 glint23.py dpo runs/glint23/sft.pt --blocks runs/glint23/holdout --out Glint-2.3.pt \
  --pairs 2048 --temperature 1.0 --top-k 0 --epochs 3 --learning-rate 1e-5

python3 glint23.py bench Glint-2.3.pt
python3 glint23.py eval  Glint-2.3.pt --sweep bias --delete-bias 3
python3 glint23.py probe Glint-2.3.pt --blocks runs/glint23/holdout --skip 0

files

Glint-2.3.pt      the weights
tokenizer.json    4096-vocab bpe, unchanged from Glint-2; <bs> is id 4096 and lives
                  in the model, not the tokenizer
config.json       shape, and the deletion token id
infer.py          standalone generation with deletions; torch + tokenizers, nothing else
requirements.txt
README.md         you are here

citation

Glint Research, 2026. Built on Glint-2; the deletion token is distilled from Shard-2 (49.4M), whose own recipe is documented at Glint-Research/Shard-1.