Zenyx-100M-Base

A 151M-parameter dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).

Status: pre-training in progress. Latest checkpoint: step 7,700 (4.04B tokens) of a 35B-token budget. The full benchmark suite was last run at step 6,000 (3.15B tokens); quick re-checks at steps 7,000, 7,100 and 7,400 are reported in Quick checks after step 6,000, and the complete suite will be refreshed at a later checkpoint. This is a base model: not instruction-tuned, no safety tuning, and it makes up facts. A post-training stage is in preparation (see Planned post-training).

At a glance

Parameters	151.0M
Architecture	18-layer Llama-style decoder, grouped-query attention, SwiGLU
Context length	2,048 tokens
Vocabulary	49,105 tokens (byte-level BPE, 248 special tokens)
Training data	web, code, math, arithmetic reasoning (18 sources)
Training hardware	2 × Tesla T4, fp16
Training budget	35B tokens (4.04B done at step 7,700); the original plan was 10B
Latest checkpoint	Step 7,700
Decode speed	373 tokens/s single stream on one T4
License	Apache-2.0

Quick results

Measured at step 6,000 (3.15B tokens). Base model, no fine-tuning. Later quick checks (steps 7,000-7,400) are in Quick checks after step 6,000.

Benchmark	Result
Zenyx-Fact1000 (4-way factual test, chance 25%)	58.5% (95% CI 55.4–61.5)
Fact1000 without the arithmetic items	61.7%
SciQ (accuracy / length-normalised)	72.4% / 67.3%
ARC-Easy (accuracy / length-normalised)	44.4% / 40.1%
ARC-Challenge (accuracy / length-normalised)	19.7% / 23.5%
Pile bits per byte, 22 domains (lower is better)	1.145
Held-out training sources, bits per byte	0.851
Arithmetic, 14,000 problems in the model's own chat format	93.7% (details)
MMLU, 5-shot, all 14,042 questions (chance 25%)	25.5% ± 0.7 (at chance)
Python, 12 small functions run against unit tests	4 / 12 greedy; 25% of sampled attempts pass
Decode throughput, one T4	373 tokens/s, batch 1

Against the Muse2 research base models, evaluated with the same harness, the model is level or ahead on Fact1000, SciQ and the Pile, and level on ARC; see Comparison with Muse research models. Against the author's larger Zenyx-V3 it is clearly behind on knowledge and ahead on arithmetic; see Comparison with Zenyx-V3. Details and caveats are in Evaluation.

Model details

tokens → embedding (tied with the output head)
       → 18 × [ RMSNorm → grouped-query attention → add ]
               [ RMSNorm → SwiGLU MLP             → add ]
       → RMSNorm → output head

Component	Setting
Width / depth	d_model 768, 18 blocks
Attention	12 query heads × 64, 4 key/value heads (grouped-query), causal, per-head QK-norm
Positions	RoPE, θ = 500,000
MLP	SwiGLU, hidden size 2,048
Normalisation	RMSNorm, pre-norm
Embeddings	tied input/output, 49,152 rows
Bias / dropout	none
Precision	fp16 sub-layers with an fp32 residual stream (the T4 has no bf16)
Init	normal(0, 0.02); residual output projections scaled by 1/√(2·layers)
KV cache	18,432 bytes per token in fp16 (37.7 MB for a full 2,048-token sequence)

Why only grouped-query attention? Sliding-window, gated linear attention (GLA), plain linear attention and Gated DeltaNet-2 were benchmarked as replacements on a T4 at 2k context. None beat flash-attention GQA on speed, so every layer is plain GQA. Quality of a hybrid at equal compute was not trained or compared.

Tokenizer

Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens; the 248 special tokens never merge). Compression on 63 MB of fresh text from 18 sources, weighted by the data mixture, in bytes per token (higher means fewer tokens for the same text):

Tokenizer	Vocabulary	Bytes / token	Tokens per MB of text
Zenyx (this model)	49,105	4.56	219,250
Llama-3.2	128,256	4.17	239,808
DeepSeek-V4-Pro (identical to the tokenizer shipped with Zenyx-V3)	129,280	4.14	241,429
Qwen3	151,669	4.03	248,077
Same BPE without the superword stage	about 49k	3.94	n/a

The tokenizer needs about 9% fewer tokens than DeepSeek-V4-Pro for the same text with a vocabulary 2.6 times smaller (a 37.7M-parameter embedding table at width 768, against 99.3M for 129,280 rows).

• Where the gain is: web text (+9%), HTML (+22%), CSS (+16%), math text (+13% to +35%), Java (+13%), Python (+10%).
• Where it is not: Wikipedia and WikiText-103 are ties (4.31 against 4.34, and 4.51 against 4.53), markdown is 5% worse, and GSM8K is 3% worse.
• Caveats: the test sources are of the same kinds the tokenizer was tuned on, so the gain is probably overstated for unrelated text. Only English, code and math were measured; no other language was tested, and a larger multilingual vocabulary such as DeepSeek's is built for that. Compression does not by itself make a better model; bits per byte (below) is the tokenizer-independent comparison.

File: tokenizer_candidates_v3/F-49152-f70/tokenizer.json.

Training

Optimiser	Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms
Schedule	warmup, constant until 3.9B tokens, then linear decay to 10% of peak (AdamW 2e-3 → 2e-4; Muon 0.012 → 0.0012) between 3.9B and 15B tokens, then held at 10% to the 35B-token budget. Revised on 2026-10-21; before that the plan was constant until 8B tokens and a decay to 10B
Batch	524,288 tokens per step
Throughput	about 24.7k tokens/s on 2 × T4
Kernels	flash-attention for Turing, compiled blocks, fused cross-entropy

Data mixture (share of training tokens)

The mixture was revised at step 7,280 (3.82B tokens): less Python and arithmetic, more non-Python code, chat and fact-seeking text, and less FineWeb-Edu. The sampler tracks a per-source token quota; when the definition changes it re-bases its running tallies once so that the new shares apply from that point on and no source is starved or flooded.

Source	Steps 1 - 7,280 (earlier mixture)	From step 7,280 (current)
Code, total	35%	37.5%
of which Python (StarCoderData)	16%	12%
of which Java / JavaScript	4.5% / 5%	6.5% / 7%
of which HTML / CSS	2% / 0.8%	2.9% / 1.2%
of which Markdown / TeX	2.5% / 2%	2.5% / 2%
of which Nemotron code concepts	2.2%	3.4%
FineWeb-Edu	22%	17%
Nemotron-CC-Math (4+)	10.5%	10.5%
Arithmetic reasoning (SagheerLab/Arithmetic-Reasoning)	8%	7%
FineMath-4+	6%	6%
InfiWebMath-4+	5%	5%
Wikipedia (English)	5%	6%
Ultra-FineWeb (multi-style)	4%	4%
Nemotron fact-seeking	2%	3%
SmolTalk (chat, with varied system prompts)	1.5%	3%
Nemotron SFT-math	1%	1%

A second (decay-phase) mixture with more math is planned for the last part of the run (provisionally from about 31B tokens); its composition will be fixed closer to that point.

Evaluation

Progress across checkpoints

Checkpoint	Tokens	Fact1000 (all)	Fact1000 (no arithmetic)	SciQ acc / acc_norm	Pile bits per byte	Held-out bits per byte
Step 1,810	0.95B	50.6%	53.6%	not run	not run	0.944
Step 2,100	1.10B	53.0%	56.3%	69.9% / 63.0%	not run	not run
Step 5,300	2.78B	57.7%	61.3%	71.1% / 66.4%	1.150	not run
Step 5,400	2.83B	59.3%	62.2%	71.7% / 66.9%	1.151	0.856
Step 6,000	3.15B	58.5%	61.7%	72.4% / 67.3%	1.145	0.851

Fact1000 is scored by summed log-probability of each answer. Step 6,000 in detail: 585/1000 correct (95% CI 55.4–61.5); 510/826 without the 174 arithmetic items; 51.9% under mean-per-token scoring and 55.5% under per-character scoring; the prior-only control (answers scored in a neutral context) reaches 29.7%. Arithmetic items: 75/174. Fact1000 moves by about ±1.5 points between nearby checkpoints (57.7%, 59.3% and 58.5% at steps 5,300, 5,400 and 6,000), which is sampling noise; the slow upward trend across the table is the signal.

Quick checks after step 6,000

Re-runs of the same harness on a reduced set (Fact1000, SciQ, ARC-Easy and the 14,000-problem arithmetic suite; no Pile, held-out, MMLU or code tests). Step 6,000 is repeated for reference.

Checkpoint	Tokens	Fact1000 (all)	Fact1000 (no arithmetic)	SciQ acc / acc_norm	ARC-Easy acc / acc_norm	Arithmetic, exact match (14,000)
Step 6,000	3.15B	58.5%	61.7%	72.4% / 67.3%	44.4% / 40.1%	93.7%
Step 7,000	3.67B	61.0% (610/1000)	64.5% (533/826)	70.9% / 65.7%	44.4% / 40.0%	91.9%
Step 7,100	3.72B	not run	not run	not run	not run	82.4%
Step 7,400	3.88B	60.9% (609/1000)	63.8% (527/826)	71.0% / 63.4%	44.0% / 40.2%	72.1% (92.5% of first steps correct, see below)

Knowledge-style scores are flat to slightly higher (Fact1000 moves within its roughly ±1.5-point checkpoint-to-checkpoint noise). The exact-match arithmetic score fell, and the next section explains why that is a formatting failure and not a loss of the skill. Checkpoint 7,700 has not been benchmarked yet.

Other multiple-choice benchmarks (step 6,000)

Benchmark	Items	Accuracy	Length-normalised
SciQ	1,000	72.4%	67.3%
ARC-Easy	2,376	44.4%	40.1%
ARC-Challenge	1,172	19.7%	23.5%

SciQ prompts include the supporting passage. ARC-Challenge is at chance level (25%) for a model of this size and training budget.

Pile bits per byte, against Zenyx-V3 (step 6,000 and step 110,000)

Byte-weighted over 22 Pile domains (20.4 MB of text, window 1,024, stride 512): 1.145 for this model and 0.999 for Zenyx-V3 (unweighted domain mean 1.308 and 1.159). Bits per byte is independent of the tokenizer, so the two models are directly comparable.

Domain	Zenyx-V3	Zenyx-100M	Difference (100M − V3)
GitHub	0.576	0.719	+0.143
PubMed Central	0.741	0.867	+0.126
USPTO Backgrounds	0.779	0.908	+0.129
NIH ExPorter	0.861	1.003	+0.142
ArXiv	0.855	1.007	+0.152
PubMed Abstracts	0.873	1.029	+0.156
StackExchange	0.917	1.069	+0.152
Wikipedia (en)	0.987	1.111	+0.124
FreeLaw	0.958	1.127	+0.169
Pile-CC	1.098	1.225	+0.127
OpenWebText2	1.126	1.271	+0.145
BookCorpus2	1.142	1.273	+0.131
Enron Emails	1.222	1.348	+0.126
HackerNews	1.278	1.405	+0.127
Gutenberg (PG-19)	1.268	1.408	+0.140
OpenSubtitles	1.333	1.473	+0.140
PhilPapers	1.332	1.479	+0.147
DM Mathematics	1.326	1.487	+0.161
Books3	1.330	1.503	+0.173
Ubuntu IRC	1.714	1.899	+0.185
YouTube Subtitles	1.816	2.019	+0.203
EuroParl	1.964	2.149	+0.185
All 22 domains, byte-weighted	0.999	1.145	+0.146
Unweighted mean of domains	1.159	1.308	+0.149

V3 is ahead by 0.12 to 0.20 bits per byte in every one of the 22 domains, and the gap is close to uniform. It looks like a general quality difference, not a weakness in any one domain.

Held-out training sources (step 6,000)

Bits per byte on text held out from the training sources. All sources: 0.851 (step 1,810: 0.944).

Held-out source	Step 1,810	Step 6,000
FineWeb-Edu	1.200	1.105
Wikipedia	1.244	1.138
Python	0.693	0.622
FineMath-4+	1.035	0.921
Nemotron-CC-Math	0.739	0.657
Nemotron fact-seeking	0.985	0.830

Zenyx-Fact1000 is a frozen, hash-checked 1,000-item four-way test written by the author, so treat it as a progress measure, not an independent benchmark.

Arithmetic

About 8% of the mixture up to step 7,280 (7% since) is arithmetic reasoning data, roughly 230M arithmetic tokens by step 6,000 by the author's estimate (Zenyx-V3 saw about 1.8M). The model answers arithmetic prompts in a fixed chat-style format that it learned only for this task:

<|im_start|>user
What is 683 + 479?<|im_end|>
<|im_start|>assistant
<think>
683 + 479 = 1162
</think>

1162<|im_end|>

The <think> block is a one-line scratchpad (the problem restated with its result), not step-by-step working. The model computes the answer in a single forward pass. This is a base model, not an SFT chat model; the format comes from pre-training data. (Chat prompts need the model's chat marker at the start of the text; the bare format without it is not meaningful, 0 / 240 at step 6,000.)

14,000 problems in the model's own format (step 6,000). The problems are those of the author's 19-task benchmark suite: 2,000 generated with fixed seeds for each of seven types. Answers are read from the text after </think> and must match exactly.

Problem type	Correct
2-digit addition	99.7%
2-digit subtraction	97.8%
3-digit addition	98.9%
3-digit subtraction	86.9%
4-digit addition	98.3%
5-digit addition	90.1%
2-digit × 2-digit	84.1%
All 14,000	93.7% (13,114)

Fresh problems, new seeds (step 6,000, 300 per row, native format). A check that the skill is real and not memorised or leaked:

Operation	2 digits	3 digits	4 digits	5 digits	6 digits	7 digits	8 digits
Addition	99.7%	99.0%	99.0%	91.7%	5.7%	0.0%	0.0%
Subtraction	97.7%	89.3%	86.0%	67.3%	0.3%	n/a	n/a

Multiplication	2×2	3×2	3×3	4×2	4×3	4×4
Correct	84.3%	47.3%	8.0%	15.0%	1.7%	0.0%

Expressions with three terms: 0%. Scoring the same outputs against the true answer plus one gives at most 1.0%, so the checker is not lenient. A pair of 5-digit numbers has about 80 billion combinations, so 92% on those cannot come from recall. The pattern is a learned procedure that holds up to about 5 digits and breaks beyond the lengths it trained on. The final answer matches the number in the <think> line in 96% of cases.

Why the exact-match score fell after step 6,000 (steps 7,000 - 7,400). The first <think> line is still correct on 92.5% of the 14,000 problems at step 7,400, close to the 93.7% exact-match score at step 6,000. The loss is that the model often appends a spurious second step after a correct first one and then answers with the result of that extra step, for example 384 + 527 = 911 | 911 * 38 = 34718 followed by the answer 34718. In the pre-training arithmetic data 65% of documents are chained multi-step scratchpads (a+b=r | r*c=r2 | ...) and only about 2% are plain single-operation prompts of the form "What is a op b?", so continuing a chain is the model's default habit and it fires on some single-step prompts. By problem type at step 7,400:

Problem type	Exact match	First step correct
2-digit addition	90.0%	100.0%
2-digit subtraction	95.9%	95.9%
3-digit addition	31.5%	99.1%
3-digit subtraction	86.9%	86.9%
4-digit addition	98.6%	98.6%
5-digit addition	82.8%	82.9%
2-digit × 2-digit	19.1%	83.9%
All 14,000	72.1%	92.5%

The remaining errors (about 17% on 5-digit addition, 16% on 2-digit multiplication, 13% on 3-digit subtraction) are genuine arithmetic mistakes. The planned remedy is post-training with explicit single-step examples that stop after one scratchpad line, together with step-by-step working at higher reasoning-effort levels.

What it cannot do (12 hand-written prompts, step 6,000; 6 correct): 3-digit × 3-digit multiplication (150434 for 362 × 417, correct 150954), 6-digit addition (105419.4 for 482913 + 571264, correct 1054177), word problems (Tom's 12 apples minus 5 answered "14 apples"), algebra (3x + 5 = 20 answered 15) and derivatives (rambles). A written-out, step-by-step format would be needed for these; that belongs to post-training.

Against Zenyx-V3 (1.58B mixture-of-experts, about 80B tokens, little arithmetic data). Each model is scored on the same 14,000-problem suite in the prompt format that suited it best:

Model	Prompt format	Correct
Zenyx-100M-Base	its own chat format (above)	93.7%
Zenyx-V3	best of six formats tried (3 worked examples, X + Y = Z; step 110,000, 7,000 problems)	17.1%

V3 has no chat format, and its strict score depends strongly on the prompt format and varies between checkpoints.

MMLU and chat-format multiple choice

MMLU, standard 5-shot protocol (the answer letter chosen by probability), all 14,042 test questions, step 6,000: 25.5% ± 0.7 (95% interval; chance is 25%). Zenyx-V3 scores 26.5% (step 110,000; 26.9% at step 107,200). Both models are at chance level. The 151M model's answers lean on one letter (D for 33% of questions, A for 18%), which is a position bias and not knowledge. Prompts were the standard ones, except that 212 of the longest questions (1.5%) used 3–4 example questions instead of 5 so they fit the 2,048-token context; those scored 28.3%, within noise.

Chat format for multiple choice does not help (200 questions per task, step 6,000). With lettered options in the model's chat format, picking the letter is at chance; the standard scoring of the same questions is far better:

Task	Standard scoring	Chat format, letter chosen by probability	Chat format, answer generated
SciQ	74.5%	22.0%	5.5%
ARC-Easy	46.0%	25.0%	8.5%
ARC-Challenge	22.0%	24.0%	7.5%

Chance is 25%. The model has the knowledge (the standard scoring shows it) but not the "read four lettered options and pick one" skill. The headline numbers above therefore use the standard scoring for every model.

Code generation (small test)

Twelve short Python functions (primes, factorial, palindrome, gcd, Fibonacci and similar), each run against unit tests (step 6,000; 8 samples at temperature 0.6 per task):

Prompt style	Greedy passes	Sampled attempts that pass
Signature and docstring (plain completion)	4 / 12	25% (24 of 96)
Chat format, model starts freely	5 / 12	25% (24 of 96)
Chat format, reply forced to open with <think>	1 / 12	5% (5 of 96)

Plain completion and the free-start chat format are about equal. It writes correct is_prime, reverse_string, is_palindrome and celsius_to_f in plain completion, and fails on factorial, fibonacci, gcd and fizzbuzz. Forcing <think> makes it clearly worse: its thinking habit exists only for arithmetic, and with <think> forced it drifts into inventing arithmetic questions. Code reasoning would need post-training data.

Inference speed (one T4, fp16)

CUDA-graph decode step with torch.compile. Batch columns are totals across all streams; short context.

Batch	Eager	CUDA graph	Graph + compile
1	41 tok/s	186 tok/s	373 tok/s
8	330 tok/s	1,399 tok/s	2,668 tok/s
32	1,286 tok/s	4,475 tok/s	7,040 tok/s
64	2,541 tok/s	7,038 tok/s	9,821 tok/s

A full 2,048-token generation with sampling and a repetition penalty runs at 248–259 tokens/s.

Long generation

Decoding	Behaviour
Greedy, no repetition penalty	repeats itself within about 30 words
Greedy, penalty 1.2–1.3	loops delayed, or ends cleanly with the end-of-text token
Sampling (temperature 0.8, top-k 50), penalty 1.1–1.2	no repetition loops

A 2,048-token window holds about 1,000–1,900 words. Text is grammatical, but after roughly 100 words it drifts off topic and invents facts. Suggested settings for now: sampling with a repetition penalty of 1.1–1.2.

Comparison with Muse research models

Zenyx-100M-Base (step 6,000, 3.15B tokens) against the Muse-research models, run through the same harness: same Fact1000 items, same SciQ prompt format (support passage included), same Pile text, same fp16 inference.

Base models

Model	Parameters	Fact1000 (all)	Fact1000 (no arithmetic)	SciQ acc / acc_norm	ARC-Easy acc / acc_norm	ARC-Challenge acc / acc_norm	Pile bits per byte	Pile domain mean
Zenyx-100M-Base (step 6,000)	151.0M	58.5%	61.7%	72.4% / 67.3%	44.4% / 40.1%	19.7% / 23.5%	1.145	1.308
Muse2-230M-Base	196.1M	56.3%	61.7%	68.3% / 61.7%	44.3% / 39.8%	18.9% / 24.1%	1.193	1.387
Muse2-125M-Base	122.9M	51.9%	57.1%	67.3% / 58.7%	44.3% / 39.8%	17.3% / 22.6%	1.227	1.424

Reading the table:

• Fact1000: Zenyx-100M is 2.2 points ahead of Muse2-230M overall and 6.6 points ahead of Muse2-125M. The first gap is within the noise (about 1 standard error of the difference); the second is clear (about 3). Without the arithmetic items the two are level with Muse2-230M (61.7% each), so the overall lead over it comes from arithmetic.
• SciQ: Zenyx-100M leads on both metrics. The accuracy gap to Muse2-230M (4.1 points) is about 2 standard errors of the difference; the length-normalised gap (5.6 points) is about 2.7.
• ARC: level. On ARC-Easy all three models are within a point of each other on both metrics; on ARC-Challenge they are within noise (standard error about 1.1 points) and close to the 25% chance level.
• Pile bits per byte: Zenyx-100M is lowest, by 0.048 against Muse2-230M and 0.082 against Muse2-125M, on identical text.
• Size: Zenyx-100M has 23% fewer parameters than Muse2-230M, counted as loaded in our harness (151.0M against 196.1M).

Chat Muse models (reference only)

Muse-1B and Muse-3B are chat models, not next-token base models. They are scored here as raw-text language models without a chat template, which is the only protocol the shared harness supports; this penalises them and the comparison is not like-for-like. The numbers show how such models behave under a base-model protocol, not how good they are as assistants.

Model	Parameters	Fact1000 (all)	SciQ acc / acc_norm	ARC-Easy acc / acc_norm	ARC-Challenge acc / acc_norm	Pile bits per byte
Muse-3B (chat)	2,969.7M	38.7%	38.8% / 38.8%	34.4% / 32.0%	18.3% / 23.6%	1.689
Muse-1B (chat)	1,044.5M	38.0%	40.3% / 39.7%	30.4% / 30.8%	20.4% / 24.1%	1.961
Zenyx-100M-Base	151.0M	58.5%	72.4% / 67.3%	44.4% / 40.1%	19.7% / 23.5%	1.145

On ARC-Challenge the three models are indistinguishable (all near chance). The large gaps elsewhere should not be read as a capability ranking of the underlying models; scores this low for 1B–3B models suggest the setup, not only the models, is at fault, and this has not been investigated.

Caveats

• Muse2 numbers come from our own reimplementation. The Muse2 base models use a convolution-plus-attention hybrid for which we evaluated an unofficial reimplementation, choosing among architecture variants by loss on neutral text (selected variant: parallel blocks, BOS prepended, NeoX-style RoPE). A mismatch with the reference implementation would lower their scores, so treat their results as a likely lower bound.
• Not token-matched. Zenyx-100M has seen 3.15B tokens and is not yet in its decay phase. The Muse2 training budgets are not known to us, so this is a comparison of released checkpoints, not of training efficiency.
• Fact1000 is author-written. It is a frozen, hash-checked test, but it was written by the Zenyx author and its categories (capitals, elements, arithmetic and similar) overlap with what our data mixture emphasises. SciQ and the Pile are independent of us.
• SciQ includes the support passage in the prompt, which makes it easier than closed-book question answering.
• Single run, single checkpoint per model, no repeated seeds.

Comparison with Zenyx-V3

Zenyx-V3-Base is the author's larger model (1.58B-parameter mixture-of-experts, about 0.38B active per token, trained on about 80B tokens). Same harness and test items, step 110,000 for V3 and step 6,000 for this model:

Benchmark	Zenyx-V3	Zenyx-100M-Base
Fact1000, all items	79.8%	58.5%
Fact1000, without arithmetic	79.2%	61.7%
SciQ (acc / acc_norm)	79.4 / 70.6	72.4 / 67.3
ARC-Easy	52.0 / 46.6	44.4 / 40.1
ARC-Challenge	21.6 / 26.1	19.7 / 23.5
MMLU, 5-shot	26.5%	25.5%
Pile bits per byte (per-domain table above)	0.999	1.145
Arithmetic (14,000 problems, best prompt format for each)	17.1%	93.7%

V3 is clearly ahead on knowledge and language modelling, with about 26 times more training tokens and 10 times more parameters. The 151M model is ahead on arithmetic because arithmetic data is part of its training mixture. Neither is above chance on MMLU or ARC-Challenge.

Planned post-training

A supervised fine-tuning stage is being prepared; no instruct model has been released and no result exists yet. Design decisions so far:

• Chat template: ChatML with the tokenizer's real special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, <tool_response>), the same template the pre-training chat documents used. The loss is applied to assistant content and its <|im_end|> only.
• System versus user: varied real system prompts plus explicit scenarios in which the system prompt must win over conflicting user instructions (format rules, topic restrictions, confidential values), so the model learns to tell the two apart from the first turn.
• Reasoning effort: a Reasoning effort: none | low | medium | high line in the system prompt, with reasoning budgets of 256, 512 and 1,024 tokens for low, medium and high; the reasoning must finish inside its budget. Training data pairs the same arithmetic problems at all three verbosity levels (compact line; column-by-column working; working plus an inverse-operation check) and uses distilled reasoning traces that fit the budgets.
• Data: English, code and math only (a language filter removes other scripts and non-English prose), sources audited on real samples before inclusion, built shard by shard with exact resume.

Intended use and limitations

• Use: research, and as a starting point for fine-tuning.
• Not an assistant: it ignores system prompts and may continue a prompt with an unrelated document (post-training is planned, see above).
• Facts: expect errors and invented details; long generations repeat themselves (use sampling with a repetition penalty).
• Arithmetic: in its own chat format, 90–100% on addition up to 5 digits, 86–98% on subtraction up to 4 digits and 84% on 2×2 multiplication; no step-by-step working, and it fails on longer numbers, three-term expressions and word problems.
• Reasoning, MMLU, multiple choice by letter, code: at or near chance or weak (see Evaluation). The <think> habit exists only for arithmetic.
• Context: 2,048 tokens.
• Language: English only; no multilingual evaluation.
• Safety: not evaluated for bias or harmful content.
• Evaluation: single training run, mid-training checkpoints, some tests on small samples (the sample sizes are given where they apply). The full suite is from step 6,000; later checkpoints have only the reduced quick checks.

Repository contents

Path	Contents
checkpoints/<run>/step_N/	model weights (model.pt), optimiser state, training state. Only the two newest checkpoints are kept, so the step-6,000 weights behind the full benchmark tables are no longer available
tokenizer_candidates_v3/	tokenizers, including the one used
wheels/	flash-attention build used for training
runs/, data/	training manifests

The model code is not published yet.