Card updated: core benchmarks at step 16,100, wider suite and MMLU at step 12,650

README.md

@@ -16,7 +16,7 @@ tags:
 
 A **151M-parameter** dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).
 
-> **Status: pre-training in progress** (7.71B of a 50B-token budget at step 14,700; newer checkpoints may be in the repository). This is a **base model**: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate has not finished decaying). A post-training stage is planned ([below](#planned-post-training)).
+> **Status: pre-training in progress** (8.44B of a 50B-token budget at step 16,100; newer checkpoints may be in the repository). This is a **base model**: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate has not finished decaying). A post-training stage is planned ([below](#planned-post-training)).
 
 ## At a glance
 
@@ -26,8 +26,8 @@ A **151M-parameter** dense decoder-only language model trained from scratch on w
 | Architecture | 18-layer Llama-style decoder, grouped-query attention, SwiGLU, context 2,048 |
 | Vocabulary | 49,105 tokens (byte-level BPE) |
 | Training | 2 × Tesla T4, fp16; Muon + AdamW; 524,288 tokens per step |
-| Progress | step 14,700 = 7.71B tokens of 50B; learning rate at 69% of peak, decaying until 15B tokens |
-| Latest benchmarks | step 14,700 (core suite, below); step 12,650 (wider suite, MMLU); step 10,900 (code, speed, letter-choice); step 10,800 (wide arithmetic) |
+| Progress | step 16,100 = 8.44B tokens of 50B; learning rate at 63% of peak, decaying until 15B tokens |
+| Latest benchmarks | step 16,100 (core suite, below); step 12,650 (wider suite, MMLU); step 10,900 (code, speed, letter-choice); step 10,800 (wide arithmetic) |
 | License | Apache-2.0 |
 
 ## Results
@@ -36,24 +36,24 @@ Base model, no fine-tuning, 0-shot, all test items. Fact1000 is a 1,000-item fou
 
 ### Comparison at the latest checkpoints
 
-| | **Zenyx-100M** (step 14,700) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) |
+| | **Zenyx-100M** (step 16,100) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) |
 |---|---|---|---|---|---|
 | Parameters | 151.0M | 152.6M | 196.1M | 122.9M | 1.58B (0.38B active) |
-| Training tokens | 7.7B | about 15B | not published | not published | about 80B |
-| Fact1000 | 70.1% | 76.7% | 56.3% | 51.9% | **79.8%** |
-| Fact1000, without the arithmetic items | 67.2% | 78.3% | 61.7% | 57.1% | **79.2%** |
-| SciQ (acc / length-normalised) | 76.6 / 72.5 | **80.7 / 75.6** | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
-| ARC-Easy | 46.9 / 42.2 | **54.0 / 48.5** | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
-| ARC-Challenge | 21.4 / 23.1 | **25.5 / 27.2** | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
-| Pile bits per byte, 22 domains (lower is better) | 1.090 | 1.066 | 1.193 | 1.227 | **0.999** |
-| Pile, mean of domains | 1.248 | 1.232 | 1.387 | 1.424 | **1.159** |
-| Arithmetic, 14,000 problems, best prompt format | **96.5% (own chat format)** | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |
+| Training tokens | 8.4B | about 15B | not published | not published | about 80B |
+| Fact1000 | 64.6% | 76.7% | 56.3% | 51.9% | **79.8%** |
+| Fact1000, without the arithmetic items | 67.3% | 78.3% | 61.7% | 57.1% | **79.2%** |
+| SciQ (acc / length-normalised) | 73.6 / 66.9 | **80.7 / 75.6** | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
+| ARC-Easy | 44.1 / 40.4 | **54.0 / 48.5** | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
+| ARC-Challenge | 21.6 / 22.4 | **25.5 / 27.2** | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
+| Pile bits per byte, 22 domains (lower is better) | 1.084 | 1.066 | 1.193 | 1.227 | **0.999** |
+| Pile, mean of domains | 1.241 | 1.232 | 1.387 | 1.424 | **1.159** |
+| Arithmetic, 14,000 problems, best prompt format | **98.0% (own chat format)** | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |
 
 \* Muse2: our own reimplementation of the architecture (no public code), same harness and items, not token-matched. V3 is step 110,000 (arithmetic: step 110,800). BetterGPT is its released checkpoint with its tokenizer's BOS+EOS prefix (the choice moves its scores by about 1 point; the author-default BOS prefix gives Fact1000 75.7%).
 
-- **BetterGPT-150M leads on knowledge and language modelling** by roughly 3 to 11 points on the rows above, with about twice the training tokens (15B against 7.7B); Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ, ARC-Easy and the Pile and level on ARC-Challenge (every model in the table is close to the 25% chance level there).
+- **BetterGPT-150M leads on knowledge and language modelling** by roughly 4 to 12 points on the rows above, with about twice the training tokens (15B against 8.4B); Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ and the Pile and level on ARC-Easy and ARC-Challenge (ARC-Challenge is close to the 25% chance level for every model in the table).
 - **Zenyx-100M leads on arithmetic by a wide margin**: BetterGPT and V3 do 2-digit work and little more even with worked examples in the prompt, while Zenyx-100M was trained on arithmetic in its own chat format with a short scratchpad (see [Arithmetic](#arithmetic)).
-- **Zenyx-V3** (about 10 times the parameters, 10 times the tokens) is ahead of everything else on knowledge and the Pile.
+- **Zenyx-V3** (about 10 times the parameters, 9 times the tokens) is ahead of everything else on knowledge and the Pile.
 
 ### Wider suite (step 12,650)
 
@@ -72,36 +72,36 @@ Most of these are close to chance for models of this size (HellaSwag 25%, PIQA/W
 
 ### Pile bits per byte by domain
 
-Lower is better; bold = best of the four. Zenyx-100M step 14,700; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.
+Lower is better; bold = best of the four. Zenyx-100M step 16,100; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.
 
 | Domain | Zenyx-100M | BetterGPT-150M | Muse2-230M* | Zenyx-V3 |
 |---|---|---|---|---|
-| DM Mathematics | **1.324** | 1.348 | 1.723 | 1.326 |
-| Enron Emails | 1.287 | 1.249 | 1.416 | **1.222** |
-| Wikipedia (en) | 1.059 | 1.080 | 1.123 | **0.987** |
-| GitHub | 0.650 | 0.722 | 0.846 | **0.576** |
-| PhilPapers | 1.413 | 1.415 | 1.667 | **1.332** |
+| DM Mathematics | **1.313** | 1.348 | 1.723 | 1.326 |
+| GitHub | 0.628 | 0.722 | 0.846 | **0.576** |
+| Enron Emails | 1.285 | 1.249 | 1.416 | **1.222** |
+| YouTube Subtitles | 1.884 | 2.011 | 2.290 | **1.816** |
+| Wikipedia (en) | 1.060 | 1.080 | 1.123 | **0.987** |
+| PhilPapers | 1.410 | 1.415 | 1.667 | **1.332** |
+| StackExchange | 1.000 | 1.025 | 1.207 | **0.917** |
 | HackerNews | 1.362 | 1.305 | 1.448 | **1.278** |
-| PubMed Central | 0.825 | 0.797 | 0.922 | **0.741** |
-| Gutenberg (PG-19) | 1.355 | 1.298 | 1.430 | **1.268** |
-| Pile-CC | 1.186 | 1.124 | 1.199 | **1.098** |
-| BookCorpus2 | 1.234 | 1.155 | 1.205 | **1.142** |
-| ArXiv | 0.947 | 0.911 | 1.131 | **0.855** |
-| USPTO Backgrounds | 0.872 | 0.828 | 0.938 | **0.779** |
-| EuroParl | 2.058 | 2.155 | 2.444 | **1.964** |
-| StackExchange | 1.012 | 1.025 | 1.207 | **0.917** |
-| OpenSubtitles | 1.431 | 1.358 | 1.399 | **1.333** |
-| OpenWebText2 | 1.224 | 1.188 | 1.281 | **1.126** |
-| NIH ExPorter | 0.962 | 0.906 | 1.018 | **0.861** |
-| YouTube Subtitles | 1.917 | 2.011 | 2.290 | **1.816** |
-| PubMed Abstracts | 0.982 | 0.932 | 1.072 | **0.873** |
-| Ubuntu IRC | 1.828 | 1.852 | 2.121 | **1.714** |
-| Books3 | 1.446 | 1.398 | 1.495 | **1.330** |
-| FreeLaw | 1.083 | 1.040 | 1.129 | **0.958** |
-| **22 domains, byte-weighted** | 1.090 | 1.066 | 1.193 | **0.999** |
-| **Mean of domains** | 1.248 | 1.232 | 1.387 | **1.159** |
-
-Zenyx-100M beats BetterGPT in 8 of 22 domains (GitHub, YouTube Subtitles, EuroParl, Wikipedia (en), DM Mathematics, Ubuntu IRC, StackExchange, PhilPapers) and is 3 to 7% behind in the other 14 (overall 2.2% behind). It is ahead of Muse2-230M* in 20 of 22 domains.
+| PubMed Central | 0.826 | 0.797 | 0.922 | **0.741** |
+| Pile-CC | 1.184 | 1.124 | 1.199 | **1.098** |
+| ArXiv | 0.941 | 0.911 | 1.131 | **0.855** |
+| OpenSubtitles | 1.421 | 1.358 | 1.399 | **1.333** |
+| Gutenberg (PG-19) | 1.356 | 1.298 | 1.430 | **1.268** |
+| EuroParl | 2.053 | 2.155 | 2.444 | **1.964** |
+| BookCorpus2 | 1.232 | 1.155 | 1.205 | **1.142** |
+| USPTO Backgrounds | 0.869 | 0.828 | 0.938 | **0.779** |
+| OpenWebText2 | 1.219 | 1.188 | 1.281 | **1.126** |
+| Ubuntu IRC | 1.808 | 1.852 | 2.121 | **1.714** |
+| NIH ExPorter | 0.960 | 0.906 | 1.018 | **0.861** |
+| Books3 | 1.436 | 1.398 | 1.495 | **1.330** |
+| PubMed Abstracts | 0.980 | 0.932 | 1.072 | **0.873** |
+| FreeLaw | 1.082 | 1.040 | 1.129 | **0.958** |
+| **22 domains, byte-weighted** | 1.084 | 1.066 | 1.193 | **0.999** |
+| **Mean of domains** | 1.241 | 1.232 | 1.387 | **1.159** |
+
+Zenyx-100M beats BetterGPT in 8 of 22 domains (GitHub, YouTube Subtitles, EuroParl, DM Mathematics, StackExchange, Ubuntu IRC, Wikipedia (en), PhilPapers) and is 3 to 7% behind in the other 14 (overall 1.7% behind). It is ahead of Muse2-230M* in 20 of 22 domains.
 
 ### Over training
 
@@ -116,9 +116,10 @@ Zenyx-100M beats BetterGPT in 8 of 22 domains (GitHub, YouTube Subtitles, EuroPa
 | 10,800 | 5.66 | 68.1% (66.0%) | 75.2 / 68.5 | 45.4 / 40.7 | 1.111 | 89.4% |
 | 12,650 | 6.63 | 70.5% (68.0%) | 76.1 / 70.8 | 46.0 / 42.3 | 1.103 | 91.1% |
 | 14,500 | 7.60 | 68.9% (66.9%) | 75.7 / 71.6 | 47.3 / 43.9 | 1.095 | 85.5% |
-| **14,700** | **7.71** | **70.1% (67.2%)** | **76.6 / 72.5** | **46.9 / 42.2** | **1.090** | **96.5%** |
+| 14,700 | 7.71 | 70.1% (67.2%) | 76.6 / 72.5 | 46.9 / 42.2 | 1.090 | 96.5% |
+| **16,100** | **8.44** | **64.6% (67.3%)** | **73.6 / 66.9** | **44.1 / 40.4** | **1.084** | **98.0%** |
 
-Compared with step 6,000, Fact1000 without the arithmetic items rose from 61.7% to 67.2%, SciQ from 72.4 to 76.6 and ARC-Easy from 44.4 to 46.9 (accuracy), the Pile from 1.145 to 1.090; ARC-Challenge stays at about chance. Between the last three checkpoints (12,650, 14,500 and 14,700) knowledge scores moved by about two standard errors or less; Fact1000 changed by +2.0 points since step 10,800 (the standard error of such a difference is about 2). **Arithmetic oscillates between checkpoints** (72% to 96% over the run). 2-digit multiplication: 92% at step 9,500, 58% at 10,800, 95% at 12,650, 23% at 14,500 (the first scratchpad line was right 95% of the time, but 76% of the replies then added a spurious second step) and 87% at 14,700; its cause is not established.
+Compared with step 6,000, Fact1000 without the arithmetic items rose from 61.7% to 67.3%, SciQ from 72.4 to 73.6 and ARC-Easy from 44.4 to 44.1 (accuracy), the Pile from 1.145 to 1.084; ARC-Challenge stays at about chance. Since step 14,700 (one standard error is about 1.4 points on 1,000 items, 1.0 on ARC-Easy): Fact1000 without the arithmetic items +0.1, SciQ -3.0, ARC-Easy -2.8, ARC-Challenge +0.2, the Pile -0.0060 bits per byte; SciQ and ARC-Easy dipped by about two standard errors, which is not yet explained. The all-items Fact1000 score changed by -5.5 points since step 14,700 and by -3.5 since step 10,800 (the standard error of such a difference is about 2), while the score without the arithmetic items did not move: the change is in its 174 arithmetic items. **Arithmetic oscillates between checkpoints** (72% to 98% over the run). 2-digit multiplication: 92% at step 9,500, 58% at 10,800, 95% at 12,650, 23% at 14,500 (the first scratchpad line was right 95% of the time, but 76% of the replies then added a spurious second step), 87% at 14,700 and 95% at 16,100; its cause is not established.
 
 ### Arithmetic
 
@@ -135,7 +136,7 @@ What is 683 + 479?<|im_end|>
 1162<|im_end|>
 ```
 
-14,000 problems (7 types × 2,000, fixed seeds, exact match), step 14,700: 2-digit addition 100.0%, 2-digit subtraction 98.4%, 3-digit addition 100.0%, 3-digit subtraction 91.4%, 4-digit addition 99.5%, 5-digit addition 98.8%, 2-digit × 2-digit 87.4%; **all 96.5%** (13,509 of 14,000). Without the chat format the same model scores under 1%: this is a trained, narrow skill.
+14,000 problems (7 types × 2,000, fixed seeds, exact match), step 16,100: 2-digit addition 100.0%, 2-digit subtraction 99.7%, 3-digit addition 99.8%, 3-digit subtraction 94.0%, 4-digit addition 98.9%, 5-digit addition 98.6%, 2-digit × 2-digit 95.1%; **all 98.0%** (13,719 of 14,000). Without the chat format the same model scores under 1%: this is a trained, narrow skill.
 
 **Wide suite** (7,636 further problems in 28 types, exact match, step 10,800): **7.3%** overall (13.0% at step 9,500; not rerun at the latest step). Best types: negative numbers, mul / div 31%; powers 30%; decimals, add / sub 30%; negative numbers, add / sub 23%. Near zero: 6-8 digit numbers, fractions, GCD / LCM, series sums, averages, unit conversion, linear equations and word problems. Between steps 9,500 and 10,800, 3-digit × 1-digit multiplication fell from 94.1% to 22.7% and 3-digit × 2-digit from 61.2% to 0.0%: replies now often follow a correct first scratchpad line with a spurious second step (for example `691 * 8 = 5528 | 691 + 5528 = 6219`). This is the same chaining that caused the 2-digit multiplication dip above, and the failure seen at step 7,400 coming back.