Release 1gpu-llm-medium-v2 step 34200

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+1gpu-llm-official.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,154 @@
+---
+language: [en, it]
+license: cc-by-sa-4.0
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- epfml/FineWeb-HQ
+- epfml/FineWeb2-HQ
+- google/wiki40b
+tags:
+- 1gpu-llm
+- official-release
+- single-gpu
+- trained-from-scratch
+- gpt2preln
+- bilingual
+- english
+- italian
+- pretraining
+- base-model
+- causal-lm
+- llm-nanochat
+- medium
+- decay-only
+---
+
+![1gpu-llm official model](https://cdn-uploads.huggingface.co/production/uploads/61cf263eb7eeed127532a758/rYYwbJLnEkb8CRD_m6sWu.png)
+
+# 1gpu-llm-medium-v2
+
+Official medium bilingual base model of the `1gpu-llm` family, trained from
+scratch and continued on a single NVIDIA GeForce RTX 4060 Ti 16GB.
+
+This release is a base language model, not an instruction-tuned chat model.
+
+## Release summary
+
+- Released checkpoint: `step_34200`
+- Parent/reference checkpoint: `step_34000` from the main CPT run
+- Continuation: optimizer-preserving decay-only continuation, global steps
+  `34000 → 35700`; this release is the early scalar winner at `34200`
+- Architecture: GPT-2-style decoder-only Transformer with pre-layernorm blocks
+- Repo-native configuration: `architecture: gpt2`, `block_type: gpt2_prelayernorm`,
+  `norm_order: preln`
+- Parameters: `337,639,424` (`~337.6M`)
+- Context window and training sequence length: `2500` tokens
+- Languages: English and Italian
+
+## Why step_34200
+
+The checkpoint was selected by a controlled 1000-token decoding comparison of
+`step_34000`, `step_34200`, and `step_34800`, using the same tokenizer, seed,
+device, precision, prompts, and generation budget across four presets.
+
+Recommended pair:
+
+- checkpoint: `step_34200`
+- preset: `creative`
+- tuning score: `3.1683`
+- holdout score: `3.8405`
+- EOS termination: `100%` on tuning and holdout
+- truncation at 1000 tokens: `0%`
+- strict loop rate: `0%`
+- mean length: `410.6` tuning / `494.8` holdout tokens
+- distinct-2: `0.9531` tuning / `0.9536` holdout
+- language switches: `0%`
+
+The model remains imperfect: long completions can show factual or semantic
+drift. The conservative alternative is the parent `step_34000` with the
+`anti_loop` preset, which is shorter and more controlled.
+
+## Recommended generation
+
+The public default is the `creative` preset selected by the holdout:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+repo_id = "nazdef/1gpu-llm-medium-v2"
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForCausalLM.from_pretrained(repo_id)
+
+prompt = "La capitale d'Italia è"
+prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
+bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
+input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
+attention_mask = torch.ones_like(input_ids)
+
+with torch.no_grad():
+    outputs = model.generate(
+        input_ids=input_ids,
+        attention_mask=attention_mask,
+        do_sample=True,
+        max_new_tokens=1000,
+        temperature=1.0,
+        top_k=100,
+        top_p=0.95,
+        repetition_penalty=1.1,
+        no_repeat_ngram_size=0,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id,
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+For a more conservative response style, use `anti_loop`:
+
+- `temperature=0.8`
+- `top_k=50`
+- `top_p=0.9`
+- `repetition_penalty=1.15`
+- `no_repeat_ngram_size=4`
+
+## Training provenance
+
+The model was trained on a balanced English/Italian web + wiki corpus derived
+from:
+
+- English FineWeb-HQ (`epfml/FineWeb-HQ`)
+- Italian FineWeb2-HQ (`epfml/FineWeb2-HQ`)
+- English and Italian Wiki40B (`google/wiki40b`)
+
+The local packed dataset used a 2500-token sequence length and a 50/50 EN/IT
+source-balanced mix. The continuation loaded the model and AdamW optimizer
+state from the `step_34000` checkpoint, deliberately ignored the saved scheduler
+state, and created a local `wsd-decay-only` scheduler with no warmup or rewarm.
+
+Included provenance files:
+
+- `training_config.yaml`
+- `decoding_grid_config.yaml`
+- `decoding_grid_report.md`
+- `export_manifest.json`
+- `step_34200.safetensors.json`
+
+## License
+
+The model card uses `CC BY-SA 4.0` as the release license. Training data comes
+from mixed upstream sources with their own terms, including FineWeb/FineWeb2
+and Wiki40B. Downstream users are responsible for checking the applicable
+upstream dataset terms, attribution requirements, share-alike obligations, and
+any restrictions connected to the data or generated outputs in their intended
+use case.
+
+## Limitations
+
+- This is a pretrained base model; it is not instruction-following aligned.
+- It may hallucinate facts and drift semantically during long generations.
+- English/Italian language consistency is strong in the selected decoding
+  holdout, but it is not a guarantee for arbitrary prompts.
+- The public checkpoint is an experimental single-GPU family release, not a
+  claim of state-of-the-art performance.

config.json

@@ -0,0 +1,26 @@
+{
+  "activation_function": "gelu",
+  "architecture": "gpt2",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "block_type": "gpt2_prelayernorm",
+  "causal_mask_mode": "buffered_upper_triangular",
+  "embd_pdrop": 0.0,
+  "init_strategy": "gpt2_std_0.02_residual_scale",
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 2500,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_layer": 24,
+  "n_positions": 2500,
+  "norm_order": "preln",
+  "norm_type": "layernorm",
+  "positional_encoding": "learned_absolute",
+  "resid_pdrop": 0.0,
+  "tie_word_embeddings": true,
+  "use_cache": true,
+  "vocab_size": 32000
+}

decoding_grid_config.yaml

@@ -0,0 +1,47 @@
+name: medium_checkpoint_decoding_grid_1000
+checkpoint_path: null
+tokenizer_dir: null
+tuning_prompts_path: eval_prompts/decoding_tuning.jsonl
+holdout_prompts_path: eval_prompts/decoding_tuning_holdout.jsonl
+seeds:
+  - 1337
+holdout_top_k: 4
+target_length_ratio: 1.0
+ranking_weights:
+  prompt_pass_rate: 2.0
+  completion_rate: 0.5
+  distinct_2: 1.2
+  language_consistency: 0.75
+  length_closeness: 1.0
+  loop_rate: -2.0
+  repeated_4gram_rate: -1.5
+  language_switch_rate: -0.75
+decoding_presets:
+  - name: anti_loop_conservative
+    max_new_tokens: 1000
+    temperature: 0.3
+    top_k: 50
+    top_p: 0.9
+    repetition_penalty: 1.15
+    no_repeat_ngram_size: 4
+  - name: anti_loop
+    max_new_tokens: 1000
+    temperature: 0.8
+    top_k: 50
+    top_p: 0.9
+    repetition_penalty: 1.15
+    no_repeat_ngram_size: 4
+  - name: balanced
+    max_new_tokens: 1000
+    temperature: 0.8
+    top_k: 50
+    top_p: 0.95
+    repetition_penalty: 1.1
+    no_repeat_ngram_size: 0
+  - name: creative
+    max_new_tokens: 1000
+    temperature: 1.0
+    top_k: 100
+    top_p: 0.95
+    repetition_penalty: 1.1
+    no_repeat_ngram_size: 0

decoding_grid_report.md

@@ -0,0 +1,219 @@
+# Comparative 1000-token decoding grid — medium checkpoint selection
+
+## Executive verdict
+
+Recommended candidate for `1gpu-llm-medium`:
+
+- checkpoint: `step_34200`
+- preset: `creative`
+- tuning score: `3.1683`
+- holdout score: `3.8405`
+- EOS termination: `100%` on tuning and holdout
+- truncation at 1000 tokens: `0%`
+- loop rate: `0%` on both splits
+- repeated 4-gram rate: `0%` tuning, `25%` holdout
+- mean generated length: `410.6` tuning / `494.8` holdout tokens
+- median generated length: `392` tuning / `582.5` holdout tokens
+- distinct-2: `0.9531` tuning / `0.9536` holdout
+- language switches: `0%` on both splits
+
+This is the strongest overall combination because it is the only checkpoint/preset
+pair that combines the scalar champion checkpoint, a tuning winner confirmed by the
+holdout, long completions, zero truncation, zero loops, high diversity, and no
+language switching.
+
+Conservative alternative:
+
+- checkpoint: `step_34000`
+- preset: `anti_loop`
+- tuning score: `3.4540`
+- holdout score: `3.6036`
+- EOS termination: `100%` on both splits
+- truncation: `0%`
+- loop rate: `0%` on both splits
+- repeated 4-gram rate: `0%` tuning, `25%` holdout
+- mean length: `183.0` tuning / `265.5` holdout
+
+It is cleaner against repetition but produces shorter, more conservative answers
+and remains behind `step_34200 + creative` on the holdout score.
+
+The behavior champion `step_34800` is not promoted. Its tuning winner is `creative`,
+but holdout selects `balanced`; balanced is very short (`141.8` tokens mean) and
+therefore is not rewarded as a final choice merely for stopping early. `step_34800`
+does not provide a robust behavior advantage over `step_34200` under this 1000-token
+comparison.
+
+## Experimental controls
+
+- Checkpoints: `step_34000`, `step_34200`, `step_34800`
+- Presets: `anti_loop_conservative`, `anti_loop`, `balanced`, `creative`
+- `max_new_tokens`: `1000` for every preset
+- Tuning prompts: 7
+- Holdout prompts: 4, disjoint from tuning
+- Seed: `1337`
+- Tokenizer: `/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M`
+- Device/dtype: CUDA / `bf16`
+- Generation count: 7 per preset on tuning and 4 per preset on holdout, one seed
+- Early stopping: only the model EOS path; no artificial stop was introduced
+- Holdout coverage: all four presets were evaluated, not only the tuning top-k
+
+The repo runner does not serialize an explicit `terminated_with_eos` boolean. Since
+the generator has only two exits — EOS or the `max_new_tokens` loop limit — this
+report derives EOS/truncation as follows:
+
+- `num_generated_tokens < 1000`: EOS termination
+- `num_generated_tokens == 1000`: truncation at the configured limit
+
+## Complete tuning table
+
+`rep` is repeated-4gram rate; `loop` is the stricter repeated-4gram loop rate.
+Higher EOS, distinct-1/2 and language consistency are better; lower rep/loop and
+switch rates are better.
+
+| checkpoint | preset | score | EOS | trunc. | mean | median | distinct-1 | distinct-2 | rep | loop | lang. consistency |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 34000 | anti_loop_conservative | 2.9462 | 71.4% | 28.6% | 351.0 | 125.0 | 0.4763 | 0.8471 | 42.9% | 0% | 0.9848 |
+| 34000 | anti_loop | **3.4540** | 100% | 0% | 183.0 | 183.0 | 0.5750 | 0.9567 | 0% | 0% | 0.9262 |
+| 34000 | balanced | 2.0965 | 100% | 0% | 312.1 | 256.0 | 0.4833 | 0.8582 | 71.4% | 14.3% | 0.9286 |
+| 34000 | creative | 2.3089 | 85.7% | 14.3% | 299.4 | 220.0 | 0.5312 | 0.9237 | 28.6% | 14.3% | 0.9203 |
+| 34200 | anti_loop_conservative | 3.0936 | 85.7% | 14.3% | 265.9 | 66.0 | 0.4779 | 0.8545 | 14.3% | 0% | 0.7850 |
+| 34200 | anti_loop | 3.0177 | 100% | 0% | 268.0 | 243.0 | 0.5552 | 0.9439 | 0% | 0% | 0.9259 |
+| 34200 | balanced | 2.5360 | 100% | 0% | 202.7 | 146.0 | 0.5922 | 0.9508 | 28.6% | 0% | 0.9286 |
+| 34200 | creative | **3.1683** | 100% | 0% | 410.6 | 392.0 | 0.5119 | 0.9531 | 0% | 0% | 0.9339 |
+| 34800 | anti_loop_conservative | 2.5619 | 85.7% | 14.3% | 279.3 | 130.0 | 0.5378 | 0.7822 | 28.6% | 14.3% | 0.8571 |
+| 34800 | anti_loop | 2.5392 | 85.7% | 14.3% | 258.9 | 133.0 | 0.5580 | 0.9134 | 28.6% | 0% | 0.9259 |
+| 34800 | balanced | 2.6047 | 100% | 0% | 98.1 | 57.0 | 0.5764 | 0.9290 | 14.3% | 0% | 0.9286 |
+| 34800 | creative | **2.8813** | 100% | 0% | 323.4 | 95.0 | 0.5315 | **0.9591** | 14.3% | 0% | 0.9263 |
+
+Tuning winners:
+
+- `step_34000`: `anti_loop`
+- `step_34200`: `creative`
+- `step_34800`: `creative`
+
+## Complete holdout table
+
+| checkpoint | preset | score | EOS | trunc. | mean | median | distinct-1 | distinct-2 | rep | loop | lang. consistency |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 34000 | anti_loop_conservative | 3.4012 | 100% | 0% | 129.5 | 110.5 | 0.6005 | 0.9024 | 25% | 0% | 1.0000 |
+| 34000 | anti_loop | **3.6036** | 100% | 0% | 265.5 | 199.5 | 0.6262 | 0.9524 | 25% | 0% | 1.0000 |
+| 34000 | balanced | 3.1137 | 100% | 0% | 205.0 | 195.0 | 0.5928 | 0.9180 | 50% | 0% | 0.9886 |
+| 34000 | creative | 3.3935 | 100% | 0% | 322.0 | 358.0 | 0.6184 | **0.9644** | 0% | 0% | 1.0000 |
+| 34200 | anti_loop_conservative | 3.0267 | 100% | 0% | 142.0 | 127.5 | 0.6136 | 0.8899 | 50% | 0% | 1.0000 |
+| 34200 | anti_loop | 3.5983 | 100% | 0% | 285.0 | 351.5 | 0.6158 | 0.9389 | 25% | 0% | 0.9931 |
+| 34200 | balanced | 2.5511 | 100% | 0% | 157.5 | 112.0 | **0.6856** | 0.9550 | 0% | 0% | 1.0000 |
+| 34200 | creative | **3.8405** | 100% | 0% | **494.8** | **582.5** | 0.5777 | 0.9536 | 25% | 0% | 1.0000 |
+| 34800 | anti_loop_conservative | 3.5655 | 75% | 25% | 348.0 | 192.0 | 0.5101 | 0.8564 | 25% | 0% | 0.9975 |
+| 34800 | anti_loop | 3.5889 | 100% | 0% | 224.0 | 184.0 | 0.6499 | 0.9696 | 25% | 0% | 1.0000 |
+| 34800 | balanced | **3.8743** | 100% | 0% | 141.8 | 149.5 | 0.6645 | 0.9621 | 0% | 0% | 1.0000 |
+| 34800 | creative | 2.8716 | 100% | 0% | 179.0 | 159.0 | **0.6790** | 0.9591 | 25% | 0% | 1.0000 |
+
+Holdout winners by checkpoint:
+
+- `step_34000`: `anti_loop`
+- `step_34200`: `creative`
+- `step_34800`: `balanced`
+
+The `step_34200` tuning winner is confirmed by holdout. The `step_34800` tuning
+winner is not confirmed: holdout prefers `balanced`, but that preset terminates at
+only 141.8 tokens on average. That shortness is treated as a weakness, not a bonus.
+
+## First degeneration / loop analysis
+
+The first-loop metric is conservative: it reports the first generated word position
+where a 4-gram has appeared three times. When no such event occurs, the value is
+`none`; first repeated-4gram position is also tracked separately.
+
+### Tuning observations
+
+- `step_34000 + anti_loop`: no repeated 4-gram and no loop across all 7 samples.
+- `step_34200 + creative`: no repeated 4-gram and no loop across all 7 samples.
+- `step_34800 + creative`: one repeated 4-gram sample, but no strict loop.
+- `step_34000 + balanced`: first strict loop at token 41 in one sample; repeated-4gram rate 71.4%.
+- `step_34000 + creative`: first strict loop at mean token 142 in one sample.
+- `step_34800 + anti_loop_conservative`: one strict loop, first occurring at token 248.
+
+### Holdout observations
+
+No holdout configuration produced a strict loop (`loop_rate=0%`). Repeated 4-grams
+remain in several cases, especially `step_34200 + creative` and the anti-loop
+variants, but they occur without reaching the stricter three-repetition threshold.
+
+## Qualitative inspection
+
+Representative long generations were inspected for factuality, relevance, language
+stability and late degeneration.
+
+- `step_34000 + anti_loop` is the most controlled: it reaches EOS consistently and
+  avoids strict loops, but often produces conservative encyclopedic continuations
+  with factual drift (for example, Paris geography and historical details).
+- `step_34200 + creative` produces the longest useful continuations. It stays in the
+  requested language and avoids strict loops, but factual/semantic drift appears in
+  long completions. The issue is quality drift, not a decoding collapse.
+- `step_34800 + creative` is lively and diverse, but is less stable as a checkpoint
+  choice: tuning and holdout select different presets, and the median tuning length
+  is only 95 tokens despite a 1000-token allowance.
+- `balanced` on `step_34800` wins the raw holdout score mainly with short, clean
+  completions. It is rejected as the final preset because early EOS must not be
+  mistaken for superior long-form behavior.
+
+No preset systematically truncates on the selected final pair `step_34200 + creative`.
+The main remaining limitation is factual/semantic degradation in long text, not
+EOS handling, language switching or strict repetition loops.
+
+## Final operational recommendation
+
+### Definitive candidate
+
+Keep and use:
+
+```text
+checkpoint: /mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt
+preset: creative
+```
+
+Reason: tuning winner confirmed by holdout, zero truncation, zero strict loops,
+longest useful completions, high distinct-2, and stable EN/IT language behavior.
+
+### Behavior-oriented alternative
+
+Keep as a conservative fallback:
+
+```text
+checkpoint: /mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt
+preset: anti_loop
+```
+
+This pair is the most robust against repetition and is confirmed by holdout, at the
+cost of shorter and less expressive completions.
+
+### Retention and deletion candidates
+
+Retain now:
+
+1. `step_34200.pt` — definitive candidate with `creative`
+2. `step_34000.pt` — parent/reference and conservative fallback with `anti_loop`
+3. `step_34800.pt` — behavior experiment retained until the final release decision
+
+Potentially eliminable only after explicit confirmation:
+
+- `step_34800.pt`, if the project keeps only the definitive candidate plus parent
+  reference.
+
+No checkpoint was deleted by this operation. The other 34k-tail checkpoints were
+not part of this three-candidate comparison and are outside this cleanup decision.
+
+## Raw artifacts
+
+- Config:
+  `configs/eval/20260716_medium_checkpoint_decoding_grid_1000.yaml`
+- Output root:
+  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000`
+- Parent output:
+  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000/parent_step34000`
+- Scalar output:
+  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000/scalar_step34200`
+- Behavior output:
+  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000/behavior_step34800`
+- Launch log:
+  `/mnt/apps/llm-nanochat/launch_logs/20260716_135423_decoding_grid_medium_three_checkpoints_1000.log`

export_command.json

@@ -0,0 +1,519 @@
+{
+  "checkpoint_config": {
+    "activation": "gelu",
+    "actual_precision": "bf16",
+    "adamw_betas": [
+      0.9,
+      0.95
+    ],
+    "adamw_eps": 1e-08,
+    "architecture": "gpt2",
+    "attention_kernel_policy": "auto",
+    "batch_size": 2,
+    "benchmark": {
+      "enable_central_tensorboard": true,
+      "enable_local_tensorboard": true,
+      "enabled": false,
+      "output_path": "/mnt/apps/llm-nanochat/artifacts/runs/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/throughput_benchmark.json",
+      "warmup_steps": 0
+    },
+    "bias": true,
+    "block_type": "gpt2_prelayernorm",
+    "causal_mask_mode": "buffered_upper_triangular",
+    "checkpoint_dir": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki",
+    "clip_grad_norm": 1.0,
+    "compile": {
+      "backend": null,
+      "compile_setup_sec": 0.0,
+      "diagnostic": null,
+      "dynamic": false,
+      "enabled": false,
+      "error_policy": "raise",
+      "fullgraph": false,
+      "mode": null,
+      "requested": false,
+      "status": "disabled"
+    },
+    "dataset": {
+      "storage_mode": "indexed_jsonl"
+    },
+    "dataset_dir": "/mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M",
+    "decay_shape": "inverse_proportional",
+    "decay_steps": 1700,
+    "decay_steps_source": "explicit",
+    "deterministic_algorithms": false,
+    "device": "cuda",
+    "dim": 1024,
+    "dropout": 0.0,
+    "effective_global_step": 34200,
+    "effective_layer_count": 24,
+    "effective_learning_rate": 3.309994670030697e-05,
+    "final_lr": 1e-05,
+    "final_lr_source": "explicit",
+    "fp8_backend": null,
+    "grad_accum_steps": 48,
+    "init_from": "/mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt",
+    "init_strategy": "gpt2_std_0.02_residual_scale",
+    "learning_rate": 4.7832243797285384e-05,
+    "local_max_steps": 35700,
+    "local_step": 200,
+    "logging": {
+      "enable_central_tensorboard": true,
+      "enable_local_tensorboard": true,
+      "metrics_flush_every_steps": 1,
+      "metrics_writer": "persistent_jsonl_handle"
+    },
+    "lr": 4.7832243797285384e-05,
+    "lr_schedule": "wsd-decay-only",
+    "max_seq_len": 2500,
+    "max_steps": 35700,
+    "n_heads": 16,
+    "n_layers": 24,
+    "norm_order": "preln",
+    "norm_type": "layernorm",
+    "optimizer": {
+      "backend": "torch",
+      "betas": [
+        0.9,
+        0.95
+      ],
+      "eps": 1e-08,
+      "implementation": "torch.optim.AdamW",
+      "learning_rate": 4.7832243797285384e-05,
+      "state_precision": "full_precision",
+      "type": "adamw",
+      "weight_decay": 0.1
+    },
+    "optimizer_backend": "torch",
+    "optimizer_implementation": "torch.optim.AdamW",
+    "optimizer_reset": false,
+    "optimizer_state_precision": "full_precision",
+    "optimizer_type": "adamw",
+    "peak_lr": 4.7832243797285384e-05,
+    "positional_encoding": "learned_absolute",
+    "repro": {
+      "attention_kernel_policy": "auto",
+      "cublas_workspace_config": null,
+      "cudnn_benchmark": true,
+      "cudnn_deterministic": false,
+      "deterministic_algorithms": false,
+      "flash_sdp_enabled": true,
+      "math_sdp_enabled": true,
+      "mem_efficient_sdp_enabled": true,
+      "pythonhashseed": "1337",
+      "seed": 1337
+    },
+    "requested_precision": "bf16",
+    "resume_checkpoint_step": 34000,
+    "resume_from": "/mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt",
+    "resume_mode": "optimizer_only",
+    "resume_step": 34000,
+    "save_every_steps": 100,
+    "save_final_checkpoint": true,
+    "schedule_total_steps": 35700,
+    "scheduler": {
+      "decay_shape": "inverse_proportional",
+      "decay_steps": 1700,
+      "decay_steps_source": "explicit",
+      "final_lr": 1e-05,
+      "final_lr_source": "explicit",
+      "peak_lr": 4.7832243797285384e-05,
+      "resume_step": 34000,
+      "schedule_type": "wsd-decay-only",
+      "stable_steps": 0,
+      "target_final_step": 35700,
+      "total_steps": 35700,
+      "warmup_steps": 0
+    },
+    "scheduler_reset": true,
+    "seed": 1337,
+    "source_global_step": 34000,
+    "stable_steps": 0,
+    "target_final_step": 35700,
+    "target_global_step": 35700,
+    "tie_word_embeddings": true,
+    "train_cache_ram_bytes": 1073741824,
+    "train_cache_ram_mb": 1024,
+    "vocab_size": 32000,
+    "warmup_steps": 0,
+    "weight_decay": 0.1
+  },
+  "checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt",
+  "config_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/config.json",
+  "exported_at": "2026-07-16T13:41:05.693620+00:00",
+  "format": "llm-nanochat-safetensors-export",
+  "global_step": 34200,
+  "metadata_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/step_34200.safetensors.json",
+  "model_config": {
+    "activation": "gelu",
+    "architecture": "gpt2",
+    "bias": true,
+    "block_type": "gpt2_prelayernorm",
+    "causal_mask_mode": "buffered_upper_triangular",
+    "dim": 1024,
+    "dropout": 0.0,
+    "init_strategy": "gpt2_std_0.02_residual_scale",
+    "max_seq_len": 2500,
+    "n_heads": 16,
+    "n_layers": 24,
+    "norm_order": "preln",
+    "norm_type": "layernorm",
+    "positional_encoding": "learned_absolute",
+    "tie_word_embeddings": true,
+    "vocab_size": 32000
+  },
+  "num_parameters": 370439424,
+  "num_tensors": 294,
+  "provenance": {
+    "checkpoint_dir": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki",
+    "checkpoint_name": "step_34200.pt",
+    "checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt",
+    "global_step": 34200,
+    "packed_dataset_config_path": null,
+    "run_dir": "/mnt/apps/llm-nanochat/checkpoints",
+    "tokenizer_dir": "/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M",
+    "training_config_path": null
+  },
+  "safetensors_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/step_34200.safetensors",
+  "source_checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt",
+  "source_global_step": 34200,
+  "tensor_names": [
+    "token_emb.weight",
+    "pos_emb.weight",
+    "blocks.0.ln_1.weight",
+    "blocks.0.ln_1.bias",
+    "blocks.0.ln_2.weight",
+    "blocks.0.ln_2.bias",
+    "blocks.0.attn.qkv.weight",
+    "blocks.0.attn.qkv.bias",
+    "blocks.0.attn.out_proj.weight",
+    "blocks.0.attn.out_proj.bias",
+    "blocks.0.mlp.fc.weight",
+    "blocks.0.mlp.fc.bias",
+    "blocks.0.mlp.proj.weight",
+    "blocks.0.mlp.proj.bias",
+    "blocks.1.ln_1.weight",
+    "blocks.1.ln_1.bias",
+    "blocks.1.ln_2.weight",
+    "blocks.1.ln_2.bias",
+    "blocks.1.attn.qkv.weight",
+    "blocks.1.attn.qkv.bias",
+    "blocks.1.attn.out_proj.weight",
+    "blocks.1.attn.out_proj.bias",
+    "blocks.1.mlp.fc.weight",
+    "blocks.1.mlp.fc.bias",
+    "blocks.1.mlp.proj.weight",
+    "blocks.1.mlp.proj.bias",
+    "blocks.2.ln_1.weight",
+    "blocks.2.ln_1.bias",
+    "blocks.2.ln_2.weight",
+    "blocks.2.ln_2.bias",
+    "blocks.2.attn.qkv.weight",
+    "blocks.2.attn.qkv.bias",
+    "blocks.2.attn.out_proj.weight",
+    "blocks.2.attn.out_proj.bias",
+    "blocks.2.mlp.fc.weight",
+    "blocks.2.mlp.fc.bias",
+    "blocks.2.mlp.proj.weight",
+    "blocks.2.mlp.proj.bias",
+    "blocks.3.ln_1.weight",
+    "blocks.3.ln_1.bias",
+    "blocks.3.ln_2.weight",
+    "blocks.3.ln_2.bias",
+    "blocks.3.attn.qkv.weight",
+    "blocks.3.attn.qkv.bias",
+    "blocks.3.attn.out_proj.weight",
+    "blocks.3.attn.out_proj.bias",
+    "blocks.3.mlp.fc.weight",
+    "blocks.3.mlp.fc.bias",
+    "blocks.3.mlp.proj.weight",
+    "blocks.3.mlp.proj.bias",
+    "blocks.4.ln_1.weight",
+    "blocks.4.ln_1.bias",
+    "blocks.4.ln_2.weight",
+    "blocks.4.ln_2.bias",
+    "blocks.4.attn.qkv.weight",
+    "blocks.4.attn.qkv.bias",
+    "blocks.4.attn.out_proj.weight",
+    "blocks.4.attn.out_proj.bias",
+    "blocks.4.mlp.fc.weight",
+    "blocks.4.mlp.fc.bias",
+    "blocks.4.mlp.proj.weight",
+    "blocks.4.mlp.proj.bias",
+    "blocks.5.ln_1.weight",
+    "blocks.5.ln_1.bias",
+    "blocks.5.ln_2.weight",
+    "blocks.5.ln_2.bias",
+    "blocks.5.attn.qkv.weight",
+    "blocks.5.attn.qkv.bias",
+    "blocks.5.attn.out_proj.weight",
+    "blocks.5.attn.out_proj.bias",
+    "blocks.5.mlp.fc.weight",
+    "blocks.5.mlp.fc.bias",
+    "blocks.5.mlp.proj.weight",
+    "blocks.5.mlp.proj.bias",
+    "blocks.6.ln_1.weight",
+    "blocks.6.ln_1.bias",
+    "blocks.6.ln_2.weight",
+    "blocks.6.ln_2.bias",
+    "blocks.6.attn.qkv.weight",
+    "blocks.6.attn.qkv.bias",
+    "blocks.6.attn.out_proj.weight",
+    "blocks.6.attn.out_proj.bias",
+    "blocks.6.mlp.fc.weight",
+    "blocks.6.mlp.fc.bias",
+    "blocks.6.mlp.proj.weight",
+    "blocks.6.mlp.proj.bias",
+    "blocks.7.ln_1.weight",
+    "blocks.7.ln_1.bias",
+    "blocks.7.ln_2.weight",
+    "blocks.7.ln_2.bias",
+    "blocks.7.attn.qkv.weight",
+    "blocks.7.attn.qkv.bias",
+    "blocks.7.attn.out_proj.weight",
+    "blocks.7.attn.out_proj.bias",
+    "blocks.7.mlp.fc.weight",
+    "blocks.7.mlp.fc.bias",
+    "blocks.7.mlp.proj.weight",
+    "blocks.7.mlp.proj.bias",
+    "blocks.8.ln_1.weight",
+    "blocks.8.ln_1.bias",
+    "blocks.8.ln_2.weight",
+    "blocks.8.ln_2.bias",
+    "blocks.8.attn.qkv.weight",
+    "blocks.8.attn.qkv.bias",
+    "blocks.8.attn.out_proj.weight",
+    "blocks.8.attn.out_proj.bias",
+    "blocks.8.mlp.fc.weight",
+    "blocks.8.mlp.fc.bias",
+    "blocks.8.mlp.proj.weight",
+    "blocks.8.mlp.proj.bias",
+    "blocks.9.ln_1.weight",
+    "blocks.9.ln_1.bias",
+    "blocks.9.ln_2.weight",
+    "blocks.9.ln_2.bias",
+    "blocks.9.attn.qkv.weight",
+    "blocks.9.attn.qkv.bias",
+    "blocks.9.attn.out_proj.weight",
+    "blocks.9.attn.out_proj.bias",
+    "blocks.9.mlp.fc.weight",
+    "blocks.9.mlp.fc.bias",
+    "blocks.9.mlp.proj.weight",
+    "blocks.9.mlp.proj.bias",
+    "blocks.10.ln_1.weight",
+    "blocks.10.ln_1.bias",
+    "blocks.10.ln_2.weight",
+    "blocks.10.ln_2.bias",
+    "blocks.10.attn.qkv.weight",
+    "blocks.10.attn.qkv.bias",
+    "blocks.10.attn.out_proj.weight",
+    "blocks.10.attn.out_proj.bias",
+    "blocks.10.mlp.fc.weight",
+    "blocks.10.mlp.fc.bias",
+    "blocks.10.mlp.proj.weight",
+    "blocks.10.mlp.proj.bias",
+    "blocks.11.ln_1.weight",
+    "blocks.11.ln_1.bias",
+    "blocks.11.ln_2.weight",
+    "blocks.11.ln_2.bias",
+    "blocks.11.attn.qkv.weight",
+    "blocks.11.attn.qkv.bias",
+    "blocks.11.attn.out_proj.weight",
+    "blocks.11.attn.out_proj.bias",
+    "blocks.11.mlp.fc.weight",
+    "blocks.11.mlp.fc.bias",
+    "blocks.11.mlp.proj.weight",
+    "blocks.11.mlp.proj.bias",
+    "blocks.12.ln_1.weight",
+    "blocks.12.ln_1.bias",
+    "blocks.12.ln_2.weight",
+    "blocks.12.ln_2.bias",
+    "blocks.12.attn.qkv.weight",
+    "blocks.12.attn.qkv.bias",
+    "blocks.12.attn.out_proj.weight",
+    "blocks.12.attn.out_proj.bias",
+    "blocks.12.mlp.fc.weight",
+    "blocks.12.mlp.fc.bias",
+    "blocks.12.mlp.proj.weight",
+    "blocks.12.mlp.proj.bias",
+    "blocks.13.ln_1.weight",
+    "blocks.13.ln_1.bias",
+    "blocks.13.ln_2.weight",
+    "blocks.13.ln_2.bias",
+    "blocks.13.attn.qkv.weight",
+    "blocks.13.attn.qkv.bias",
+    "blocks.13.attn.out_proj.weight",
+    "blocks.13.attn.out_proj.bias",
+    "blocks.13.mlp.fc.weight",
+    "blocks.13.mlp.fc.bias",
+    "blocks.13.mlp.proj.weight",
+    "blocks.13.mlp.proj.bias",
+    "blocks.14.ln_1.weight",
+    "blocks.14.ln_1.bias",
+    "blocks.14.ln_2.weight",
+    "blocks.14.ln_2.bias",
+    "blocks.14.attn.qkv.weight",
+    "blocks.14.attn.qkv.bias",
+    "blocks.14.attn.out_proj.weight",
+    "blocks.14.attn.out_proj.bias",
+    "blocks.14.mlp.fc.weight",
+    "blocks.14.mlp.fc.bias",
+    "blocks.14.mlp.proj.weight",
+    "blocks.14.mlp.proj.bias",
+    "blocks.15.ln_1.weight",
+    "blocks.15.ln_1.bias",
+    "blocks.15.ln_2.weight",
+    "blocks.15.ln_2.bias",
+    "blocks.15.attn.qkv.weight",
+    "blocks.15.attn.qkv.bias",
+    "blocks.15.attn.out_proj.weight",
+    "blocks.15.attn.out_proj.bias",
+    "blocks.15.mlp.fc.weight",
+    "blocks.15.mlp.fc.bias",
+    "blocks.15.mlp.proj.weight",
+    "blocks.15.mlp.proj.bias",
+    "blocks.16.ln_1.weight",
+    "blocks.16.ln_1.bias",
+    "blocks.16.ln_2.weight",
+    "blocks.16.ln_2.bias",
+    "blocks.16.attn.qkv.weight",
+    "blocks.16.attn.qkv.bias",
+    "blocks.16.attn.out_proj.weight",
+    "blocks.16.attn.out_proj.bias",
+    "blocks.16.mlp.fc.weight",
+    "blocks.16.mlp.fc.bias",
+    "blocks.16.mlp.proj.weight",
+    "blocks.16.mlp.proj.bias",
+    "blocks.17.ln_1.weight",
+    "blocks.17.ln_1.bias",
+    "blocks.17.ln_2.weight",
+    "blocks.17.ln_2.bias",
+    "blocks.17.attn.qkv.weight",
+    "blocks.17.attn.qkv.bias",
+    "blocks.17.attn.out_proj.weight",
+    "blocks.17.attn.out_proj.bias",
+    "blocks.17.mlp.fc.weight",
+    "blocks.17.mlp.fc.bias",
+    "blocks.17.mlp.proj.weight",
+    "blocks.17.mlp.proj.bias",
+    "blocks.18.ln_1.weight",
+    "blocks.18.ln_1.bias",
+    "blocks.18.ln_2.weight",
+    "blocks.18.ln_2.bias",
+    "blocks.18.attn.qkv.weight",
+    "blocks.18.attn.qkv.bias",
+    "blocks.18.attn.out_proj.weight",
+    "blocks.18.attn.out_proj.bias",
+    "blocks.18.mlp.fc.weight",
+    "blocks.18.mlp.fc.bias",
+    "blocks.18.mlp.proj.weight",
+    "blocks.18.mlp.proj.bias",
+    "blocks.19.ln_1.weight",
+    "blocks.19.ln_1.bias",
+    "blocks.19.ln_2.weight",
+    "blocks.19.ln_2.bias",
+    "blocks.19.attn.qkv.weight",
+    "blocks.19.attn.qkv.bias",
+    "blocks.19.attn.out_proj.weight",
+    "blocks.19.attn.out_proj.bias",
+    "blocks.19.mlp.fc.weight",
+    "blocks.19.mlp.fc.bias",
+    "blocks.19.mlp.proj.weight",
+    "blocks.19.mlp.proj.bias",
+    "blocks.20.ln_1.weight",
+    "blocks.20.ln_1.bias",
+    "blocks.20.ln_2.weight",
+    "blocks.20.ln_2.bias",
+    "blocks.20.attn.qkv.weight",
+    "blocks.20.attn.qkv.bias",
+    "blocks.20.attn.out_proj.weight",
+    "blocks.20.attn.out_proj.bias",
+    "blocks.20.mlp.fc.weight",
+    "blocks.20.mlp.fc.bias",
+    "blocks.20.mlp.proj.weight",
+    "blocks.20.mlp.proj.bias",
+    "blocks.21.ln_1.weight",
+    "blocks.21.ln_1.bias",
+    "blocks.21.ln_2.weight",
+    "blocks.21.ln_2.bias",
+    "blocks.21.attn.qkv.weight",
+    "blocks.21.attn.qkv.bias",
+    "blocks.21.attn.out_proj.weight",
+    "blocks.21.attn.out_proj.bias",
+    "blocks.21.mlp.fc.weight",
+    "blocks.21.mlp.fc.bias",
+    "blocks.21.mlp.proj.weight",
+    "blocks.21.mlp.proj.bias",
+    "blocks.22.ln_1.weight",
+    "blocks.22.ln_1.bias",
+    "blocks.22.ln_2.weight",
+    "blocks.22.ln_2.bias",
+    "blocks.22.attn.qkv.weight",
+    "blocks.22.attn.qkv.bias",
+    "blocks.22.attn.out_proj.weight",
+    "blocks.22.attn.out_proj.bias",
+    "blocks.22.mlp.fc.weight",
+    "blocks.22.mlp.fc.bias",
+    "blocks.22.mlp.proj.weight",
+    "blocks.22.mlp.proj.bias",
+    "blocks.23.ln_1.weight",
+    "blocks.23.ln_1.bias",
+    "blocks.23.ln_2.weight",
+    "blocks.23.ln_2.bias",
+    "blocks.23.attn.qkv.weight",
+    "blocks.23.attn.qkv.bias",
+    "blocks.23.attn.out_proj.weight",
+    "blocks.23.attn.out_proj.bias",
+    "blocks.23.mlp.fc.weight",
+    "blocks.23.mlp.fc.bias",
+    "blocks.23.mlp.proj.weight",
+    "blocks.23.mlp.proj.bias",
+    "ln_f.weight",
+    "ln_f.bias",
+    "head.weight",
+    "head.bias"
+  ],
+  "tokenizer_bundle": {
+    "special_tokens_map.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/special_tokens_map.json",
+    "tokenizer.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/tokenizer.json",
+    "tokenizer_config.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/tokenizer_config.json",
+    "tokenizer_meta.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/tokenizer_meta.json"
+  },
+  "tokenizer_reference": {
+    "packed_dataset_config_path": null,
+    "tokenizer_dir": "/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M",
+    "training_config_path": null
+  },
+  "transformers_config": {
+    "activation_function": "gelu",
+    "architecture": "gpt2",
+    "architectures": [
+      "GPT2LMHeadModel"
+    ],
+    "attn_pdrop": 0.0,
+    "block_type": "gpt2_prelayernorm",
+    "causal_mask_mode": "buffered_upper_triangular",
+    "embd_pdrop": 0.0,
+    "init_strategy": "gpt2_std_0.02_residual_scale",
+    "layer_norm_epsilon": 1e-05,
+    "model_type": "gpt2",
+    "n_ctx": 2500,
+    "n_embd": 1024,
+    "n_head": 16,
+    "n_layer": 24,
+    "n_positions": 2500,
+    "norm_order": "preln",
+    "norm_type": "layernorm",
+    "positional_encoding": "learned_absolute",
+    "resid_pdrop": 0.0,
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "vocab_size": 32000
+  },
+  "transformers_export_notes": {
+    "dropped_lm_head_bias_for_gpt2_compat": true,
+    "dropped_lm_head_bias_max_abs": 1.3327739238739014,
+    "dropped_lm_head_bias_mean_abs": 0.046846117824316025
+  },
+  "transformers_safetensors_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/model.safetensors"
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "do_sample": true,
+  "max_new_tokens": 1000,
+  "temperature": 1.0,
+  "top_k": 100,
+  "top_p": 0.95,
+  "repetition_penalty": 1.1,
+  "no_repeat_ngram_size": 0
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e654d2b995c4a2cd293f4d55d7a6147351537298ba5109525e979ef8484e75d2
+size 1350587904

recommended_decoding_params.json

@@ -0,0 +1,20 @@
+{
+  "checkpoint": "step_34200",
+  "preset": "creative",
+  "max_new_tokens": 1000,
+  "temperature": 1.0,
+  "top_k": 100,
+  "top_p": 0.95,
+  "repetition_penalty": 1.1,
+  "no_repeat_ngram_size": 0,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "evidence": {
+    "tuning_score": 3.1683,
+    "holdout_score": 3.8405,
+    "eos_rate_tuning": 1.0,
+    "eos_rate_holdout": 1.0,
+    "strict_loop_rate_tuning": 0.0,
+    "strict_loop_rate_holdout": 0.0
+  }
+}

release_note.md

@@ -0,0 +1,7 @@
+This is the definitive official `1gpu-llm-medium-v2` release.
+
+It promotes checkpoint `step_34200` from the optimizer-preserving decay-only
+continuation of the main medium CPT checkpoint `step_34000`.
+
+The release was selected by the complete 1000-token tuning/holdout decoding
+grid and is published with the official 1gpu-llm card image.

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

step_34200.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aef70a6e99512059ed203375f82608083f2ce657b1f57f70e2cb06f055d28182
+size 4052410819

step_34200.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5d59935e646531d069bce475d4220b0a82c28a7d0f1636c01b0d0db6441947a
+size 1481791640

step_34200.safetensors.json

@@ -0,0 +1,519 @@
+{
+  "checkpoint_config": {
+    "activation": "gelu",
+    "actual_precision": "bf16",
+    "adamw_betas": [
+      0.9,
+      0.95
+    ],
+    "adamw_eps": 1e-08,
+    "architecture": "gpt2",
+    "attention_kernel_policy": "auto",
+    "batch_size": 2,
+    "benchmark": {
+      "enable_central_tensorboard": true,
+      "enable_local_tensorboard": true,
+      "enabled": false,
+      "output_path": "/mnt/apps/llm-nanochat/artifacts/runs/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/throughput_benchmark.json",
+      "warmup_steps": 0
+    },
+    "bias": true,
+    "block_type": "gpt2_prelayernorm",
+    "causal_mask_mode": "buffered_upper_triangular",
+    "checkpoint_dir": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki",
+    "clip_grad_norm": 1.0,
+    "compile": {
+      "backend": null,
+      "compile_setup_sec": 0.0,
+      "diagnostic": null,
+      "dynamic": false,
+      "enabled": false,
+      "error_policy": "raise",
+      "fullgraph": false,
+      "mode": null,
+      "requested": false,
+      "status": "disabled"
+    },
+    "dataset": {
+      "storage_mode": "indexed_jsonl"
+    },
+    "dataset_dir": "/mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M",
+    "decay_shape": "inverse_proportional",
+    "decay_steps": 1700,
+    "decay_steps_source": "explicit",
+    "deterministic_algorithms": false,
+    "device": "cuda",
+    "dim": 1024,
+    "dropout": 0.0,
+    "effective_global_step": 34200,
+    "effective_layer_count": 24,
+    "effective_learning_rate": 3.309994670030697e-05,
+    "final_lr": 1e-05,
+    "final_lr_source": "explicit",
+    "fp8_backend": null,
+    "grad_accum_steps": 48,
+    "init_from": "/mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt",
+    "init_strategy": "gpt2_std_0.02_residual_scale",
+    "learning_rate": 4.7832243797285384e-05,
+    "local_max_steps": 35700,
+    "local_step": 200,
+    "logging": {
+      "enable_central_tensorboard": true,
+      "enable_local_tensorboard": true,
+      "metrics_flush_every_steps": 1,
+      "metrics_writer": "persistent_jsonl_handle"
+    },
+    "lr": 4.7832243797285384e-05,
+    "lr_schedule": "wsd-decay-only",
+    "max_seq_len": 2500,
+    "max_steps": 35700,
+    "n_heads": 16,
+    "n_layers": 24,
+    "norm_order": "preln",
+    "norm_type": "layernorm",
+    "optimizer": {
+      "backend": "torch",
+      "betas": [
+        0.9,
+        0.95
+      ],
+      "eps": 1e-08,
+      "implementation": "torch.optim.AdamW",
+      "learning_rate": 4.7832243797285384e-05,
+      "state_precision": "full_precision",
+      "type": "adamw",
+      "weight_decay": 0.1
+    },
+    "optimizer_backend": "torch",
+    "optimizer_implementation": "torch.optim.AdamW",
+    "optimizer_reset": false,
+    "optimizer_state_precision": "full_precision",
+    "optimizer_type": "adamw",
+    "peak_lr": 4.7832243797285384e-05,
+    "positional_encoding": "learned_absolute",
+    "repro": {
+      "attention_kernel_policy": "auto",
+      "cublas_workspace_config": null,
+      "cudnn_benchmark": true,
+      "cudnn_deterministic": false,
+      "deterministic_algorithms": false,
+      "flash_sdp_enabled": true,
+      "math_sdp_enabled": true,
+      "mem_efficient_sdp_enabled": true,
+      "pythonhashseed": "1337",
+      "seed": 1337
+    },
+    "requested_precision": "bf16",
+    "resume_checkpoint_step": 34000,
+    "resume_from": "/mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt",
+    "resume_mode": "optimizer_only",
+    "resume_step": 34000,
+    "save_every_steps": 100,
+    "save_final_checkpoint": true,
+    "schedule_total_steps": 35700,
+    "scheduler": {
+      "decay_shape": "inverse_proportional",
+      "decay_steps": 1700,
+      "decay_steps_source": "explicit",
+      "final_lr": 1e-05,
+      "final_lr_source": "explicit",
+      "peak_lr": 4.7832243797285384e-05,
+      "resume_step": 34000,
+      "schedule_type": "wsd-decay-only",
+      "stable_steps": 0,
+      "target_final_step": 35700,
+      "total_steps": 35700,
+      "warmup_steps": 0
+    },
+    "scheduler_reset": true,
+    "seed": 1337,
+    "source_global_step": 34000,
+    "stable_steps": 0,
+    "target_final_step": 35700,
+    "target_global_step": 35700,
+    "tie_word_embeddings": true,
+    "train_cache_ram_bytes": 1073741824,
+    "train_cache_ram_mb": 1024,
+    "vocab_size": 32000,
+    "warmup_steps": 0,
+    "weight_decay": 0.1
+  },
+  "checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt",
+  "config_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/config.json",
+  "exported_at": "2026-07-16T13:41:05.693620+00:00",
+  "format": "llm-nanochat-safetensors-export",
+  "global_step": 34200,
+  "metadata_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/step_34200.safetensors.json",
+  "model_config": {
+    "activation": "gelu",
+    "architecture": "gpt2",
+    "bias": true,
+    "block_type": "gpt2_prelayernorm",
+    "causal_mask_mode": "buffered_upper_triangular",
+    "dim": 1024,
+    "dropout": 0.0,
+    "init_strategy": "gpt2_std_0.02_residual_scale",
+    "max_seq_len": 2500,
+    "n_heads": 16,
+    "n_layers": 24,
+    "norm_order": "preln",
+    "norm_type": "layernorm",
+    "positional_encoding": "learned_absolute",
+    "tie_word_embeddings": true,
+    "vocab_size": 32000
+  },
+  "num_parameters": 370439424,
+  "num_tensors": 294,
+  "provenance": {
+    "checkpoint_dir": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki",
+    "checkpoint_name": "step_34200.pt",
+    "checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt",
+    "global_step": 34200,
+    "packed_dataset_config_path": null,
+    "run_dir": "/mnt/apps/llm-nanochat/checkpoints",
+    "tokenizer_dir": "/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M",
+    "training_config_path": null
+  },
+  "safetensors_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/step_34200.safetensors",
+  "source_checkpoint_path": "/mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt",
+  "source_global_step": 34200,
+  "tensor_names": [
+    "token_emb.weight",
+    "pos_emb.weight",
+    "blocks.0.ln_1.weight",
+    "blocks.0.ln_1.bias",
+    "blocks.0.ln_2.weight",
+    "blocks.0.ln_2.bias",
+    "blocks.0.attn.qkv.weight",
+    "blocks.0.attn.qkv.bias",
+    "blocks.0.attn.out_proj.weight",
+    "blocks.0.attn.out_proj.bias",
+    "blocks.0.mlp.fc.weight",
+    "blocks.0.mlp.fc.bias",
+    "blocks.0.mlp.proj.weight",
+    "blocks.0.mlp.proj.bias",
+    "blocks.1.ln_1.weight",
+    "blocks.1.ln_1.bias",
+    "blocks.1.ln_2.weight",
+    "blocks.1.ln_2.bias",
+    "blocks.1.attn.qkv.weight",
+    "blocks.1.attn.qkv.bias",
+    "blocks.1.attn.out_proj.weight",
+    "blocks.1.attn.out_proj.bias",
+    "blocks.1.mlp.fc.weight",
+    "blocks.1.mlp.fc.bias",
+    "blocks.1.mlp.proj.weight",
+    "blocks.1.mlp.proj.bias",
+    "blocks.2.ln_1.weight",
+    "blocks.2.ln_1.bias",
+    "blocks.2.ln_2.weight",
+    "blocks.2.ln_2.bias",
+    "blocks.2.attn.qkv.weight",
+    "blocks.2.attn.qkv.bias",
+    "blocks.2.attn.out_proj.weight",
+    "blocks.2.attn.out_proj.bias",
+    "blocks.2.mlp.fc.weight",
+    "blocks.2.mlp.fc.bias",
+    "blocks.2.mlp.proj.weight",
+    "blocks.2.mlp.proj.bias",
+    "blocks.3.ln_1.weight",
+    "blocks.3.ln_1.bias",
+    "blocks.3.ln_2.weight",
+    "blocks.3.ln_2.bias",
+    "blocks.3.attn.qkv.weight",
+    "blocks.3.attn.qkv.bias",
+    "blocks.3.attn.out_proj.weight",
+    "blocks.3.attn.out_proj.bias",
+    "blocks.3.mlp.fc.weight",
+    "blocks.3.mlp.fc.bias",
+    "blocks.3.mlp.proj.weight",
+    "blocks.3.mlp.proj.bias",
+    "blocks.4.ln_1.weight",
+    "blocks.4.ln_1.bias",
+    "blocks.4.ln_2.weight",
+    "blocks.4.ln_2.bias",
+    "blocks.4.attn.qkv.weight",
+    "blocks.4.attn.qkv.bias",
+    "blocks.4.attn.out_proj.weight",
+    "blocks.4.attn.out_proj.bias",
+    "blocks.4.mlp.fc.weight",
+    "blocks.4.mlp.fc.bias",
+    "blocks.4.mlp.proj.weight",
+    "blocks.4.mlp.proj.bias",
+    "blocks.5.ln_1.weight",
+    "blocks.5.ln_1.bias",
+    "blocks.5.ln_2.weight",
+    "blocks.5.ln_2.bias",
+    "blocks.5.attn.qkv.weight",
+    "blocks.5.attn.qkv.bias",
+    "blocks.5.attn.out_proj.weight",
+    "blocks.5.attn.out_proj.bias",
+    "blocks.5.mlp.fc.weight",
+    "blocks.5.mlp.fc.bias",
+    "blocks.5.mlp.proj.weight",
+    "blocks.5.mlp.proj.bias",
+    "blocks.6.ln_1.weight",
+    "blocks.6.ln_1.bias",
+    "blocks.6.ln_2.weight",
+    "blocks.6.ln_2.bias",
+    "blocks.6.attn.qkv.weight",
+    "blocks.6.attn.qkv.bias",
+    "blocks.6.attn.out_proj.weight",
+    "blocks.6.attn.out_proj.bias",
+    "blocks.6.mlp.fc.weight",
+    "blocks.6.mlp.fc.bias",
+    "blocks.6.mlp.proj.weight",
+    "blocks.6.mlp.proj.bias",
+    "blocks.7.ln_1.weight",
+    "blocks.7.ln_1.bias",
+    "blocks.7.ln_2.weight",
+    "blocks.7.ln_2.bias",
+    "blocks.7.attn.qkv.weight",
+    "blocks.7.attn.qkv.bias",
+    "blocks.7.attn.out_proj.weight",
+    "blocks.7.attn.out_proj.bias",
+    "blocks.7.mlp.fc.weight",
+    "blocks.7.mlp.fc.bias",
+    "blocks.7.mlp.proj.weight",
+    "blocks.7.mlp.proj.bias",
+    "blocks.8.ln_1.weight",
+    "blocks.8.ln_1.bias",
+    "blocks.8.ln_2.weight",
+    "blocks.8.ln_2.bias",
+    "blocks.8.attn.qkv.weight",
+    "blocks.8.attn.qkv.bias",
+    "blocks.8.attn.out_proj.weight",
+    "blocks.8.attn.out_proj.bias",
+    "blocks.8.mlp.fc.weight",
+    "blocks.8.mlp.fc.bias",
+    "blocks.8.mlp.proj.weight",
+    "blocks.8.mlp.proj.bias",
+    "blocks.9.ln_1.weight",
+    "blocks.9.ln_1.bias",
+    "blocks.9.ln_2.weight",
+    "blocks.9.ln_2.bias",
+    "blocks.9.attn.qkv.weight",
+    "blocks.9.attn.qkv.bias",
+    "blocks.9.attn.out_proj.weight",
+    "blocks.9.attn.out_proj.bias",
+    "blocks.9.mlp.fc.weight",
+    "blocks.9.mlp.fc.bias",
+    "blocks.9.mlp.proj.weight",
+    "blocks.9.mlp.proj.bias",
+    "blocks.10.ln_1.weight",
+    "blocks.10.ln_1.bias",
+    "blocks.10.ln_2.weight",
+    "blocks.10.ln_2.bias",
+    "blocks.10.attn.qkv.weight",
+    "blocks.10.attn.qkv.bias",
+    "blocks.10.attn.out_proj.weight",
+    "blocks.10.attn.out_proj.bias",
+    "blocks.10.mlp.fc.weight",
+    "blocks.10.mlp.fc.bias",
+    "blocks.10.mlp.proj.weight",
+    "blocks.10.mlp.proj.bias",
+    "blocks.11.ln_1.weight",
+    "blocks.11.ln_1.bias",
+    "blocks.11.ln_2.weight",
+    "blocks.11.ln_2.bias",
+    "blocks.11.attn.qkv.weight",
+    "blocks.11.attn.qkv.bias",
+    "blocks.11.attn.out_proj.weight",
+    "blocks.11.attn.out_proj.bias",
+    "blocks.11.mlp.fc.weight",
+    "blocks.11.mlp.fc.bias",
+    "blocks.11.mlp.proj.weight",
+    "blocks.11.mlp.proj.bias",
+    "blocks.12.ln_1.weight",
+    "blocks.12.ln_1.bias",
+    "blocks.12.ln_2.weight",
+    "blocks.12.ln_2.bias",
+    "blocks.12.attn.qkv.weight",
+    "blocks.12.attn.qkv.bias",
+    "blocks.12.attn.out_proj.weight",
+    "blocks.12.attn.out_proj.bias",
+    "blocks.12.mlp.fc.weight",
+    "blocks.12.mlp.fc.bias",
+    "blocks.12.mlp.proj.weight",
+    "blocks.12.mlp.proj.bias",
+    "blocks.13.ln_1.weight",
+    "blocks.13.ln_1.bias",
+    "blocks.13.ln_2.weight",
+    "blocks.13.ln_2.bias",
+    "blocks.13.attn.qkv.weight",
+    "blocks.13.attn.qkv.bias",
+    "blocks.13.attn.out_proj.weight",
+    "blocks.13.attn.out_proj.bias",
+    "blocks.13.mlp.fc.weight",
+    "blocks.13.mlp.fc.bias",
+    "blocks.13.mlp.proj.weight",
+    "blocks.13.mlp.proj.bias",
+    "blocks.14.ln_1.weight",
+    "blocks.14.ln_1.bias",
+    "blocks.14.ln_2.weight",
+    "blocks.14.ln_2.bias",
+    "blocks.14.attn.qkv.weight",
+    "blocks.14.attn.qkv.bias",
+    "blocks.14.attn.out_proj.weight",
+    "blocks.14.attn.out_proj.bias",
+    "blocks.14.mlp.fc.weight",
+    "blocks.14.mlp.fc.bias",
+    "blocks.14.mlp.proj.weight",
+    "blocks.14.mlp.proj.bias",
+    "blocks.15.ln_1.weight",
+    "blocks.15.ln_1.bias",
+    "blocks.15.ln_2.weight",
+    "blocks.15.ln_2.bias",
+    "blocks.15.attn.qkv.weight",
+    "blocks.15.attn.qkv.bias",
+    "blocks.15.attn.out_proj.weight",
+    "blocks.15.attn.out_proj.bias",
+    "blocks.15.mlp.fc.weight",
+    "blocks.15.mlp.fc.bias",
+    "blocks.15.mlp.proj.weight",
+    "blocks.15.mlp.proj.bias",
+    "blocks.16.ln_1.weight",
+    "blocks.16.ln_1.bias",
+    "blocks.16.ln_2.weight",
+    "blocks.16.ln_2.bias",
+    "blocks.16.attn.qkv.weight",
+    "blocks.16.attn.qkv.bias",
+    "blocks.16.attn.out_proj.weight",
+    "blocks.16.attn.out_proj.bias",
+    "blocks.16.mlp.fc.weight",
+    "blocks.16.mlp.fc.bias",
+    "blocks.16.mlp.proj.weight",
+    "blocks.16.mlp.proj.bias",
+    "blocks.17.ln_1.weight",
+    "blocks.17.ln_1.bias",
+    "blocks.17.ln_2.weight",
+    "blocks.17.ln_2.bias",
+    "blocks.17.attn.qkv.weight",
+    "blocks.17.attn.qkv.bias",
+    "blocks.17.attn.out_proj.weight",
+    "blocks.17.attn.out_proj.bias",
+    "blocks.17.mlp.fc.weight",
+    "blocks.17.mlp.fc.bias",
+    "blocks.17.mlp.proj.weight",
+    "blocks.17.mlp.proj.bias",
+    "blocks.18.ln_1.weight",
+    "blocks.18.ln_1.bias",
+    "blocks.18.ln_2.weight",
+    "blocks.18.ln_2.bias",
+    "blocks.18.attn.qkv.weight",
+    "blocks.18.attn.qkv.bias",
+    "blocks.18.attn.out_proj.weight",
+    "blocks.18.attn.out_proj.bias",
+    "blocks.18.mlp.fc.weight",
+    "blocks.18.mlp.fc.bias",
+    "blocks.18.mlp.proj.weight",
+    "blocks.18.mlp.proj.bias",
+    "blocks.19.ln_1.weight",
+    "blocks.19.ln_1.bias",
+    "blocks.19.ln_2.weight",
+    "blocks.19.ln_2.bias",
+    "blocks.19.attn.qkv.weight",
+    "blocks.19.attn.qkv.bias",
+    "blocks.19.attn.out_proj.weight",
+    "blocks.19.attn.out_proj.bias",
+    "blocks.19.mlp.fc.weight",
+    "blocks.19.mlp.fc.bias",
+    "blocks.19.mlp.proj.weight",
+    "blocks.19.mlp.proj.bias",
+    "blocks.20.ln_1.weight",
+    "blocks.20.ln_1.bias",
+    "blocks.20.ln_2.weight",
+    "blocks.20.ln_2.bias",
+    "blocks.20.attn.qkv.weight",
+    "blocks.20.attn.qkv.bias",
+    "blocks.20.attn.out_proj.weight",
+    "blocks.20.attn.out_proj.bias",
+    "blocks.20.mlp.fc.weight",
+    "blocks.20.mlp.fc.bias",
+    "blocks.20.mlp.proj.weight",
+    "blocks.20.mlp.proj.bias",
+    "blocks.21.ln_1.weight",
+    "blocks.21.ln_1.bias",
+    "blocks.21.ln_2.weight",
+    "blocks.21.ln_2.bias",
+    "blocks.21.attn.qkv.weight",
+    "blocks.21.attn.qkv.bias",
+    "blocks.21.attn.out_proj.weight",
+    "blocks.21.attn.out_proj.bias",
+    "blocks.21.mlp.fc.weight",
+    "blocks.21.mlp.fc.bias",
+    "blocks.21.mlp.proj.weight",
+    "blocks.21.mlp.proj.bias",
+    "blocks.22.ln_1.weight",
+    "blocks.22.ln_1.bias",
+    "blocks.22.ln_2.weight",
+    "blocks.22.ln_2.bias",
+    "blocks.22.attn.qkv.weight",
+    "blocks.22.attn.qkv.bias",
+    "blocks.22.attn.out_proj.weight",
+    "blocks.22.attn.out_proj.bias",
+    "blocks.22.mlp.fc.weight",
+    "blocks.22.mlp.fc.bias",
+    "blocks.22.mlp.proj.weight",
+    "blocks.22.mlp.proj.bias",
+    "blocks.23.ln_1.weight",
+    "blocks.23.ln_1.bias",
+    "blocks.23.ln_2.weight",
+    "blocks.23.ln_2.bias",
+    "blocks.23.attn.qkv.weight",
+    "blocks.23.attn.qkv.bias",
+    "blocks.23.attn.out_proj.weight",
+    "blocks.23.attn.out_proj.bias",
+    "blocks.23.mlp.fc.weight",
+    "blocks.23.mlp.fc.bias",
+    "blocks.23.mlp.proj.weight",
+    "blocks.23.mlp.proj.bias",
+    "ln_f.weight",
+    "ln_f.bias",
+    "head.weight",
+    "head.bias"
+  ],
+  "tokenizer_bundle": {
+    "special_tokens_map.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/special_tokens_map.json",
+    "tokenizer.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/tokenizer.json",
+    "tokenizer_config.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/tokenizer_config.json",
+    "tokenizer_meta.json": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/tokenizer_meta.json"
+  },
+  "tokenizer_reference": {
+    "packed_dataset_config_path": null,
+    "tokenizer_dir": "/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M",
+    "training_config_path": null
+  },
+  "transformers_config": {
+    "activation_function": "gelu",
+    "architecture": "gpt2",
+    "architectures": [
+      "GPT2LMHeadModel"
+    ],
+    "attn_pdrop": 0.0,
+    "block_type": "gpt2_prelayernorm",
+    "causal_mask_mode": "buffered_upper_triangular",
+    "embd_pdrop": 0.0,
+    "init_strategy": "gpt2_std_0.02_residual_scale",
+    "layer_norm_epsilon": 1e-05,
+    "model_type": "gpt2",
+    "n_ctx": 2500,
+    "n_embd": 1024,
+    "n_head": 16,
+    "n_layer": 24,
+    "n_positions": 2500,
+    "norm_order": "preln",
+    "norm_type": "layernorm",
+    "positional_encoding": "learned_absolute",
+    "resid_pdrop": 0.0,
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "vocab_size": 32000
+  },
+  "transformers_export_notes": {
+    "dropped_lm_head_bias_for_gpt2_compat": true,
+    "dropped_lm_head_bias_max_abs": 1.3327739238739014,
+    "dropped_lm_head_bias_mean_abs": 0.046846117824316025
+  },
+  "transformers_safetensors_path": "/mnt/apps/llm-nanochat/hf_exports/1gpu-llm-medium-v2/model.safetensors"
+}

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<eos>",
+  "model_max_length": 2500,
+  "pad_token": "<pad>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

tokenizer_meta.json

@@ -0,0 +1,10 @@
+{
+  "vocab_size_requested": 32000,
+  "vocab_size_actual": 32000,
+  "special_tokens": [
+    "<pad>",
+    "<bos>",
+    "<eos>",
+    "<unk>"
+  ]
+}

training_config.yaml

@@ -0,0 +1,66 @@
+# Optimizer-preserving decay-only continuation for the benchmark-winning medium CPT checkpoint at step_34000.
+# Launch with --resume-mode optimizer_only so the checkpoint model + optimizer state are restored,
+# while the configured local wsd-decay-only scheduler is initialized from this config instead of
+# reusing the checkpoint scheduler state.
+# Target window: global step 34000 -> 35700.
+
+resume_from: /mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt
+dataset_dir: /mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
+output_dir: /mnt/apps/llm-nanochat/artifacts/runs/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki
+tokenizer_dir: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
+seed: 1337
+
+model:
+  architecture: gpt2
+  block_type: gpt2_prelayernorm
+  tie_word_embeddings: true
+  vocab_size: 32000
+  dim: 1024
+  n_layers: 24
+  n_heads: 16
+
+training:
+  sequence_length: 2500
+  max_steps: 35700
+  batch_size: 2
+  grad_accum_steps: 48
+
+  learning_rate: 4.7832243797285384e-05
+  peak_lr: 4.7832243797285384e-05
+  lr_schedule: wsd-decay-only
+
+  warmup_steps: 0
+  stable_steps: 0
+  decay_steps: 1700
+  final_lr: 1.0e-05
+  decay_shape: inverse_proportional
+
+  adamw_betas:
+    - 0.9
+    - 0.95
+  adamw_eps: 1.0e-08
+  weight_decay: 0.1
+  clip_grad_norm: 1.0
+
+  save_every_steps: 100
+  save_final_checkpoint: true
+  checkpoint_dir: /mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki
+  precision: bf16
+
+evaluation:
+  validation_every_steps: 100
+  validation_max_batches: 128
+  probe_every_steps: 500
+  probe_tokenizer_dir: /mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M
+  probe_max_new_tokens: 32
+  probe_prompts:
+    en:
+      - prompt: "The capital of Italy is"
+        expected_next_text: " Rome"
+      - prompt: "A small language model should"
+        expected_next_text: " be"
+    it:
+      - prompt: "La capitale d'Italia è"
+        expected_next_text: " Roma"
+      - prompt: "Un piccolo modello linguistico dovrebbe"
+        expected_next_text: " essere"