Chimera-64M

Research artifact — private. This is an experiment, not a product. Do not use it for anything real unless it is trained much longer (see Intended use & limitations).

Chimera-64M is a ~64M-parameter, from-scratch decoder-only language model — the capstone of a family of models that ask whether a transformer's generalizing ability can be carried by unusual dynamical systems in place of the feed-forward block. Chimera doesn't pick one: its per-layer channel mixer is a council of three physics cores — coupled oscillators, a fungal-colony growth model, and the Wheeler–DeWitt wave equation — and a small reinforcement-learning rule (GRPO) learns which law of physics to apply to each token.

Model summary

Parameters	~64.0M
Type	Decoder-only autoregressive LM (custom QuazimotoLM)
Layers / hidden	10 / 768
Attention	MLA + partial RoPE + QK-Norm + GQA + Elo/Bradley–Terry ratings
Channel mixer	ChimeraBlock — GRPO council over 3 physics cores
Context	block_size 1024 (RoPE max_position_embeddings 4096)
Vocab	16,512 (custom SpikeWhale byte-merge tokenizer)
Traits	HRM refinement, MoE-SwiGLU, MTP, JEPA (train-time), fractal phase seed

Two checkpoints are included:

• chkpt/quazimoto.pt — base, pretraining step 40,000 (continued at block 1024).
• chkpt/quazimoto_sft.pt — chat SFT, step 4,250.

The ChimeraBlock — a GRPO council of physics

Instead of an MLP, each layer runs three physics cores in parallel, each proposing a candidate update of the hidden state, then combines them with a differentiable, in-forward-pass GRPO step:

• KuramotoCore — a bank of coupled phase oscillators (Kuramoto dynamics), integrated by a few differentiable Euler steps. (from Quazimoto/Positronic)
• GrowthCore — a Neighbour-Sensing fungal-colony growth model: tips grow in a bounded latent region, sensing their own density and steering away from it. (from Mycel)
• WaveCore — the Wheeler–DeWitt equation: wdw_modes minisuperspace modes under a learnable Lorentzian supermetric, run as a leapfrog wave; exposes a Hamiltonian-constraint aux loss ⟨H²⟩. (from Wheeler)

The GRPO council (the meta-mixer, from grpo_lm) then decides the blend, per token:

1. an internal critic scores each candidate → reward r_g;
2. group-relative advantage A_g = (r_g − mean)/(std + ε), clamped;
3. selection weights softmax(A/temp), clipped around uniform (the PPO clip);
4. anchored back toward uniform (the KL-to-reference anchor);
5. output = the resulting convex combination of the three candidates.

So the network learns which mechanism to trust for each token rather than committing to one. Everything else is the shared "SpikeWhale/Byrne family": MLA attention with Elo/Bradley–Terry key ratings, HRM refinement, MoE-SwiGLU, MTP heads, a JEPA objective, and the SpikeWhale tokenizer. Each trait is a gated near-no-op at init that activates only if it helps.

Usage

Self-contained — load with the bundled code:

import torch
from model import QuazimotoLM, QuazimotoConfig
from spike_tokenizer import SpikeTokenizer

tok = SpikeTokenizer(vocab_file="tokenizer.json")
ck = torch.load("chkpt/quazimoto.pt", map_location="cpu", weights_only=False)
cfg = QuazimotoConfig(**ck["family_config"])
model = QuazimotoLM(cfg); model.load_state_dict(ck["model"], strict=False); model.eval()

ids = torch.tensor([tok.encode("The history of science", add_special_tokens=False)])
out = model.generate(ids, 80, temperature=0.8, top_k=40, use_cache=True)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Also bundled: generate.py (KV cache + self-speculative decoding), chat_sft.py (ChatML REPL), and visualize.py (a live GRPO-council dashboard showing which physics law wins each token).

Intended use & limitations

This is a research artifact for studying architecture generalization — not a usable assistant.

• The base produces fluent, locally-coherent free-form text, but is not factual or reliable over long spans.
• The SFT checkpoint (only 4,250 steps) stays on-topic and produces structured, math-formatted answers, but hallucinates (its formulas and facts are unreliable) and does not truly follow instructions. This is a capacity-and-scale limit of a 64M model with a short SFT run.
• Do not deploy this. To be actually useful it needs to be trained substantially longer (more tokens, larger, longer SFT). It is shared to demonstrate that three exotic physics mixers, GRPO-selected per token, can learn language — the point is that it generalizes, not what it knows.
• Trained on public web/edu/math corpora; inherits their biases and can produce incorrect or offensive text.

Family & thesis

Chimera is the synthesis of a controlled series of mixer swaps (Kuramoto → Quazimoto/Positronic; Neighbour-Sensing growth → Mycel; Wheeler–DeWitt → Wheeler). The shared thesis: the specific mixing mechanism matters far less than the loop of attend → compress → repeat — and a network can even learn to arbitrate between several exotic mechanisms per token.

Citation

@misc{byrne2026chimera,
  title  = {Chimera-64M: a language model whose channel mixer is a GRPO council of
            coupled oscillators, fungal-colony growth, and the Wheeler--DeWitt equation},
  author = {Byrne, Dean},
  year   = {2026},
  note   = {Research artifact; custom PyTorch architecture (Quazim0t0 / SpikeWhale family)}
}

Update: format-blended SFT

Short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train ('#### N' reasoning), and MMLU-style ('Answer: ') examples, so chat fluency improves while the benchmark output-formats are preserved. Held-out (test-split) before->after:

held-out 2.23->1.34, MMLU acc 0.035->0.250, format 0.140->1.000

Note: fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.