Byrne-86M-Base

The base model of the Byrne family (distilled step-4000 checkpoint) — a strong general base for continued pretraining / fine-tuning. A ~86M-parameter, from-scratch SpikeWhaleLM decoder (Multi-head Latent Attention, n-gram engram memory, hash-lookup layers, hyper-connections, HRM refinement, MTP) with a custom ChatML-aware tokenizer. Trained with Modal credits during the Small Models, Big Adventures Hackathon.

Related: main model → Byrne-86M

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M-Base", trust_remote_code=True)

Architecture

These models are built on SpikeWhaleLM, a custom ~86M-parameter decoder-only transformer (16 layers, hidden size 640, 4096-token context, 16,512 vocab, tied input/output embeddings). It combines several non-standard components:

• Multi-head Latent Attention (MLA + XSA) — queries and the output projection are LoRA-compressed (rank 128); each head splits into a decoupled RoPE part (dim 16) and a position-agnostic NoPE part (dim 48); 10 query heads share a single KV head (multi-query attention), with QK-norm for stable logits.
• Engram n-gram memory — a gated associative memory that hashes local n-grams (up to trigrams) into a learned 4,096-entry table and mixes the result back into the residual stream.
• Hash-lookup layers (×2) — multi-head content-addressable features alongside the token embeddings.
• Hyper-Connections — learned, width-expanded residual connections mixed via Sinkhorn-normalized routing, in place of the plain residual add.
• HRM refinement — a Hierarchical Reasoning Model block that performs an extra latent "think a bit more" refinement pass over the hidden states before the output head.
• Multi-Token Prediction (MTP) — a DeepSeek-V3-style auxiliary training head predicting more than one next token (no inference cost).
• Feed-forward is dense (the block is MoE-capable, but MoE is disabled in this release).

JEPA vs HRM. The Byrne models are Non-JEPA: they are trained with HRM refinement only (use_hrm_refine=True, use_jepa=False). The sibling Escarda models add a JEPA (Joint-Embedding Predictive) auxiliary objective on top of HRM refinement.

Tokenizer

These models use SpikeTokenizer, a custom byte-level "length-max" (greedy longest-match) tokenizer with a 16,512-token vocabulary — not a standard BPE/HF tokenizer. Text is UTF-8 encoded, each byte mapped to a latin-1 character, then greedily matched against the vocab using the longest key that fits at each position. It is ChatML-aware, with atomic special tokens for framing and reasoning/tool markers (<|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers) plus <bos>/<eos>/<pad>/<unk>. It ships as a PreTrainedTokenizer subclass (spike_tokenizer.py) and loads via AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

log-likelihood, acc_norm = byte-length-normalized).

Task	acc	acc_norm
arc_easy	0.4205	0.3931
arc_challenge	0.1877	0.2389
hellaswag	0.2792	0.2927
winogrande	0.5193	—
piqa	0.5941	0.5860
openbookqa	0.1420	0.2820
boolq	0.6171	—

ArithMark-2.0 (AxiomicLabs) — official metric is raw acc: 0.2732.

Language modeling: WikiText-2 byte_ppl (↓) 2.3753 · BLiMP (↑) 0.7356.

Citation

If you use this model, please cite:

@misc{byrne86mbase,
  title        = {Byrne-86M-Base: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-86M-Base}},
  note         = {Quazim0t0/Byrne-86M-Base}
}

Update: engram repair (behavior-preserving)

The n-gram Engram memory in the original weights was degenerate: with the frozen LSH compressor at init scale, every token hashed to bucket 0, so only one table row ever received gradient. This revision rescales the (frozen) compressor and broadcasts the learned bucket-0 vector across all table rows.

Outputs are bit-identical to the previous revision (verified: max logit difference 0.0 across a prompt battery). The only change: the Engram's hash now spreads across the full table and every bucket is independently trainable — so if you distill or SFT on top of this base, the n-gram memory will actually learn instead of staying a constant bias.