README.md

---
language:
- ks
library_name: transformers
pipeline_tag: text-generation
tags:
- kashmiri
- byte-level
- causal-lm
- spacebyte
- pytorch
- transformers
license: apache-2.0
datasets:
- Omarrran/KS-PRET-5M_5_million_kashmiri_Pretrainning_LLM_dataset_12M_tokens_2026
model-index:
- name: ks_byte_lm SpaceByte Transformers
  results:
  - task:
      type: text-generation
      name: Byte-level causal language modeling
    dataset:
      type: Omarrran/KS-PRET-5M_5_million_kashmiri_Pretrainning_LLM_dataset_12M_tokens_2026
      name: Kashmiri pretraining corpus
      split: validation
    metrics:
    - type: bpb
      name: Best validation bits-per-byte
      value: 0.9593
    - type: bpb
      name: Final validation bits-per-byte
      value: 0.9862
    - type: accuracy
      name: Validation next-byte top-1 accuracy
      value: 0.7642
    - type: cross_entropy
      name: Final validation cross entropy
      value: 0.6836
---

# ks_byte_lm SpaceByte — Transformers-compatible release

This repo is the easier-to-load Hugging Face Transformers-style package for the trained Kashmiri byte-level `ks_byte_lm` SpaceByte model.

The model is a custom **SpaceByte-style byte-level Transformer causal LM**. Because the architecture is custom, load it with `trust_remote_code=True`.

Recommended checkpoint: `model.safetensors` converted from the original `best.pt` checkpoint.

## Quick install

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## Installation

```bash
pip install torch transformers safetensors regex
```

### Google Colab note

Some Colab images ship a mismatched `torchvision` package. This model does not
use vision at all, but recent `transformers` imports can still touch
`torchvision` and fail with:

```text
RuntimeError: operator torchvision::nms does not exist
ModuleNotFoundError: Could not import module 'PreTrainedModel'
```

If you see that error, run this in a fresh Colab runtime before loading:

```python
!pip uninstall -y torchvision torchaudio
!pip install -U transformers safetensors regex
```

Then restart the runtime and load the model again. Authentication is optional
for this public repo; `HF_TOKEN` warnings only affect rate limits.

## Quick generation

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Omarrran/ks-byte-lm-spacebyte-transformers"

tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("کشمیر", return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)

print(tokenizer.decode(out[0], skip_special_tokens=True))
```

## Recommended generation helper

The repo also includes a small helper that uses the original byte-LM generation loop:

```python
from generation_ksbyte import generate_text

print(generate_text(
    "Omarrran/ks-byte-lm-spacebyte-transformers",
    prompt="کشمیر",
    max_new_tokens=200,
    temperature=0.8,
    top_k=50,
))
```

## Local usage after cloning/downloading

```bash
git clone https://huggingface.co/Omarrran/ks-byte-lm-spacebyte-transformers
cd ks-byte-lm-spacebyte-transformers
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

python - <<'PY'
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "."
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
inputs = tok("کشمیر", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8, top_k=50)
print(tok.decode(out[0], skip_special_tokens=True))
PY
```

## What changed from the original release?

Original release:

- custom project checkpoint: `checkpoints/best.pt`
- loaded with `ksbyte.generate`
- not directly loadable by `AutoModelForCausalLM`

This release:

- root `config.json`
- root `model.safetensors`
- custom `configuration_ksbyte.py`
- custom `modeling_ksbyte.py`
- custom `tokenization_ksbyte.py`
- loadable with `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`
- loadable with `AutoTokenizer.from_pretrained(..., trust_remote_code=True)`

## Metrics

Validation/evaluation artifacts from the source run:

- Best validation BPB: **0.9593**
- Final validation BPB: **0.9862**
- Final validation cross entropy: **0.6836**
- Validation next-byte top-1 accuracy with best checkpoint: **76.42%**
- Training byte tokens: **45,362,173**
- Validation byte tokens: **1,622,371**
- Test byte tokens: **3,074,698**
- Model parameters: **15,837,440**
- Original training stopped at step **4,751 / 5,000** by early stopping

Note: 76.42% is **byte-token top-1 accuracy**, not word-level accuracy.

## Architecture

- task: byte-level causal language modeling
- variant: SpaceByte
- vocab size: 259 = 256 byte values + BOS/EOS/PAD
- hidden size: 384
- layers: 2 local input + 6 global + 2 local output
- attention heads: 6
- KV heads: 2
- context length: 2048 byte tokens
- parameters: 15.84M

## Caveats

- This is a custom architecture, so `trust_remote_code=True` is required.
- It is a byte-level LM; outputs are decoded from UTF-8 bytes.
- Generations can be semantically weak or incomplete; use human review before strong claims.
- This is not a built-in GPT-2/LLaMA/Mistral architecture, but it is Transformers-compatible via custom code.