Supernova tokenizer
Collection
Supernova tokenizer collection v5 strong in all of them implement the complete different Moe Architecture • 4 items • Updated • 1
Supernova NepaliFast V5
A From-Scratch Unicode-Safe Mixture-of-Experts Tokenizer
Supernova NepaliFast V5 is a custom tokenizer architecture designed primarily for Nepali, English, Devanagari, and multilingual Unicode text.
V5 is not simply a new vocabulary placed on top of an existing BPE, Unigram, or WordPiece engine.
The core V5 system includes a custom main tokenizer, custom trie-based matching, a dedicated Unicode fallback expert, grapheme-aware routing, exact reconstruction validation, a global token-ID namespace, and deterministic decoding.
Built in Nepal. Built from the ground up.
Architecture
Supernova V5 uses a tokenizer-level Mixture-of-Experts (MoE) architecture.
INPUT TEXT
│
▼
EXTENDED GRAPHEME CLUSTERS
│
▼
┌─────────────────┐
│ MoE Router V2 │
└────────┬────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
MAIN V5 EXPERT UNICODE EXPERT
47,908 IDs 292,555 entries
│ │
▼ ▼
Custom Trie Unicode Trie
Longest Match Sparse IDs
│ │
└────────────┬────────────┘
▼
GLOBAL TOKEN SPACE
│
▼
DETERMINISTIC DECODER
│
▼
ORIGINAL TEXT
The central idea is:
> Use the compact main tokenizer whenever it can represent a grapheme cluster exactly. If it cannot, route that cluster to the Unicode expert instead of allowing information loss or raw unsupported Unicode IDs to leak into the main namespace.
---
Main V5 Expert
The main V5 tokenizer contains:
47,908 vocabulary entries
Its tokenization engine is based on a custom trie implementation with longest-match behavior.
The main namespace is:
0 ... 47,907
The main tokenizer is designed around frequently occurring Nepali, English, Devanagari, punctuation, symbols, and other common text patterns.
---
Unicode Expert
V5 contains a separate Unicode fallback expert containing:
292,555 Unicode entries
The Unicode expert is designed to safely represent Unicode sequences that the main tokenizer cannot represent and reconstruct exactly.
Examples include:
Δ
Ω
❤
❤️
🏔
🏔️
👍🏽
🇳🇵
👨👩👧👦
中国
क्ष
क्
The Unicode expert uses sparse local IDs.
The IDs do not need to be densely numbered from zero.
---
Global Token-ID Namespace
The two experts occupy separate namespaces.
Main vocabulary:
0 ... 47,907
Unicode expert:
global_id = 47,908 + unicode_local_id
Therefore:
MAIN_VOCAB_SIZE = 47,908
Current V5 artifact counts:
Main vocabulary entries: 47,908
Unicode expert entries: 292,555
Total vocabulary entries: 340,463
Because Unicode local IDs are sparse, the maximum global token ID can be larger than the number of total vocabulary entries.
This distinction is intentional.
---
MoE Router V2
The V5 router operates on extended grapheme clusters rather than blindly processing individual Unicode code points.
Conceptually:
Input
│
├── cluster 1
├── cluster 2
├── cluster 3
└── ...
For each cluster, the router first attempts the main expert.
The main expert is accepted only if:
1. Every returned ID belongs to the main namespace.
2. Decoding those IDs reproduces the original cluster exactly.
If both conditions are satisfied:
MAIN EXPERT
If either condition fails:
UNICODE EXPERT
This makes the router correctness-oriented rather than simply trusting the first tokenizer output.
---
Exact Reconstruction Gate
For a cluster text, V5 checks:
ids = tokenizer.encode(text)
Then verifies:
all(0 <= id < 47_908 for id in ids)
and:
tokenizer.decode(ids) == text
Only then is the main result accepted.
Otherwise the cluster is routed to the Unicode expert.
The core invariant is:
> A main-expert result is accepted only when it can reconstruct the original text exactly.
---
Deterministic Decoding
During decoding:
global ID < 47,908
│
▼
Main Expert
For a Unicode ID:
global ID >= 47,908
│
▼
global ID - 47,908
│
▼
Unicode Local ID
│
▼
Unicode Expert
Invalid sparse Unicode IDs are rejected instead of being silently interpreted.
This keeps the two expert namespaces separate.
---
Why Use Mixture of Experts?
A single tokenizer vocabulary has a natural tradeoff.
A smaller vocabulary can reduce vocabulary-related memory requirements, but rare characters and sequences become harder to represent.
A very large vocabulary can improve coverage, but increases vocabulary size and can increase the embedding/storage cost when the tokenizer is used with a language model.
V5 separates these responsibilities.
Common text
↓
Main V5 Expert
Rare / unsupported Unicode
↓
Unicode Expert
The router decides which expert should handle each grapheme cluster.
The goal is therefore not simply:
> "Make the vocabulary bigger."
The goal is:
> Give different kinds of text to different specialized tokenizer components while preserving exact reconstruction.
---
What Makes V5 a Custom Architecture?
There is an important difference between:
Existing tokenizer engine
+
new vocabulary
and:
Custom tokenizer architecture
+
custom vocabulary
+
custom trie
+
Unicode expert
+
grapheme-aware routing
+
exact reconstruction gate
+
global ID namespace
+
deterministic decoding
V5 belongs to the second category.
This does not mean that every individual technique in V5 was invented for the first time in history.
Trie matching, Unicode processing, grapheme segmentation, and tokenization all have extensive prior research and engineering history.
The claim is more precise:
> Supernova V5 combines these mechanisms into an independently engineered tokenizer system rather than merely replacing the vocabulary of an existing tokenizer engine.
---
V4 → V5
V5 is the result of what we learned from V4.
Supernova V4 demonstrated that a relatively small Nepali-focused tokenizer could achieve strong results against tested general-purpose tokenizer configurations.
In one documented Nepali comparison, approximately:
2,890 vocabulary entries
outperformed a tokenizer with approximately:
30,000 vocabulary entries
on the tested benchmark.
This demonstrated an important point:
> Vocabulary size alone does not determine tokenizer quality.
However, V4 also revealed another side of the problem.
When stronger tokenizer configurations were specifically adapted or optimized for Nepali, V4 could not dominate every competitor.
That was not treated as a reason to dismiss the stronger competitors.
It became an engineering signal.
The direction changed from:
Make one specialized tokenizer better
toward:
Build a stronger tokenizer architecture
V5 therefore introduced:
Main Expert
+
Unicode Expert
+
MoE Router
+
Exact Reconstruction
+
Global ID Namespace
---
What V5 Has Demonstrated
The project does not claim that V5 beats every tokenizer on every benchmark.
The documented results include:
V4 achieved strong results against tested general-purpose tokenizer configurations on Nepali.
V4 demonstrated that a much smaller vocabulary can outperform a substantially larger vocabulary on a defined Nepali benchmark.
V5 introduces a multi-expert tokenizer architecture rather than simply increasing vocabulary size.
V5 contains 47,908 main vocabulary entries.
V5 contains 292,555 Unicode expert entries.
The complete V5 artifacts were successfully downloaded from the published repository.
The Unicode expert was successfully reconstructed from the published artifacts.
The MoE architecture passed the documented 15-case functional validation.
Exact reconstruction passed across the tested Nepali, English, Unicode, emoji, and mixed-text cases.
Deterministic encoding was verified.
Global namespace validation was verified.
These are the claims supported by the current evidence.
---
V5 Functional Validation
A fresh copy of the published V5 artifacts was used to reconstruct the tokenizer components and run the MoE router.
The documented functional suite produced:
15 / 15 PASS
Test cases included:
Hello
नेपाल
नमस्ते नेपाल
Supernova AI
😀
🇳🇵
Δ
Ω
❤
❤️
🏔
🏔️
👍🏽
👨👩👧👦
नेपालमा Supernova AI 🚀 छ। 🇳🇵 ❤️ 🏔️
The validation checked:
Exact reconstruction
Deterministic encoding
Valid token namespace
Result:
FUNCTIONAL RESULT: 15/15
---
Benchmark Philosophy
A tokenizer should not be judged using vocabulary size alone.
A serious comparison should consider multiple metrics:
Exact reconstruction
NFC reconstruction
Unknown/error rate
Token count
Tokens per word
Characters per token
Bytes per token
CPU characters/sec
CPU tokens/sec
Determinism
Memory usage
It is also important to distinguish:
General-purpose tokenizer
from:
Tokenizer specifically optimized for the target language
These are different competitive conditions.
A tokenizer that performs well without language-specific optimization and a tokenizer specifically tuned for that language should be evaluated separately.
---
Performance
The current V5 MoE router is primarily a Python-level reference implementation.
Its current priority is:
Correctness
↓
Unicode safety
↓
Determinism
↓
Architecture validation
↓
Performance optimization
Future optimization work can include:
Rust implementation
C++ implementation
SIMD acceleration
compact trie layouts
cache-aware traversal
optimized Unicode lookup
batch encoding
batch decoding
parallel processing
native routing
Therefore, the current Python implementation should not be interpreted as the theoretical performance ceiling of the architecture.
---
Known Limitations
V5 is an experimental research tokenizer and still has areas that require further work.
Known limitations include:
1. The current MoE router is Python-level.
2. The frozen V5 release contains pickle-based artifacts.
3. Native acceleration has not yet been implemented.
4. Memory efficiency can be improved.
5. Installation ergonomics can be improved.
6. Larger multilingual evaluations are still required.
7. More independent benchmark suites are required.
8. More Nepali-specialized competitors should be evaluated.
9. Production-scale batch throughput has not yet been established.
10. Current evidence does not establish universal tokenizer superiority.
These limitations are intentionally documented.
---
Using Supernova V5
Install dependencies
pip install -U huggingface_hub regex
Download the complete repository
The V5 repository is:
Supernova11c/Supernova-NepaliFast-V5
Hugging Face's snapshot_download() can download the complete repository while preserving its directory structure.
from huggingface_hub import snapshot_download
REPO_ID = "Supernova11c/Supernova-NepaliFast-V5"
local_dir = snapshot_download(
repo_id=REPO_ID,
repo_type="model",
local_dir="/content/supernova_v5"
)
print("V5 downloaded to:", local_dir)
---
Load the Main V5 Tokenizer
import os
import pickle
tokenizer_path = os.path.join(
local_dir,
"tokenizer_v5.pkl"
)
with open(tokenizer_path, "rb") as f:
tokenizer_v5 = pickle.load(f)
print(type(tokenizer_v5))
---
Load the Unicode Expert
import json
unicode_dir = os.path.join(
local_dir,
"v5_final",
"unicode_fallback"
)
with open(
os.path.join(unicode_dir, "unicode_trie.pkl"),
"rb"
) as f:
unicode_trie = pickle.load(f)
with open(
os.path.join(unicode_dir, "unicode_vocab.json"),
"r",
encoding="utf-8"
) as f:
unicode_vocab = json.load(f)
The Unicode expert uses the main vocabulary size as its global namespace offset:
MAIN_VOCAB_SIZE = 47_908
---
Load the MoE Router
The repository includes:
supernova_moe_v2.py
Import it:
from supernova_moe_v2 import MoERouterV2
Construct the Unicode expert:
unicode_expert = UnicodeFallbackTokenizer(
trie=unicode_trie,
vocab=unicode_vocab,
id_base=MAIN_VOCAB_SIZE
)
Then construct the router:
router = MoERouterV2(
tokenizer=tokenizer_v5,
unicode_expert=unicode_expert
)
---
Encode
text = "नेपालमा Supernova AI 🚀 छ। 🇳🇵 ❤️ 🏔️"
token_ids = router.encode(text)
print("Token IDs:")
print(token_ids)
---
Decode
decoded = router.decode(token_ids)
print("Decoded:")
print(decoded)
assert decoded == text
print("Exact reconstruction: PASS")
---
Complete Usage Example
import os
import json
import pickle
from huggingface_hub import snapshot_download
# =========================================================
# 1. Download Supernova V5
# =========================================================
REPO_ID = "Supernova11c/Supernova-NepaliFast-V5"
local_dir = snapshot_download(
repo_id=REPO_ID,
repo_type="model",
local_dir="/content/supernova_v5"
)
# =========================================================
# 2. Load main tokenizer
# =========================================================
with open(
os.path.join(local_dir, "tokenizer_v5.pkl"),
"rb"
) as f:
tokenizer_v5 = pickle.load(f)
# =========================================================
# 3. Load Unicode fallback artifacts
# =========================================================
unicode_dir = os.path.join(
local_dir,
"v5_final",
"unicode_fallback"
)
with open(
os.path.join(unicode_dir, "unicode_trie.pkl"),
"rb"
) as f:
unicode_trie = pickle.load(f)
with open(
os.path.join(unicode_dir, "unicode_vocab.json"),
"r",
encoding="utf-8"
) as f:
unicode_vocab = json.load(f)
# =========================================================
# 4. Create Unicode expert
# =========================================================
MAIN_VOCAB_SIZE = 47_908
unicode_expert = UnicodeFallbackTokenizer(
trie=unicode_trie,
vocab=unicode_vocab,
id_base=MAIN_VOCAB_SIZE
)
# =========================================================
# 5. Create MoE Router
# =========================================================
from supernova_moe_v2 import MoERouterV2
router = MoERouterV2(
tokenizer=tokenizer_v5,
unicode_expert=unicode_expert
)
# =========================================================
# 6. Test
# =========================================================
text = "नेपालमा Supernova AI 🚀 छ। 🇳🇵 ❤️ 🏔️"
token_ids = router.encode(text)
decoded = router.decode(token_ids)
print("Input:")
print(text)
print("\nToken IDs:")
print(token_ids)
print("\nDecoded:")
print(decoded)
print("\nExact reconstruction:", decoded == text)
assert decoded == text
print("TOKENIZER TEST: PASS")
---
Important Serialization Note
The current frozen V5 release contains .pkl artifacts.
Python pickle is a serialization mechanism that can execute arbitrary Python behavior when unpickling malicious files.
Only load the V5 pickle artifacts from a source you trust.
A future V5.x release should move toward safer data-only serialization where practical.
---
Reproducibility
The V5 benchmark work uses a frozen benchmark corpus.
Characters: 1,000,000
UTF-8 bytes: 1,867,683
Words: 140,627
Seed: 20260906
Corpus SHA-256:
5fbd88b3284f7843495f1f129623367c273fb511c5a989b4354cdc0248822e9d
Main vocabulary SHA-256:
f81af768752cb6e7a8a724a68003b38eac07a6dbde19b39a320b674653061dbd
These hashes allow benchmark inputs and frozen artifacts to be independently checked.
---
Repository Structure
Important V5 files include:
vocab.json
vocab_list.json
tokenizer_v5.pkl
trie_v5.pkl
object_state.json
metadata.json
v5_final/
├── unicode/
├── unicode_fallback/
│ ├── unicode_trie.pkl
│ ├── unicode_vocab.json
│ └── metadata
└── unicode_inventory/
supernova_moe_v2.py
---
Future Development
Potential V5.x improvements include:
Native tokenizer core
Rust acceleration
C++ acceleration
SIMD optimization
Memory optimization
Safer serialization
Faster Unicode lookup
Batch encoding
Batch decoding
Parallel processing
Expanded multilingual testing
More Nepali benchmark suites
Additional tokenizer experts
Adaptive routing
A future architecture could evolve toward:
INPUT
│
▼
ADAPTIVE ROUTER
│
┌────────────────┼────────────────┐
▼ ▼ ▼
MAIN EXPERT NEPALI EXPERT UNICODE EXPERT
│ │ │
└────────────────┼────────────────┘
▼
GLOBAL TOKENS
│
▼
DECODER
The goal is not simply to increase vocabulary size.
The goal is to make the tokenizer increasingly specialized in deciding how different kinds of text should be represented.
---
From-Scratch Engineering Philosophy
Supernova does not claim that existing tokenizer projects are bad.
Mature tokenizer systems have years of engineering, optimization, testing, and deployment experience behind them.
Supernova follows a different engineering philosophy:
> Understand the technology, then build the core system ourselves.
Using an existing tokenizer engine with a new vocabulary is useful engineering.
Building the tokenizer engine itself is a different engineering challenge.
V5 focuses on the latter.
---
Built in Nepal
Supernova is being developed in Nepal.
The goal is not to claim that technology is automatically better because it comes from Nepal.
The goal is to demonstrate that advanced technology can be researched and engineered from Nepal.
> We do not need to inherit someone else's foundation to participate in advanced technology. We can study it, understand it, challenge it, and build our own.
---
Engineering Over Marketing
Supernova V5 does not claim perfection.
It does not claim to beat every tokenizer.
It does not claim that every mechanism inside the architecture is historically unprecedented.
Instead, the project documents:
what was built
how it works
what was tested
what passed
what failed
what remains unfinished
The principle is:
> From-scratch engineering is not a claim that the result is automatically better. It is a claim about what had to be engineered to obtain the result.
And:
> A benchmark result tells us where we stand today. It does not determine where the architecture can go next.
---
Status
Project: Supernova NepaliFast V5
Architecture: Custom Trie + Unicode Fallback + Mixture-of-Experts Router
Main vocabulary: 47,908 entries
Unicode expert: 292,555 entries
Combined vocabulary entries: 340,463
Functional MoE validation: 15/15
Exact reconstruction: PASS
Determinism: PASS
Unicode namespace validation: PASS
Development status: Experimental / Research
---
Supernova AI
Built in Nepal.
Built from the ground up.
Focused on Nepali and Unicode-aware language technology.also it doesn't mean version 5 is useless it is way more powerful but the main task was Optimisation that is not done in v5 but surely will be done in V6
---
*************************************((******))
# ================================================================
# SUPERNOVA NEPALIFAST V5
# BUILD + TEST + SAVE RUNTIME
# ================================================================
!pip -q install -U huggingface_hub regex
import os
import sys
import json
import pickle
import time
import hashlib
import regex
# ================================================================
# CONFIG
# ================================================================
REPO_ID = "Supernova11c/Supernova-NepaliFast-V5"
REMOTE_DIR = "/content/supernova_v5"
BUILD_DIR = "/content/Supernova-NepaliFast-V5-BUILD"
MAIN_VOCAB_SIZE = 47_908
UNICODE_BASE = 47_908
# ================================================================
# 1. DOWNLOAD REPOSITORY
# ================================================================
from huggingface_hub import snapshot_download
print("=" * 70)
print("SUPERNOVA NEPALIFAST V5 — BUILD")
print("=" * 70)
local_dir = snapshot_download(
repo_id=REPO_ID,
repo_type="model",
local_dir=REMOTE_DIR,
)
print("Repository:", REPO_ID)
print("Downloaded:", local_dir)
# ================================================================
# 2. CREATE BUILD DIRECTORY
# ================================================================
os.makedirs(BUILD_DIR, exist_ok=True)
# ================================================================
# 3. LOAD VOCABULARIES
# ================================================================
with open(
os.path.join(local_dir, "vocab.json"),
"r",
encoding="utf-8",
) as f:
raw_main_vocab = json.load(f)
main_vocab = {
int(k): v
for k, v in raw_main_vocab.items()
}
assert len(main_vocab) == MAIN_VOCAB_SIZE
assert set(main_vocab.keys()) == set(
range(MAIN_VOCAB_SIZE)
)
print("\n✓ Main vocabulary loaded")
print(" Entries:", len(main_vocab))
unicode_vocab_file = os.path.join(
local_dir,
"v5_final",
"unicode_fallback",
"unicode_vocab.json",
)
with open(
unicode_vocab_file,
"r",
encoding="utf-8",
) as f:
raw_unicode_vocab = json.load(f)
if isinstance(raw_unicode_vocab, list):
unicode_vocab = {
i: token
for i, token in enumerate(raw_unicode_vocab)
}
else:
try:
unicode_vocab = {
int(k): v
for k, v in raw_unicode_vocab.items()
}
except Exception:
unicode_vocab = {
int(v): k
for k, v in raw_unicode_vocab.items()
}
UNICODE_VOCAB_SIZE = len(unicode_vocab)
assert set(unicode_vocab.keys()) == set(
range(UNICODE_VOCAB_SIZE)
)
print("✓ Unicode vocabulary loaded")
print(" Entries:", len(unicode_vocab))
# ================================================================
# 4. BUILD GENERIC LONGEST-MATCH TRIE
# ================================================================
class TrieTokenizer:
def __init__(self, vocab):
self.vocab = {
int(k): str(v)
for k, v in vocab.items()
}
self.inverse_vocab = {
token: idx
for idx, token in self.vocab.items()
}
self.trie = {}
for token, token_id in self.inverse_vocab.items():
node = self.trie
for ch in token:
node = node.setdefault(ch, {})
node["__ID__"] = token_id
def encode(self, text):
output = []
i = 0
n = len(text)
while i < n:
node = self.trie
j = i
last_id = None
last_end = i
while j < n:
ch = text[j]
if ch not in node:
break
node = node[ch]
j += 1
if "__ID__" in node:
last_id = node["__ID__"]
last_end = j
if last_id is None:
raise ValueError(
f"Unknown character at position "
f"{i}: {text[i]!r}"
)
output.append(int(last_id))
i = last_end
return output
def decode(self, token_ids):
output = []
for token_id in token_ids:
token_id = int(token_id)
if token_id not in self.vocab:
raise ValueError(
f"Unknown token ID: {token_id}"
)
output.append(
self.vocab[token_id]
)
return "".join(output)
# ================================================================
# 5. BUILD MAIN EXPERT
# ================================================================
print("\nBuilding main expert...")
main_expert = TrieTokenizer(main_vocab)
print("✓ Main expert built")
# ================================================================
# 6. BUILD UNICODE EXPERT
# ================================================================
print("Building Unicode fallback expert...")
unicode_expert = TrieTokenizer(
unicode_vocab
)
print("✓ Unicode expert built")
# ================================================================
# 7. V5 MOE ROUTER
# ================================================================
class SupernovaNepaliFastV5:
"""
Supernova NepaliFast V5
Main Expert:
47,908-token longest-match Trie
Unicode Expert:
292,555-token Unicode fallback Trie
Router:
Extended grapheme cluster routing
Global namespace:
Main: 0 ... 47,907
Unicode: 47,908 + local_id
"""
def __init__(
self,
main_expert,
unicode_expert,
):
self.main_expert = main_expert
self.unicode_expert = unicode_expert
self.main_vocab_size = MAIN_VOCAB_SIZE
self.unicode_base = UNICODE_BASE
self.unicode_ids = set(
unicode_expert.vocab.keys()
)
def _try_main(self, cluster):
try:
ids = self.main_expert.encode(
cluster
)
decoded = self.main_expert.decode(
ids
)
if decoded != cluster:
return None
if not all(
0 <= x < MAIN_VOCAB_SIZE
for x in ids
):
return None
return ids
except Exception:
return None
def encode_with_stats(self, text):
output = []
main_clusters = 0
unicode_clusters = 0
main_tokens = 0
unicode_tokens = 0
clusters = regex.findall(
r"\X",
text
)
for cluster in clusters:
main_ids = self._try_main(
cluster
)
if main_ids is not None:
output.extend(main_ids)
main_clusters += 1
main_tokens += len(main_ids)
continue
# ------------------------------------------------
# Unicode fallback
# ------------------------------------------------
local_ids = self.unicode_expert.encode(
cluster
)
for local_id in local_ids:
local_id = int(local_id)
if local_id not in self.unicode_ids:
raise ValueError(
f"Invalid Unicode local ID: "
f"{local_id}"
)
global_id = (
self.unicode_base
+ local_id
)
output.append(global_id)
unicode_tokens += 1
unicode_clusters += 1
stats = {
"clusters": len(clusters),
"main_clusters": main_clusters,
"unicode_clusters": unicode_clusters,
"main_tokens": main_tokens,
"unicode_tokens": unicode_tokens,
"total_tokens": len(output),
}
return output, stats
def encode(self, text):
ids, _ = self.encode_with_stats(text)
return ids
def decode(self, token_ids):
output = []
main_buffer = []
def flush_main():
if main_buffer:
output.append(
self.main_expert.decode(
main_buffer
)
)
main_buffer.clear()
for token_id in token_ids:
token_id = int(token_id)
# ------------------------------------------------
# Main namespace
# ------------------------------------------------
if 0 <= token_id < MAIN_VOCAB_SIZE:
main_buffer.append(
token_id
)
continue
# ------------------------------------------------
# Unicode namespace
# ------------------------------------------------
flush_main()
local_id = (
token_id
- self.unicode_base
)
if local_id not in self.unicode_ids:
raise ValueError(
f"Invalid global Unicode ID: "
f"{token_id}"
)
output.append(
self.unicode_expert.vocab[
local_id
]
)
flush_main()
return "".join(output)
def tokenize(self, text):
return self.encode(text)
def detokenize(self, token_ids):
return self.decode(token_ids)
# ================================================================
# 8. CREATE V5
# ================================================================
v5 = SupernovaNepaliFastV5(
main_expert=main_expert,
unicode_expert=unicode_expert,
)
print("\n✓ Supernova NepaliFast V5 constructed")
# ================================================================
# 9. MAIN VOCABULARY TEST
# ================================================================
print("\n" + "=" * 70)
print("MAIN VOCABULARY VALIDATION")
print("=" * 70)
main_failures = []
for token_id in range(MAIN_VOCAB_SIZE):
token = main_vocab[token_id]
try:
ids = main_expert.encode(token)
decoded = main_expert.decode(ids)
if decoded != token:
main_failures.append(
(token_id, token, ids, decoded)
)
except Exception as e:
main_failures.append(
(token_id, token, str(e))
)
print("Vocabulary:", f"{MAIN_VOCAB_SIZE:,}")
print("Failures:", len(main_failures))
assert len(main_failures) == 0
print("✓ 47,908 / 47,908 MAIN TOKENS PASS")
# ================================================================
# 10. UNICODE VOCABULARY TEST
# ================================================================
print("\n" + "=" * 70)
print("UNICODE FALLBACK VALIDATION")
print("=" * 70)
unicode_failures = []
for token_id in range(UNICODE_VOCAB_SIZE):
token = unicode_vocab[token_id]
try:
ids = unicode_expert.encode(token)
decoded = unicode_expert.decode(ids)
if decoded != token:
unicode_failures.append(
(token_id, token, ids, decoded)
)
except Exception as e:
unicode_failures.append(
(token_id, token, str(e))
)
print("Vocabulary:", f"{UNICODE_VOCAB_SIZE:,}")
print("Failures:", len(unicode_failures))
assert len(unicode_failures) == 0
print(
f"✓ {UNICODE_VOCAB_SIZE:,} / "
f"{UNICODE_VOCAB_SIZE:,} UNICODE TOKENS PASS"
)
# ================================================================
# 11. OFFICIAL README TEST
# ================================================================
print("\n" + "=" * 70)
print("OFFICIAL README TEST")
print("=" * 70)
text = (
"नेपालमा Supernova AI 🚀 छ। 🇳🇵 ❤️ 🏔️"
)
ids, stats = v5.encode_with_stats(text)
decoded = v5.decode(ids)
print("\nInput:")
print(text)
print("\nIDs:")
print(ids)
print("\nToken count:")
print(len(ids))
print("\nDecoded:")
print(decoded)
print("\nExact reconstruction:")
print(decoded == text)
print("\nRouting:")
print(stats)
assert decoded == text
print("\n✓ README TEST PASS")
# ================================================================
# 12. FUNCTIONAL TEST SUITE
# ================================================================
print("\n" + "=" * 70)
print("FUNCTIONAL TEST SUITE")
print("=" * 70)
tests = [
"नेपाल सुन्दर देश हो।",
"काठमाडौं नेपालको राजधानी हो।",
"नमस्ते",
"Supernova AI",
"Nepali tokenizer",
"नेपालमा Supernova AI छ।",
"नमस्ते Hello",
"नेपाल 🇳🇵",
"🚀",
"❤️",
"🏔️",
"नेपाल 🇳🇵 Supernova 🚀",
"AI मा नेपाली text mixed छ।",
"१२३४५",
"संस्कृतम्",
"Unicode परीक्षण ✓",
"Hello नेपाल 123 🚀",
"कम्प्युटर विज्ञान",
"Artificial Intelligence नेपाल",
"नमस्कार 🙏",
"नेपालको हिमाल 🏔️ सुन्दर छ।",
"नेपालमा 🇳🇵 AI 🤖 Unicode परीक्षण",
]
passed = 0
for i, text in enumerate(tests, 1):
try:
ids, stats = v5.encode_with_stats(text)
decoded = v5.decode(ids)
ok = decoded == text
if ok:
passed += 1
print(
f"{i:02d}. "
f"{'PASS' if ok else 'FAIL'} | "
f"chars={len(text):3d} | "
f"tokens={len(ids):3d} | "
f"main={stats['main_clusters']:3d} | "
f"unicode={stats['unicode_clusters']:3d} | "
f"{text!r}"
)
except Exception as e:
print(
f"{i:02d}. ERROR | "
f"{text!r} | {e}"
)
print("\nPassed:", passed)
print("Total:", len(tests))
assert passed == len(tests)
print("✓ FUNCTIONAL TESTS PASS")
# ================================================================
# 13. DETERMINISM TEST
# ================================================================
print("\n" + "=" * 70)
print("DETERMINISM")
print("=" * 70)
det_text = (
"नेपालमा Supernova AI 🚀 "
"Unicode परीक्षण 🇳🇵 ❤️ 🏔️"
)
ids_a = v5.encode(det_text)
ids_b = v5.encode(det_text)
print(
"Identical:",
ids_a == ids_b
)
assert ids_a == ids_b
print("✓ DETERMINISM PASS")
# ================================================================
# 14. FIND REAL UNICODE FALLBACK CASES
# ================================================================
print("\n" + "=" * 70)
print("SEARCHING FOR UNICODE FALLBACK")
print("=" * 70)
fallback_cases = []
# Search representative Unicode ranges first.
ranges = [
(0x0000, 0x0100),
(0x0400, 0x0500),
(0x0600, 0x0700),
(0x0900, 0x0A00),
(0x2000, 0x2100),
(0x2200, 0x2300),
(0x2500, 0x2600),
(0x2600, 0x2700),
(0x1F300, 0x1F600),
(0x1F600, 0x1F700),
(0x1F900, 0x1FA00),
]
for start, end in ranges:
for codepoint in range(start, end):
if 0xD800 <= codepoint <= 0xDFFF:
continue
ch = chr(codepoint)
try:
ids, stats = v5.encode_with_stats(ch)
if stats["unicode_clusters"] > 0:
fallback_cases.append(
(ch, codepoint, ids, stats)
)
if len(fallback_cases) >= 20:
break
except Exception:
pass
if len(fallback_cases) >= 20:
break
print(
"Fallback cases found:",
len(fallback_cases)
)
for ch, cp, ids, stats in fallback_cases:
decoded = v5.decode(ids)
print(
f"U+{cp:04X}",
repr(ch),
"→",
ids,
"→",
repr(decoded),
)
assert decoded == ch
if fallback_cases:
print("✓ UNICODE FALLBACK PATH VERIFIED")
else:
print(
"⚠ No fallback case found in tested ranges"
)
# ================================================================
# 15. SAVE V5 RUNTIME
# ================================================================
print("\n" + "=" * 70)
print("SAVING V5 RUNTIME")
print("=" * 70)
runtime_file = os.path.join(
BUILD_DIR,
"supernova_nepalifast_v5_runtime.pkl"
)
runtime_state = {
"name":
"Supernova NepaliFast V5",
"version":
"V5",
"architecture":
"Deterministic MoE Unicode-safe Trie tokenizer",
"main_vocab_size":
MAIN_VOCAB_SIZE,
"unicode_vocab_size":
UNICODE_VOCAB_SIZE,
"unicode_base":
UNICODE_BASE,
"main_vocab":
main_vocab,
"unicode_vocab":
unicode_vocab,
}
with open(
runtime_file,
"wb"
) as f:
pickle.dump(
runtime_state,
f,
protocol=pickle.HIGHEST_PROTOCOL
)
print("Saved:")
print(runtime_file)
print(
"Size:",
f"{os.path.getsize(runtime_file):,}",
"bytes"
)
# ================================================================
# 16. SAVE BUILD METADATA
# ================================================================
metadata = {
"model":
"Supernova NepaliFast V5",
"repo":
REPO_ID,
"main_vocab_size":
MAIN_VOCAB_SIZE,
"unicode_vocab_size":
UNICODE_VOCAB_SIZE,
"unicode_base":
UNICODE_BASE,
"router":
"MoERouterV2",
"routing_unit":
"Extended Grapheme Cluster (regex \\X)",
"main_namespace":
"0-47907",
"unicode_namespace":
f"{UNICODE_BASE}+local_id",
"functional_tests":
len(tests),
"functional_passed":
passed,
"fallback_cases":
len(fallback_cases),
}
metadata_file = os.path.join(
BUILD_DIR,
"v5_build_metadata.json"
)
with open(
metadata_file,
"w",
encoding="utf-8"
) as f:
json.dump(
metadata,
f,
ensure_ascii=False,
indent=2
)
print("Metadata:")
print(metadata_file)
# ================================================================
# 17. HASH BUILD FILES
# ================================================================
def sha256_file(path):
h = hashlib.sha256()
with open(path, "rb") as f:
while True:
chunk = f.read(1024 * 1024)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
print("\n" + "=" * 70)
print("BUILD HASHES")
print("=" * 70)
print(
"Runtime SHA256:",
sha256_file(runtime_file)
)
print(
"Metadata SHA256:",
sha256_file(metadata_file)
)
# ================================================================
# 18. FINAL REPORT
# ================================================================
print("\n" + "=" * 70)
print("SUPERNOVA NEPALIFAST V5 — BUILD COMPLETE")
print("=" * 70)
print(
"Main vocabulary:",
f"{MAIN_VOCAB_SIZE:,}"
)
print(
"Unicode fallback:",
f"{UNICODE_VOCAB_SIZE:,}"
)
print(
"Combined reserved namespace:",
f"{MAIN_VOCAB_SIZE + UNICODE_VOCAB_SIZE:,}"
)
print(
"Functional tests:",
f"{passed}/{len(tests)}"
)
print(
"Main vocabulary validation:",
"PASS"
)
print(
"Unicode vocabulary validation:",
"PASS"
)
print(
"README test:",
"PASS"
)
print(
"Determinism:",
"PASS"
)
print(
"Unicode fallback:",
"VERIFIED"
if fallback_cases
else "NOT YET VERIFIED"
)
print("\nRuntime:")
print(runtime_file)
print("\n✓ SUPERNOVA V5 BUILD + TEST COMPLETE")
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support