Ik45/nllb-en-id-finetuned

NLLB-Indo-Bilingual: Pruned & Fine-Tuned English↔Indonesian Translation Model

Production-ready bilingual translation model optimized for English ↔ Indonesian (bidirectional) through vocabulary pruning and continual fine-tuning with GaLore memory-efficient optimization.

---

📋 Table of Contents

• Overview
• Model Details
• Performance
• Usage
• Training Methodology
• Limitations
• Citation

---

🔭 Overview

This model is a specialized bilingual variant of Meta's NLLB-200-distilled-600M, designed specifically for high-quality English-Indonesian translation in both directions. Through a two-stage pipeline of leaf-based vocabulary pruning and GaLore-optimized continual fine-tuning, we achieve:

• ~27% parameter reduction (600M → 442M active params) via vocabulary pruning
• Preserved translation quality with competitive BLEU/chrF++ scores
• Efficient inference with reduced memory footprint
• Bidirectional capability (EN→ID and ID→EN) via balanced training

Key Innovations

Technique	Purpose	Impact
Leaf-Based Frequency Pruning	Remove unused vocabulary tokens	56% vocab reduction (256K → 87K)
Embedding Remapping	Align embeddings with pruned vocab	Maintains token representations
Token Batching	Fairseq-style --max-tokens training	2-3× GPU throughput improvement
GaLore Optimization	Memory-efficient fine-tuning	Full fine-tuning on 2× T4 GPUs

---

🏗️ Model Details

Attribute	Value
Base Architecture	M2M100 (Encoder-Decoder)
Base Model	facebook/nllb-200-distilled-600M
Vocabulary Size	86,915 tokens (pruned from 256,204)
Hidden Size	1,024
Encoder/Decoder Layers	12 / 12
Attention Heads	16
FFN Dimension	4,096
Total Parameters	441,719,808 (~442M)
Tied Embeddings	Yes (shared encoder/decoder/lm_head)
Max Sequence Length	2,048 tokens
Supported Pairs	eng_Latn ↔ ind_Latn

Vocabulary Composition

The pruned vocabulary retains: - ✅ All English and Indonesian atomic tokens - ✅ Language tags (eng_Latn, ind_Latn) - ✅ Special tokens (BOS, EOS, PAD, UNK) - ✅ High-frequency merge tokens - ❌ Low-frequency multilingual tokens (pruned)

📊 Performance

FLORES-200 Benchmark Results

Evaluated on FLORES-200 devtest set using sacreBLEU and chrF++:

Direction	sacreBLEU	chrF++	Status
EN → ID	42.71	66.68	✅ Fine-tuned
ID → EN	37.87	63.04	✅ Fine-tuned
EN → ID (original)	41.21	65.65	Baseline
ID → EN (original)	39.43	63.66	Baseline

Note: The fine-tuned model shows improvement on EN→ID (+1.5 BLEU) while maintaining competitive ID→EN performance. The slight ID→EN decrease is expected when specializing a multilingual model for a single pair.

Efficiency Metrics

Metric	Original	Pruned	Improvement
Vocabulary Size	256,204	86,915	-66%
Embedding Parameters	~262M	~89M	-66%
Model File Size	~2.4 GB	~1.65 GB	-31%
Active Parameters	~600M	~442M	-26%
Inference Latency	Baseline	~15% faster	+15%

---

🚀 Usage

Quick Start

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch
# Load model and tokenizer
model_name = "ik4545/nllb-indo-bilingual"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
def translate(text: str, src_lang: str, tgt_lang: str) -> str:
    """Translate text between English and Indonesian."""
    tokenizer.src_lang = src_lang
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
    # Get forced BOS token for target language
    forced_bos_token_id = tokenizer.convert_tokens_to_ids(tgt_lang)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            forced_bos_token_id=forced_bos_token_id,
            max_length=256,
            num_beams=4,
            early_stopping=True,
            length_penalty=1.0,
        )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)
# English → Indonesian
en_text = "Artificial intelligence is rapidly advancing in Indonesia."
id_translation = translate(en_text, "eng_Latn", "ind_Latn")
print(f"EN → ID: {id_translation}")
# Output: "Kecerdasan buatan semakin berkembang pesat di Indonesia."
# Indonesian → English
id_text = "Pemerintah mengumumkan kebijakan baru tentang pendidikan."
en_translation = translate(id_text, "ind_Latn", "eng_Latn")
print(f"ID → EN: {en_translation}")
# Output: "The government announced new policies regarding education."

Using the Pipeline API

from transformers import pipeline
# English to Indonesian
en_to_id = pipeline(
    "translation",
    model="ik4545/nllb-indo-bilingual",
    src_lang="eng_Latn",
    tgt_lang="ind_Latn",
    device=0  # Use GPU if available
)
result = en_to_id("The weather is very nice today.")
print(result[0]["translation_text"])

Supported Language Codes

Language	Code	Direction
English	eng_Latn	Source & Target
Indonesian	ind_Latn	Source & Target

---

🧪 Training Methodology

Stage 1: Leaf-Based Vocabulary Pruning

Following Purason et al. (2026), we implemented leaf-based frequency pruning:

1. Frequency Analysis: Token frequencies computed on 50M characters of EN/ID corpus (FineWeb)
2. Leaf Detection: Identify tokens that are never used as merge-rule components
3. Frequency-Gated Pruning: Remove low-frequency leaf tokens (freq ≤ 5 threshold)
4. Merge Validation: Ensure no unreachable tokens are created post-pruning
5. Embedding Remapping: Slice and remap embedding matrix to new vocabulary Pruning Statistics:

• Original vocabulary: 256,204 tokens
• Pruned vocabulary: 86,915 tokens
• Tokens removed: 169,289 (66%)
• Pruning ratio: 80% of non-atomic tokens

Stage 2: GaLore Continual Fine-Tuning

Fine-tuned on 500K parallel EN-ID sentence pairs using:

Hyperparameter	Value
Optimizer	GaLoreAdamW (attention/FFN) + AdamW (embeddings)
GaLore Rank	128
Learning Rate (GaLore)	5e-4
Learning Rate (Embeddings)	1e-4
Batch Strategy	Token Batching (max 16,384 tokens/GPU)
Gradient Accumulation	8 steps
Training Epochs	2
Label Smoothing	0.0
Hardware	2× NVIDIA Tesla T4 (16GB VRAM)
Framework	PyTorch DDP + HuggingFace Transformers

Key Training Features: - 🎯 Token Batching: Fairseq-style --max-tokens for optimal GPU utilization - 🔄 Bidirectional Training: Equal mix of EN→ID and ID→EN samples - 💾 Gradient Checkpointing: Memory-efficient training - ⚡ FP16 Mixed Precision: Accelerated training on T4 GPUs

⚠️ Limitations

1. Language Scope: Optimized exclusively for English ↔ Indonesian. Performance on other language pairs is not guaranteed and likely degraded compared to the original NLLB-200 model.
2. Domain Specificity: Best performance on general-domain text. Technical, legal, or medical translations may require domain-specific fine-tuning.
3. Vocabulary Coverage: The pruned vocabulary may struggle with:
   • Rare proper nouns not in training corpus
   • Code-switching (mixed EN/ID sentences)
   • Very recent neologisms or slang
4. Sequence Length: Optimal performance for sequences up to 256 tokens. Longer sequences may experience quality degradation.
5. Inference Requirements: Requires forced_bos_token_id to be set correctly for target language. Without it, the model may generate output in an incorrect language.

---

📁 Model Files

.
├── config.json                  # Model configuration
├── tokenizer_config.json        # Tokenizer settings
├── tokenizer.json               # Fast tokenizer (BPE)
├── special_tokens_map.json      # Special token mappings
├── model.safetensors            # Model weights (~1.65 GB)
├── generation_config.json       # Generation defaults
└── README.md                    # This file

---

🤝 Contributing

This model was developed as part of research into efficient multilingual model specialization. For questions, issues, or contributions, please open an issue on the Hugging Face Hub or GitHub repository.

📄 License

This model is licensed under the Apache License 2.0, same as the original NLLB-200 model by Meta AI. The original NLLB-200 model and weights are © Meta Platforms, Inc. This derivative work (pruned vocabulary + fine-tuned weights) is released under the same license terms.

🙏 Acknowledgments

• Meta AI for the original NLLB-200 model and training recipe
• HuggingFace for the Transformers and Datasets libraries
• GaLore Team for memory-efficient optimization techniques
• FineWeb / HuggingFaceFW for high-quality web-crawled training data

---

📚 Citation

If you use this model in your research, please cite: bibtex @misc{nllb-indo-bilingual-2026, title={NLLB-Indo-Bilingual: Efficient Bilingual Translation via Vocabulary Pruning and GaLore Fine-Tuning}, author={ik4545}, year={2026}, howpublished={\url{[https://huggingface.co/ik4545/nllb-indo-bilingual](https://huggingface.co/ik4545/nllb-indo-bilingual)}} } @article{nllb2022, title={No Language Left Behind: Scaling Human-Centered Machine Translation}, author={Costa-jussà, Marta R. and Cross, James and Çelebi, Onur and Elbayad, Maha and Heafield, Kenneth and Heffernan, Kevin and Kalbassi, Elahe and Lam, Janice and Licht, Daniel and Maillard, Jean and Sun, Anna and Wang, Skyler and Wenzek, Guillaume and Youngblood, Al and Akula, Bapi and Barrault, Loïc and González, Gabriel Mejía and Hansanti, Prangthip and Hoffman, John and Jarrett, Semarley and Sadagopan, Kaushik Ram and Rowe, Dirk and Spruit, Shannon and Tran, Chau and Andrews, Pierre and Ayan, Necip Fazil and Bhosale, Shruti and Edunov, Sergey and Fan, Angela and Gao, Cynthia and Goswami, Vedanuj and Guzmán, Francisco and Koehn, Philipp and Mourachko, Alexandre and Ropers, Christophe and Saleem, Safiyyah and Schwenk, Holger and Wang, Jeff}, journal={arXiv preprint arXiv:2207.04672}, year={2022} } @article{purason2026, title={Teaching Old Tokenizers New Words: Leaf-Based Vocabulary Pruning for Multilingual Models}, author={Purason, T. and others}, journal={arXiv preprint arXiv:2512.03989v2}, year={2026} } @article{zhao2024galore, title={GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection}, author={Zhao, J. and others}, journal={arXiv preprint arXiv:2403.03507}, year={2024} }

Made with ❤️ for the Indonesian NLP community