labse-bo-en-qe
A reference-free quality-estimation model for Tibetan→English translation. Given a Tibetan source segment and an English translation, it predicts a quality score without needing a reference translation.
It is LaBSE distilling LLM-judge scores, a 471M-parameter encoder that runs locally. Training labels came from the top performing judges from our Tibetan AI Leaderboard (see the Translation Quality Evaluation tab)
Results
The test set has 100 Tibetan→English segments scored by experts. Correlations are Pearson r with the annotator mean; this model's row carries a paired-bootstrap 95% interval:
| Scorer | Needs reference | r |
|---|---|---|
| LLM judge panel (3 judges, GEMBA-DA) | no | 0.762 |
| labse-bo-en-qe | no | 0.538 (0.396–0.672) |
BLEURT (bleurt-large-512) |
yes | 0.478 |
| BLEURT-20-D12 | yes | 0.394 |
- Against BLEURT: This model needs no reference translation (which BLEURT requires) and it cannot be statistically distinguished from BLEURT (+0.060, 95% interval −0.118 to 0.257).
- Against the LLM judge panel: This keeps about 70% of the panel's correlation (retention 0.706, 95% interval 0.521–0.894). Provider-side drift could shift this figure.
The 100-segment test set is small. Treat this model as an inexpensive, local ranking signal, not a substitute for expert judgements.
Usage
The model is LaBSE run on the (source, translation) pair, followed by mean pooling and a linear head. It needs Python 3.10 or later with torch, transformers, safetensors, huggingface_hub and numpy. It does not load through a transformers pipeline; use score.py from this repo:
import sys
from pathlib import Path
from huggingface_hub import hf_hub_download
sys.path.insert(0, str(Path(hf_hub_download("khyentsevision/labse-bo-en-qe", "score.py")).parent))
from score import load
qe = load("khyentsevision/labse-bo-en-qe")
z = qe.score(["བཀྲ་ཤིས་བདེ་ལེགས།"], ["Greetings and good fortune."])
print(z, qe.to_panel_scale(z))
scorereturns z-scores. Higher is better. The scores are best used to rank or compare translations of the same segment-length material; between strong systems, and on passages, the ranking is weaker (see Limitations).to_panel_scalemaps them to an approximate 0–100 scale. It uses the mean and spread of the panel's training labels.- Inputs are truncated at 256 tokens for the source and translation together.
Training
Labels: The labels come from three sets, each z-scored separately and then pooled:
Set Candidates Label Synthetic human translations ~47k Gemini 2.5 Flash, 3 runs Synthetic poor translations ~14k Gemini 2.5 Flash Lite, 3 runs MT output ~13k GEMBA-DA without a reference, mean of grok-4.3, gpt-5.4 and gemini-3-flash Loss: mean-squared error plus a pairwise ranking term within the candidates for each source.
Settings: 2 epochs, learning rate 2e-5, effective batch 32, max length 256. The checkpoint kept is the step with the best validation Spearman (0.625) against the panel.
Splits: by text. No segment from the human-eval set, or from the held-out texts, was used in training.
Our internal study also tested two less successful variants:
- Adding BLEURT-scored local MT and deliberately corrupted translations degrades performance. Proxy Spearman fell from 0.603 to 0.511.
- Dropping the ranking term was never better, but the evidence that the term helps is weak.
Limitations
- Tibetan→English only, and mostly Buddhist literary and philosophical text.
- The labels come from LLM judges, so the model inherits their biases. It learns to imitate its judges, so expect it to trail the panel's agreement with experts, as it does here.
- Performance degrades with length. On 150 multi-segment passage translations it reached only Spearman 0.297 against the panel (95% interval 0.102–0.469); it ordered two systems' translations of the same passage as the panel did in 0.625 of pairs. Raising the input limit to 512 tokens added only 0.039, so truncation is not the primary mechanism for this degradation.
- It may separate strong systems less well. In an exploratory check on held-out segments, Spearman against the panel was 0.599 across 34 systems but 0.449 (95% interval 0.269–0.577) when only Claude Opus 4.8 and Gemini 3 Flash were compared. Against expert annotators, by contrast, its r on the 40 Claude and Gemini test segments matched BLEURT's; both checks are small.
- Scores across very different domains or segment lengths are not calibrated.
Licence
Apache-2.0, the same licence as the base model, LaBSE.
- Downloads last month
- 44
Model tree for khyentsevision/labse-bo-en-qe
Base model
sentence-transformers/LaBSE