Xiangliu
Xiangliu names the language of a piece of text. It does the same job as fastText's lid.176 and facebook/fasttext-language-identification, in a 4.8 MB file under the MIT licence, with NumPy as its only dependency.
It has 284 language labels. Outside benchmarks test 201 of them; the other 83 were tested on text held back from their own sources. 19 are constructed, ancient, historical or extinct languages, such as Klingon and Latin. It has no label for some languages you may expect, including Haitian, Tajik, Sindhi, Bosnian and Wu Chinese.
Macro F1 on FLORES-200 test text, over the 106 languages every model here has:
| Model | File | Labels | 10 chars | 50 chars | 200 chars | Licence |
|---|---|---|---|---|---|---|
| Xiangliu 1.0 | 4.8 MB | 284 | 0.738 | 0.944 | 0.977 | MIT |
fastText lid.176.ftz |
0.9 MB | 176 | 0.535 | 0.791 | 0.877 | CC BY-SA 3.0 |
fastText lid.176.bin |
131 MB | 176 | 0.581 | 0.837 | 0.907 | CC BY-SA 3.0 |
GlotLID v4 small .ftz |
35 MB | 2,060 | 0.659 | 0.930 | 0.979 | Apache 2.0 plus notices |
| GlotLID v4 | 783 MB | 2,060 | 0.763 | 0.970 | 0.987 | Apache 2.0 plus notices |
facebook/fasttext-language-identification |
1,176 MB | 218 | 0.715 | 0.970 | 0.986 | CC BY-NC 4.0 |
| OpenLID-v3 | 1,233 MB | 195 | 0.716 | 0.970 | 0.987 | GPL-3.0 |
Against each model, over the nine comparisons on this page (this table, the closed table and the WiLI-2018 table, at three lengths each), Xiangliu is as follows. A gap of 0.001 or less, the most two training runs differed by, counts as level.
- fastText
lid.176.ftz(0.9 MB): ahead in 9, behind in 0. - fastText
lid.176.bin(131 MB): ahead in 9, behind in 0. - GlotLID v4 small
.ftz(35 MB): ahead in 5, behind in 4. - GlotLID v4 (783 MB): ahead in 0, behind in 9.
facebook/fasttext-language-identification(1,176 MB): ahead in 2, behind in 7.- OpenLID-v3 (1,233 MB): ahead in 2, behind in 7.
Try it in your browser: DraconiticLabs/xiangliu-demo.
# pip install numpy huggingface_hub
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("DraconiticLabs/xiangliu", revision="v1.0.0")
sys.path.insert(0, path)
from xiangliu_lid import Xiangliu
model = Xiangliu.load(path)
print(model.identify("Le gouvernement a annoncé de nouvelles mesures."))
# Prediction(code='fra', name='French', probability=0.998)
model.top("Serikali imetangaza hatua mpya.", k=3) # the three likeliest
model.identify("Das ist gut.", only=["deu", "nld", "eng"]) # choose among these only
From the command line, after the download: python xiangliu_lid.py "some text", or pipe lines into it.
In JavaScript, with no dependencies:
import { Xiangliu } from "./xiangliu.js";
// In a browser, from the folder holding config.json and model.safetensors:
const model = await Xiangliu.fromUrl("./");
// In Node:
// import { readFileSync } from "node:fs";
// const model = new Xiangliu(JSON.parse(readFileSync("config.json", "utf8")), readFileSync("model.safetensors"));
model.identify("Le gouvernement a annoncé de nouvelles mesures.");
Codes are ISO 639-3. Text with no letters returns und. The Python and JavaScript runtimes give the same answers; tests/test_parity.py holds them to it, down to every character assigned in Unicode 15.0.
What it is
It reads the first 100,000 characters of the input. Text is lower-cased, everything but letters and marks collapses to spaces, and at most 20,000 characters of that are used. Every character sequence of 1 to 4 is hashed to two rows of a 131,072 × 32 table of 8-bit numbers. The rows are averaged, then go through one hidden layer of 512 units to 284 outputs. 4,356,892 parameters. It was trained from random initial weights, with no pretrained parts, and it is written for this job rather than exported from fastText.
Each text gets one label. Arabic varieties share the label ara, Persian varieties fas, and Akan and Twi aka. Norwegian and Serbo-Croatian are split into their member languages: nob and nno, hrv and srp. Languages in lid.176 that this model has no label for: Aragonese (arg), Bavarian (bar), Bishnupriya (bpy), Bosnian (bos), Central Bikol (bcl), Corsican (cos), Dimli (diq), Dotyali (dty), Fiji Hindi (hif), Haitian (hat), Karachay-Balkar (krc), Lezghian (lez), Limburgan (lim), Lombard (lmo), Maithili (mai), Manx (glv), Mazanderani (mzn), Mingrelian (xmf), Mirandese (mwl), Neapolitan (nap), Northern Luri (lrc), Pampanga (pam), Pfaelzisch (pfl), Piemontese (pms), Russia Buriat (bxr), Rusyn (rue), Scots (sco), Sindhi (snd), South Azerbaijani (azb), Tajik (tgk), Veps (vep), Vlaams (vls), Walloon (wln), Wu Chinese (wuu).
| Release | File | SHA256 |
|---|---|---|
| v1.0.0 | model.safetensors |
acb892e488fc5c74ea259caa07d92f09e6cc10dd4d47748fd5f47e86444d882f |
Scores
Text is cut to exactly 10, 50 or 200 characters, joining a sentence to the next where needed. Each competitor got whichever input form scored best for it on that test: as written, lower-cased, or the preprocessing on the OpenLID-v3 card. Their label lists were read from their files, and every model's codes went through one mapping table, training/sources.py. Neither benchmark was used to train the weights.
The table at the top lets each model answer from all its own labels and counts an answer outside the scored languages as wrong.
Closed comparison. The same FLORES-200 text and languages, with every model allowed to answer only among them:
| Model | File | Labels | 10 chars | 50 chars | 200 chars | Licence |
|---|---|---|---|---|---|---|
| Xiangliu 1.0 | 4.8 MB | 284 | 0.747 | 0.946 | 0.977 | MIT |
fastText lid.176.ftz |
0.9 MB | 176 | 0.538 | 0.792 | 0.877 | CC BY-SA 3.0 |
fastText lid.176.bin |
131 MB | 176 | 0.583 | 0.839 | 0.908 | CC BY-SA 3.0 |
GlotLID v4 small .ftz |
35 MB | 2,060 | 0.673 | 0.938 | 0.981 | Apache 2.0 plus notices |
| GlotLID v4 | 783 MB | 2,060 | 0.787 | 0.974 | 0.988 | Apache 2.0 plus notices |
facebook/fasttext-language-identification |
1,176 MB | 218 | 0.718 | 0.971 | 0.987 | CC BY-NC 4.0 |
| OpenLID-v3 | 1,233 MB | 195 | 0.721 | 0.974 | 0.988 | GPL-3.0 |
Wider comparison, at 50 characters. Every FLORES-200 language each pair of models shares, not only the ones all models share:
| Model | Languages shared with Xiangliu | That model | Xiangliu |
|---|---|---|---|
fastText lid.176.ftz |
106 | 0.791 | 0.944 |
fastText lid.176.bin |
106 | 0.837 | 0.944 |
GlotLID v4 small .ftz |
163 | 0.904 | 0.928 |
| GlotLID v4 | 163 | 0.964 | 0.928 |
facebook/fasttext-language-identification |
163 | 0.947 | 0.928 |
| OpenLID-v3 | 163 | 0.962 | 0.928 |
WiLI-2018 test, 106 languages, Wikipedia paragraphs. fastText and OpenLID state that they trained on Wikipedia. Training lines of 20 or more characters matching a sentence of these paragraphs were removed, with other Wikipedia text as described under Data; 0.02% of the paragraphs still hold an exact sentence from the training text, all short ones.
| Model | File | Labels | 10 chars | 50 chars | 200 chars | Licence |
|---|---|---|---|---|---|---|
| Xiangliu 1.0 | 4.8 MB | 284 | 0.617 | 0.866 | 0.940 | MIT |
fastText lid.176.ftz |
0.9 MB | 176 | 0.500 | 0.780 | 0.882 | CC BY-SA 3.0 |
fastText lid.176.bin |
131 MB | 176 | 0.540 | 0.824 | 0.904 | CC BY-SA 3.0 |
GlotLID v4 small .ftz |
35 MB | 2,060 | 0.567 | 0.889 | 0.966 | Apache 2.0 plus notices |
| GlotLID v4 | 783 MB | 2,060 | 0.677 | 0.939 | 0.975 | Apache 2.0 plus notices |
facebook/fasttext-language-identification |
1,176 MB | 218 | 0.630 | 0.927 | 0.965 | CC BY-NC 4.0 |
| OpenLID-v3 | 1,233 MB | 195 | 0.623 | 0.928 | 0.973 | GPL-3.0 |
Speed and memory, 40,000 texts of 50 characters, one thread, 12th Gen Intel(R) Core(TM) i5-12400. The fastText-format models run in C++ through a thin wrapper:
| Model | Texts per second | Memory added by loading | Load seconds |
|---|---|---|---|
| Xiangliu 1.0 (Python) | 8,922 | 6 MB | 0.002 |
fastText lid.176.ftz |
91,698 | 4 MB | 0.012 |
fastText lid.176.bin |
116,167 | 133 MB | 0.096 |
facebook/fasttext-language-identification |
22,541 | 1,154 MB | 0.691 |
GlotLID v4 small .ftz |
58,695 | 42 MB | 0.048 |
| GlotLID v4 | 10,671 | 754 MB | 0.402 |
| OpenLID-v3 | 22,308 | 1,223 MB | 0.847 |
| Xiangliu 1.0 (JavaScript, Node) | 4,357 | not measured | 0.007 |
A second training run with a different seed scored 0.943 on the 50-character FLORES-200 test.
Settings and labels were chosen on the validation text, FLORES-200 dev and WiLI's train split; FLORES-200 dev also set the probability scale. Three earlier builds were also scored on both test splits. The numbers here come from one run after this model file was frozen.
Lowest FLORES-200 test F1 at 50 characters, with where the text went:
| Language | F1 | Guessed instead (rows) |
|---|---|---|
Yue Chinese (yue) |
0.187 | Mandarin Chinese 493, Min Nan Chinese 4 |
Standard Malay (zsm) |
0.426 | Indonesian 538, Balinese 3, Banjar 3 |
Indonesian (ind) |
0.507 | Standard Malay 103, Balinese 2, Estonian 2 |
Banjar (bjn) |
0.624 | Standard Malay 397, Indonesian 329, Arabic 138 |
Balinese (ban) |
0.635 | Indonesian 214, Javanese 75, Standard Malay 25 |
Javanese (jav) |
0.672 | Indonesian 162, Balinese 88, Sundanese 35 |
Rundi (run) |
0.738 | Kinyarwanda 385, Bemba 1, Swahili 1 |
Magahi (mag) |
0.740 | Hindi 380, Mewari 6, Nepali 5 |
Sundanese (sun) |
0.758 | Indonesian 119, Balinese 73, Javanese 31 |
Hindi (hin) |
0.767 | Magahi 19, Mewari 14, Kangri 2 |
Kinyarwanda (kin) |
0.773 | Rundi 41, Swahili 2, English 1 |
Norwegian Bokmål (nob) |
0.776 | Danish 102, Norwegian Nynorsk 58, Swedish 20 |
Mandarin Chinese (cmn) |
0.796 | Yue Chinese 58, Min Nan Chinese 4 |
Norwegian Nynorsk (nno) |
0.812 | Norwegian Bokmål 190, Danish 32, Swedish 24 |
Catalan (cat) |
0.813 | Spanish 1, French 1, Interlingua 1 |
Sango (sag) |
0.830 | Lingala 44, Basa 29, Fon 23 |
Pangasinan (pag) |
0.832 | English 52, Tagalog 38, Catalan 17 |
Crimean Tatar (crh) |
0.834 | Turkish 204, Tatar 43, Northern Uzbek 14 |
Venetian (vec) |
0.835 | Italian 116, Catalan 62, Ligurian 19 |
English (eng) |
0.841 | Catalan 5, Middle English 2, Cebuano 1 |
FLORES-200 languages not scored here, because the model has no label or was not trained in that script: ace_Arab, arb_Latn, awa_Deva, azb_Arab, bho_Deva, bos_Latn, cjk_Latn, dyu_Latn, fuv_Latn, hat_Latn, hne_Deva, kam_Latn, kbp_Latn, kea_Latn, kmb_Latn, knc_Arab, kon_Latn, lim_Latn, lmo_Latn, mai_Deva, min_Arab, mni_Beng, smo_Latn, snd_Arab, taq_Latn, taq_Tfng, tgk_Cyrl, umb_Latn.
What the probability means
Probabilities are scaled by a temperature that grows as text gets shorter, fitted on FLORES-200 dev. On FLORES-200 test text, for languages it has labels for:
| Characters | Accuracy | Mean top probability | Expected calibration error |
|---|---|---|---|
| 10 | 0.678 | 0.636 | 0.043 |
| 50 | 0.928 | 0.926 | 0.004 |
| 200 | 0.976 | 0.972 | 0.004 |
Training
| Item | Value |
|---|---|
| Hardware | NVIDIA GeForce RTX 4060, 12th Gen Intel(R) Core(TM) i5-12400, at home |
| Time | 35.3 minutes for 120,000 steps of 1,024 crops |
| Rental cost | None |
| Random seed | 20261105 |
Each step takes random crops of 3 to 400 characters, so short text is trained on directly. Sampling weights a language by its character count to the power 0.1.
Data
| Source | Licence as released | Sentences, with validation | Version |
|---|---|---|---|
| Tatoeba | CC BY 2.0 FR | 6,545,197 | export of 2026-10-31 |
| Common Voice sentences | CC0 1.0 | 3,810,725 | commit 385a8f4e60b1 |
| SMOL and GATITOS | CC BY 4.0 | 456,394 | revision fdaff3a1a019 |
413,546,959 characters of training text. Text appearing under more than one language was removed, as was any exact match to a FLORES-200 sentence.
Text went in only if its source releases it as CC0 or CC BY and none of it is known to come from text under other terms. Text from Wikimedia projects (CC BY-SA), the European Parliament's proceedings (Europarl, whose terms require citing each source page) and SETimes was kept out three ways:
- By file. Common Voice publishes
server/dataas CC0, but itswiki*files are scraped from Wikipedia, and it also holds Europarl and SETimes files. Those are out, with files from news sites, literature sites and publishers whose permission could not be confirmed. - By content. Any line of 20 or more characters once cleaned, in any source, that matches a line of those files, a sentence of the Europarl corpus in all 21 of its languages, a sentence of WiLI-2018's Wikipedia paragraphs, or a line of a screened file below was removed: 95,523 lines. Common Voice's community files had copied some of them. Shorter lines were not checked.
- By screening. A Common Voice file that matches WiLI's Wikipedia sample 50 or more times per 100,000 of its lines that are that long, with at least 5 matches, was left out whole: 8 files,
cy/sentence-collector.txt,gn/sentence-collector.txt,is/sentence-collector.txt,lg/lug-batch-twenty-seven.txt,mr/sentence-collector.txt,nso/sentence-collector.txt,sw/batch-1-19-11-2021-contributors-set.txt,ug/sentence-collector.txt.
That leaves 36% of Common Voice's sentence files by size. TICO-19 is released as CC0, but its rows translate Wikimedia pages, PubMed Central articles under their own licences, and CMU sentences, so it was not used.
FLORES-200 (CC BY-SA 4.0) and WiLI-2018 (ODbL 1.0) were read to choose settings and to score, and are not redistributed here.
Labels were then removed on selection-split evidence alone (training/prune_labels.py, training/labels_removed.json).
- A label no benchmark tests left if it took in text from tested languages:
aar,ady,ahr,bbc,bew,bfy,bgq,bns,bra,bts,btx,ccp,cha,dje,gos,grt,hil,hoc,hrx,iba,kfy,ktu,lub,mad,mak,nbl,ndc,nde,nnb,ota,pam,pcm,pms,rhg,rom,spv,sus,swc,xsr,zza. - A label left if it stayed under F1 0.5 at 200 characters on every benchmark testing it:
awa,bos,dyu,glk,hne,jam,kaa,lim,lmo,lzh,pcd,snd,wuu,xmf. - A label left if it stayed under F1 0.8 at 200 characters on its own validation text:
dhd,doi,kon,mjl,shy,syl,wbr.
Reproduce
training/README.md lists every download and step. Tatoeba publishes a rolling export, so a later run reads different text. Expect scores near these rather than identical weights.
All languages
F1 at 10, 50 and 200 characters. FLORES-200 and WiLI-2018 are outside tests. "Own text" is validation text held back from the training sources. A blank means that test has no text for that language.
284 languages
| Code | Language | FLORES 10 | FLORES 50 | FLORES 200 | WiLI 10 | WiLI 50 | WiLI 200 | Own text 10 | Own text 50 | Own text 200 |
|---|---|---|---|---|---|---|---|---|---|---|
abk |
Abkhazian | 0.98 | 1.00 | 1.00 | ||||||
abq |
Abaza | 0.85 | 1.00 | 1.00 | ||||||
ace |
Achinese | 0.61 | 0.93 | 1.00 | 0.41 | 0.84 | 0.95 | 0.59 | 1.00 | 1.00 |
ach |
Acoli | 0.68 | 0.97 | 0.98 | ||||||
afr |
Afrikaans | 0.62 | 0.94 | 1.00 | 0.50 | 0.86 | 0.99 | 0.76 | 1.00 | 1.00 |
aii |
Assyrian Neo-Aramaic | 1.00 | 1.00 | 1.00 | ||||||
aka |
Akan | 0.77 | 0.96 | 0.99 | 0.94 | 1.00 | 1.00 | |||
als |
Tosk Albanian | 0.76 | 0.98 | 1.00 | 0.60 | 0.92 | 0.97 | 0.83 | 1.00 | 1.00 |
alt |
Southern Altai | 0.85 | 0.98 | 1.00 | ||||||
alz |
Alur | 0.52 | 0.99 | 1.00 | ||||||
amh |
Amharic | 0.95 | 1.00 | 1.00 | 0.87 | 0.99 | 1.00 | 0.92 | 1.00 | 1.00 |
ang |
Old English | 0.55 | 0.89 | 0.96 | 0.86 | 1.00 | 1.00 | |||
ara |
Arabic | 0.97 | 0.99 | 0.99 | 0.88 | 0.94 | 0.97 | 0.99 | 1.00 | 1.00 |
arn |
Mapudungun | 0.85 | 1.00 | 1.00 | ||||||
asm |
Assamese | 0.89 | 1.00 | 1.00 | 0.88 | 0.98 | 0.98 | 0.95 | 1.00 | 1.00 |
ast |
Asturian | 0.41 | 0.85 | 0.98 | 0.25 | 0.64 | 0.87 | 0.42 | 0.93 | 1.00 |
ava |
Avaric | 0.59 | 0.76 | 0.80 | 0.94 | 1.00 | 1.00 | |||
avk |
Kotava | 0.70 | 1.00 | 1.00 | ||||||
ayr |
Central Aymara | 0.67 | 0.98 | 1.00 | 0.38 | 0.79 | 0.89 | 0.67 | 1.00 | 1.00 |
azj |
North Azerbaijani | 0.69 | 0.98 | 1.00 | 0.59 | 0.94 | 0.99 | 0.84 | 1.00 | 1.00 |
bak |
Bashkir | 0.76 | 0.99 | 1.00 | 0.68 | 0.92 | 0.99 | 0.92 | 1.00 | 1.00 |
bal |
Baluchi | 0.67 | 1.00 | 1.00 | ||||||
bam |
Bambara | 0.62 | 0.93 | 0.95 | 0.57 | 0.99 | 1.00 | |||
ban |
Balinese | 0.27 | 0.64 | 0.80 | 0.39 | 0.97 | 1.00 | |||
bas |
Basa | 0.85 | 0.99 | 1.00 | ||||||
bci |
Baoulé | 0.76 | 1.00 | 1.00 | ||||||
bel |
Belarusian | 0.86 | 1.00 | 1.00 | 0.75 | 0.95 | 0.98 | 0.92 | 1.00 | 1.00 |
bem |
Bemba | 0.72 | 0.96 | 1.00 | 0.72 | 1.00 | 1.00 | |||
ben |
Bengali | 0.88 | 1.00 | 1.00 | 0.81 | 0.93 | 0.97 | 0.94 | 1.00 | 1.00 |
bfq |
Badaga | 0.85 | 1.00 | 1.00 | ||||||
bik |
Bikol | 0.31 | 0.86 | 1.00 | ||||||
bjn |
Banjar | 0.48 | 0.62 | 0.68 | 0.10 | 0.32 | 0.34 | 0.54 | 0.93 | 1.00 |
bod |
Tibetan | 0.71 | 0.89 | 1.00 | 0.87 | 0.94 | 1.00 | 0.75 | 0.94 | 1.00 |
bre |
Breton | 0.52 | 0.88 | 0.98 | 0.86 | 1.00 | 1.00 | |||
brh |
Brahui | 0.62 | 0.98 | 1.00 | ||||||
brx |
Bodo | 0.93 | 1.00 | 1.00 | ||||||
bua |
Buriat | 0.85 | 1.00 | 1.00 | ||||||
bug |
Buginese | 0.51 | 0.87 | 0.97 | 0.57 | 1.00 | 1.00 | |||
bul |
Bulgarian | 0.69 | 0.98 | 1.00 | 0.47 | 0.88 | 0.97 | 0.84 | 1.00 | 1.00 |
bzt |
Brithenig | 0.85 | 1.00 | 1.00 | ||||||
cat |
Catalan | 0.53 | 0.81 | 0.97 | 0.33 | 0.55 | 0.74 | 0.77 | 1.00 | 1.00 |
cbk |
Chavacano | 0.78 | 0.99 | 1.00 | ||||||
ceb |
Cebuano | 0.49 | 0.84 | 0.96 | 0.33 | 0.56 | 0.76 | 0.47 | 0.96 | 1.00 |
ces |
Czech | 0.68 | 0.96 | 1.00 | 0.46 | 0.88 | 0.97 | 0.83 | 0.99 | 1.00 |
cgg |
Chiga | 0.62 | 1.00 | 1.00 | ||||||
che |
Chechen | 0.85 | 0.98 | 1.00 | 0.83 | 1.00 | 1.00 | |||
chk |
Chuukese | 0.68 | 1.00 | 1.00 | ||||||
chv |
Chuvash | 0.58 | 0.91 | 0.98 | 0.93 | 1.00 | 1.00 | |||
ckb |
Central Kurdish | 0.90 | 1.00 | 1.00 | 0.86 | 0.98 | 0.99 | 0.93 | 0.99 | 1.00 |
ckt |
Chukot | 0.98 | 1.00 | 1.00 | ||||||
cmn |
Mandarin Chinese | 0.75 | 0.80 | 0.81 | 0.70 | 0.88 | 0.94 | 0.94 | 1.00 | 1.00 |
cnh |
Hakha Chin | 0.86 | 1.00 | 1.00 | ||||||
cor |
Cornish | 0.37 | 0.72 | 0.96 | 0.90 | 1.00 | 1.00 | |||
crh |
Crimean Tatar | 0.39 | 0.83 | 0.97 | 0.25 | 0.57 | 0.83 | 0.66 | 1.00 | 1.00 |
csb |
Kashubian | 0.46 | 0.85 | 0.96 | 0.85 | 1.00 | 1.00 | |||
ctg |
Chittagonian | 0.58 | 0.87 | 1.00 | ||||||
cym |
Welsh | 0.87 | 0.99 | 1.00 | 0.66 | 0.83 | 0.97 | 0.97 | 1.00 | 1.00 |
dan |
Danish | 0.44 | 0.89 | 0.99 | 0.29 | 0.75 | 0.95 | 0.64 | 0.97 | 1.00 |
dar |
Dargwa | 0.91 | 1.00 | 1.00 | ||||||
deu |
German | 0.65 | 0.93 | 0.99 | 0.43 | 0.68 | 0.85 | 0.89 | 1.00 | 1.00 |
din |
Dinka | 0.74 | 0.97 | 1.00 | 0.96 | 0.99 | 0.98 | |||
div |
Dhivehi | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |||
dov |
Dombe | 0.66 | 0.99 | 1.00 | ||||||
dsb |
Lower Sorbian | 0.48 | 0.68 | 0.80 | 0.79 | 0.98 | 1.00 | |||
dtp |
Kadazan Dusun | 0.75 | 1.00 | 1.00 | ||||||
dzo |
Dzongkha | 0.66 | 0.87 | 1.00 | 0.67 | 0.91 | 1.00 | |||
efi |
Efik | 0.88 | 1.00 | 1.00 | ||||||
ell |
Modern Greek | 0.99 | 1.00 | 1.00 | 0.97 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 |
eng |
English | 0.51 | 0.84 | 0.97 | 0.24 | 0.39 | 0.49 | 0.83 | 1.00 | 1.00 |
enm |
Middle English | 0.58 | 0.98 | 1.00 | ||||||
epo |
Esperanto | 0.69 | 0.97 | 1.00 | 0.42 | 0.85 | 0.93 | 0.85 | 1.00 | 1.00 |
est |
Estonian | 0.68 | 0.95 | 0.99 | 0.47 | 0.82 | 0.92 | 0.82 | 1.00 | 1.00 |
eus |
Basque | 0.61 | 0.98 | 1.00 | 0.31 | 0.89 | 0.97 | 0.82 | 1.00 | 1.00 |
ewe |
Ewe | 0.79 | 0.98 | 1.00 | 0.83 | 1.00 | 1.00 | |||
fao |
Faroese | 0.62 | 0.91 | 0.98 | 0.39 | 0.78 | 0.90 | 0.70 | 0.98 | 1.00 |
fas |
Persian | 0.85 | 1.00 | 1.00 | 0.77 | 0.97 | 0.99 | 0.94 | 1.00 | 1.00 |
fij |
Fijian | 0.75 | 0.95 | 0.99 | 0.63 | 1.00 | 1.00 | |||
fin |
Finnish | 0.64 | 0.98 | 1.00 | 0.42 | 0.76 | 0.94 | 0.88 | 0.99 | 1.00 |
fkv |
Kven Finnish | 0.50 | 0.78 | 1.00 | ||||||
fon |
Fon | 0.85 | 0.98 | 0.97 | 0.93 | 1.00 | 1.00 | |||
fra |
French | 0.63 | 0.95 | 1.00 | 0.39 | 0.66 | 0.76 | 0.85 | 1.00 | 1.00 |
frr |
Northern Frisian | 0.71 | 1.00 | 1.00 | ||||||
fry |
Western Frisian | 0.50 | 0.91 | 0.99 | 0.83 | 0.99 | 1.00 | |||
fur |
Friulian | 0.42 | 0.90 | 1.00 | 0.28 | 0.75 | 0.96 | 0.63 | 1.00 | 1.00 |
gaa |
Ga | 0.86 | 1.00 | 1.00 | ||||||
gaz |
West Central Oromo | 0.75 | 0.99 | 1.00 | 0.60 | 0.95 | 0.99 | 0.77 | 1.00 | 1.00 |
gcf |
Guadeloupean Creole French | 0.89 | 1.00 | 1.00 | ||||||
gla |
Scottish Gaelic | 0.81 | 0.96 | 1.00 | 0.77 | 0.92 | 0.96 | 0.87 | 1.00 | 1.00 |
gle |
Irish | 0.83 | 0.97 | 1.00 | 0.71 | 0.92 | 0.96 | 0.89 | 1.00 | 1.00 |
glg |
Galician | 0.49 | 0.89 | 1.00 | 0.32 | 0.78 | 0.91 | 0.70 | 0.98 | 1.00 |
gom |
Goan Konkani | 0.69 | 0.95 | 1.00 | ||||||
grc |
Ancient Greek | 0.97 | 1.00 | 1.00 | ||||||
grn |
Guarani | 0.74 | 0.98 | 1.00 | 0.67 | 0.91 | 0.98 | 0.89 | 1.00 | 1.00 |
guc |
Wayuu | 0.84 | 1.00 | 1.00 | ||||||
guj |
Gujarati | 1.00 | 1.00 | 1.00 | 0.98 | 0.98 | 0.98 | 1.00 | 1.00 | 1.00 |
hac |
Gurani | 0.57 | 0.95 | 0.98 | ||||||
hau |
Hausa | 0.68 | 0.96 | 1.00 | 0.50 | 0.92 | 0.99 | 0.89 | 1.00 | 1.00 |
heb |
Hebrew | 0.93 | 1.00 | 1.00 | 0.87 | 0.96 | 1.00 | 0.99 | 1.00 | 1.00 |
hin |
Hindi | 0.51 | 0.77 | 0.95 | 0.46 | 0.79 | 0.95 | 0.88 | 1.00 | 1.00 |
hrv |
Croatian | 0.56 | 0.89 | 0.98 | 0.45 | 0.76 | 0.94 | 0.49 | 0.71 | 0.99 |
hsb |
Upper Sorbian | 0.47 | 0.73 | 0.81 | 0.83 | 1.00 | 1.00 | |||
hun |
Hungarian | 0.78 | 0.99 | 1.00 | 0.47 | 0.93 | 0.98 | 0.93 | 1.00 | 1.00 |
hye |
Armenian | 1.00 | 1.00 | 1.00 | 0.97 | 0.98 | 0.99 | 1.00 | 1.00 | 1.00 |
ibo |
Igbo | 0.84 | 0.98 | 1.00 | 0.49 | 0.85 | 0.86 | 0.94 | 1.00 | 1.00 |
ido |
Ido | 0.44 | 0.87 | 0.97 | 0.74 | 0.99 | 1.00 | |||
iku |
Canadian Inuktut | 0.92 | 1.00 | 1.00 | ||||||
ile |
Interlingue | 0.30 | 0.73 | 0.91 | 0.73 | 0.99 | 1.00 | |||
ilo |
Iloko | 0.67 | 0.96 | 1.00 | 0.60 | 0.87 | 0.93 | 0.63 | 1.00 | 1.00 |
ina |
Interlingua | 0.43 | 0.80 | 0.97 | 0.76 | 1.00 | 1.00 | |||
ind |
Indonesian | 0.37 | 0.51 | 0.57 | 0.36 | 0.47 | 0.52 | 0.78 | 0.95 | 1.00 |
isl |
Icelandic | 0.70 | 0.93 | 0.98 | 0.59 | 0.86 | 0.93 | 0.93 | 1.00 | 1.00 |
iso |
Isoko | 0.82 | 1.00 | 1.00 | ||||||
isv |
Interslavic | 0.49 | 0.98 | 1.00 | ||||||
ita |
Italian | 0.46 | 0.89 | 0.98 | 0.30 | 0.72 | 0.87 | 0.78 | 1.00 | 1.00 |
jav |
Javanese | 0.26 | 0.67 | 0.87 | 0.18 | 0.63 | 0.88 | 0.84 | 1.00 | 1.00 |
jbo |
Lojban | 0.75 | 0.96 | 0.99 | 0.91 | 1.00 | 1.00 | |||
jdt |
Judeo-Tat | 0.85 | 1.00 | 1.00 | ||||||
jpn |
Japanese | 0.99 | 1.00 | 1.00 | 0.91 | 0.98 | 1.00 | 1.00 | 1.00 | 1.00 |
kab |
Kabyle | 0.71 | 0.99 | 1.00 | 0.52 | 0.89 | 0.97 | 0.92 | 1.00 | 1.00 |
kac |
Kachin | 0.72 | 0.98 | 1.00 | 0.83 | 1.00 | 1.00 | |||
kal |
Kalaallisut | 0.78 | 1.00 | 1.00 | ||||||
kan |
Kannada | 0.85 | 0.98 | 1.00 | 0.67 | 0.90 | 0.97 | 0.79 | 0.98 | 1.00 |
kas |
Kashmiri | 0.66 | 0.95 | 0.99 | 0.75 | 0.99 | 1.00 | |||
kat |
Georgian | 1.00 | 1.00 | 1.00 | 0.97 | 0.98 | 1.00 | 1.00 | 1.00 | 1.00 |
kaz |
Kazakh | 0.80 | 0.99 | 1.00 | 0.64 | 0.88 | 1.00 | 0.94 | 1.00 | 1.00 |
kbd |
Kabardian | 0.77 | 0.95 | 1.00 | 0.95 | 1.00 | 1.00 | |||
kek |
Kekchí | 0.70 | 1.00 | 1.00 | ||||||
kha |
Khasi | 0.88 | 1.00 | 1.00 | ||||||
khk |
Halh Mongolian | 0.82 | 1.00 | 1.00 | 0.63 | 0.94 | 1.00 | 0.91 | 1.00 | 1.00 |
khm |
Khmer | 0.99 | 1.00 | 1.00 | 0.93 | 0.93 | 0.95 | 1.00 | 1.00 | 1.00 |
kik |
Kikuyu | 0.82 | 0.96 | 0.97 | 0.87 | 1.00 | 1.00 | |||
kin |
Kinyarwanda | 0.54 | 0.77 | 0.88 | 0.53 | 0.87 | 0.88 | 0.78 | 0.97 | 1.00 |
kir |
Kirghiz | 0.50 | 0.89 | 0.99 | 0.36 | 0.73 | 0.87 | 0.73 | 0.99 | 1.00 |
kjh |
Khakas | 0.74 | 0.99 | 1.00 | ||||||
kmr |
Northern Kurdish | 0.82 | 0.99 | 1.00 | 0.69 | 0.96 | 0.99 | 0.93 | 1.00 | 1.00 |
knc |
Central Kanuri | 0.71 | 0.97 | 0.99 | 0.77 | 1.00 | 1.00 | |||
kom |
Komi | 0.53 | 0.77 | 0.83 | 0.91 | 1.00 | 1.00 | |||
kor |
Korean | 0.99 | 1.00 | 1.00 | 0.98 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 |
kri |
Krio | 0.75 | 0.99 | 1.00 | ||||||
kru |
Kurukh | 0.67 | 1.00 | 1.00 | ||||||
lad |
Ladino | 0.27 | 0.66 | 0.78 | 0.45 | 0.95 | 1.00 | |||
lao |
Lao | 0.98 | 1.00 | 1.00 | 0.93 | 0.93 | 0.94 | 1.00 | 1.00 | 1.00 |
lat |
Latin | 0.27 | 0.65 | 0.90 | 0.83 | 1.00 | 1.00 | |||
lep |
Lepcha | 1.00 | 1.00 | 1.00 | ||||||
lfn |
Lingua Franca Nova | 0.80 | 1.00 | 1.00 | ||||||
lif |
Limbu | 0.99 | 1.00 | 1.00 | ||||||
lij |
Ligurian | 0.61 | 0.96 | 1.00 | 0.36 | 0.80 | 0.95 | 0.71 | 1.00 | 1.00 |
lin |
Lingala | 0.70 | 0.95 | 0.97 | 0.43 | 0.84 | 0.94 | 0.79 | 1.00 | 1.00 |
lit |
Lithuanian | 0.70 | 0.98 | 1.00 | 0.46 | 0.92 | 0.97 | 0.92 | 1.00 | 1.00 |
ltg |
Latgalian | 0.66 | 0.96 | 1.00 | 0.48 | 0.86 | 0.97 | 0.64 | 0.96 | 1.00 |
ltz |
Luxembourgish | 0.48 | 0.87 | 0.98 | 0.39 | 0.75 | 0.89 | 0.81 | 1.00 | 1.00 |
lua |
Luba-Lulua | 0.61 | 0.88 | 0.96 | 0.66 | 0.98 | 1.00 | |||
lug |
Ganda | 0.69 | 0.98 | 1.00 | 0.74 | 0.97 | 0.98 | 0.89 | 1.00 | 1.00 |
luo |
Luo | 0.65 | 0.98 | 1.00 | 0.71 | 0.98 | 1.00 | |||
lus |
Lushai | 0.52 | 0.91 | 1.00 | 0.63 | 1.00 | 1.00 | |||
lvs |
Standard Latvian | 0.72 | 0.96 | 1.00 | 0.56 | 0.86 | 0.96 | 0.92 | 1.00 | 1.00 |
mag |
Magahi | 0.38 | 0.74 | 0.95 | 0.49 | 0.95 | 1.00 | |||
mah |
Marshallese | 0.71 | 1.00 | 1.00 | ||||||
mal |
Malayalam | 1.00 | 1.00 | 1.00 | 0.99 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 |
mam |
Mam | 0.79 | 1.00 | 1.00 | ||||||
mar |
Marathi | 0.65 | 0.97 | 1.00 | 0.49 | 0.86 | 0.97 | 0.95 | 1.00 | 1.00 |
mdf |
Moksha | 0.47 | 0.85 | 0.96 | 0.84 | 1.00 | 1.00 | |||
mfe |
Morisyen | 0.47 | 0.99 | 1.00 | ||||||
mhr |
Eastern Mari | 0.41 | 0.64 | 0.90 | 0.92 | 1.00 | 1.00 | |||
min |
Minangkabau | 0.45 | 0.84 | 0.96 | 0.24 | 0.67 | 0.53 | 0.40 | 0.97 | 1.00 |
mkd |
Macedonian | 0.66 | 0.97 | 1.00 | 0.58 | 0.92 | 0.98 | 0.85 | 1.00 | 1.00 |
mlt |
Maltese | 0.76 | 0.98 | 1.00 | 0.64 | 0.91 | 0.99 | 0.86 | 1.00 | 1.00 |
mni |
Manipuri | 1.00 | 1.00 | 1.00 | ||||||
mos |
Mossi | 0.69 | 0.98 | 0.99 | 0.76 | 1.00 | 1.00 | |||
mri |
Maori | 0.87 | 0.97 | 1.00 | 0.54 | 0.88 | 0.99 | 0.88 | 1.00 | 1.00 |
mrj |
Western Mari | 0.35 | 0.75 | 0.98 | 0.92 | 1.00 | 1.00 | |||
mtr |
Mewari | 0.62 | 0.97 | 1.00 | ||||||
mus |
Creek | 0.92 | 1.00 | 1.00 | ||||||
mwr |
Marwari | 0.65 | 0.97 | 1.00 | ||||||
mya |
Burmese | 0.99 | 1.00 | 1.00 | 0.97 | 0.99 | 1.00 | 0.99 | 1.00 | 1.00 |
myv |
Erzya | 0.42 | 0.67 | 0.81 | 0.85 | 1.00 | 1.00 | |||
nan |
Min Nan Chinese | 0.46 | 0.54 | 0.69 | 0.98 | 1.00 | 1.00 | |||
nav |
Navajo | 0.95 | 1.00 | 1.00 | 0.96 | 1.00 | 1.00 | |||
nds |
Low German | 0.41 | 0.84 | 0.97 | 0.85 | 1.00 | 1.00 | |||
new |
Nepal Bhasa | 0.64 | 0.55 | 0.93 | 0.78 | 1.00 | 1.00 | |||
nhe |
Eastern Huasteca Nahuatl | 0.81 | 1.00 | 1.00 | ||||||
nld |
Dutch | 0.63 | 0.95 | 1.00 | 0.32 | 0.80 | 0.97 | 0.83 | 1.00 | 1.00 |
nno |
Norwegian Nynorsk | 0.43 | 0.81 | 0.96 | 0.25 | 0.74 | 0.94 | 0.69 | 0.97 | 1.00 |
nob |
Norwegian Bokmål | 0.39 | 0.78 | 0.95 | 0.29 | 0.68 | 0.92 | 0.55 | 0.92 | 1.00 |
nov |
Novial | 0.45 | 0.98 | 1.00 | ||||||
npi |
Nepali | 0.64 | 0.97 | 1.00 | 0.40 | 0.79 | 0.97 | 0.87 | 1.00 | 1.00 |
nqo |
N'Ko | 1.00 | 1.00 | 1.00 | ||||||
nso |
Pedi | 0.56 | 0.95 | 1.00 | 0.34 | 0.82 | 0.97 | 0.55 | 0.93 | 1.00 |
nst |
Tase Naga | 0.97 | 1.00 | 1.00 | ||||||
nus |
Nuer | 0.90 | 0.99 | 1.00 | 0.99 | 1.00 | 1.00 | |||
nya |
Chichewa | 0.60 | 0.95 | 1.00 | 0.48 | 0.98 | 1.00 | |||
oci |
Occitan | 0.53 | 0.93 | 0.99 | 0.30 | 0.64 | 0.85 | 0.80 | 0.99 | 1.00 |
orv |
Old Russian | 0.94 | 1.00 | 1.00 | ||||||
ory |
Odia | 1.00 | 1.00 | 1.00 | 0.97 | 0.97 | 0.98 | 1.00 | 1.00 | 1.00 |
oss |
Iron Ossetic | 0.70 | 0.94 | 1.00 | 0.97 | 1.00 | 1.00 | |||
pag |
Pangasinan | 0.57 | 0.83 | 0.94 | 0.54 | 0.81 | 0.89 | 0.37 | 1.00 | 1.00 |
pan |
Panjabi | 1.00 | 1.00 | 1.00 | 0.99 | 0.99 | 1.00 | 1.00 | 1.00 | 1.00 |
pap |
Papiamento | 0.39 | 0.87 | 0.98 | 0.16 | 0.52 | 0.68 | 0.35 | 1.00 | 1.00 |
pau |
Palauan | 0.88 | 1.00 | 1.00 | ||||||
pbt |
Southern Pashto | 0.85 | 0.99 | 1.00 | 0.74 | 0.96 | 0.97 | 0.90 | 1.00 | 1.00 |
plt |
Plateau Malagasy | 0.83 | 0.99 | 1.00 | 0.62 | 0.95 | 1.00 | 0.72 | 1.00 | 1.00 |
pnb |
Western Panjabi | 0.46 | 0.78 | 0.91 | 0.38 | 0.89 | 1.00 | |||
pol |
Polish | 0.66 | 0.94 | 0.99 | 0.49 | 0.73 | 0.76 | 0.90 | 1.00 | 1.00 |
por |
Portuguese | 0.53 | 0.93 | 1.00 | 0.34 | 0.77 | 0.91 | 0.77 | 0.99 | 1.00 |
prg |
Prussian | 0.92 | 1.00 | 1.00 | ||||||
quc |
K'iche' | 0.77 | 1.00 | 1.00 | ||||||
quy |
Ayacucho Quechua | 0.69 | 0.98 | 1.00 | 0.46 | 0.84 | 0.92 | 0.75 | 1.00 | 1.00 |
rif |
Tarifit | 0.78 | 0.97 | 1.00 | ||||||
roh |
Romansh | 0.41 | 0.76 | 0.97 | 0.78 | 1.00 | 1.00 | |||
ron |
Romanian | 0.72 | 0.98 | 1.00 | 0.46 | 0.90 | 0.98 | 0.89 | 1.00 | 1.00 |
run |
Rundi | 0.46 | 0.74 | 0.86 | 0.59 | 0.81 | 0.99 | |||
rus |
Russian | 0.68 | 0.98 | 1.00 | 0.40 | 0.62 | 0.73 | 0.85 | 1.00 | 1.00 |
sag |
Sango | 0.61 | 0.83 | 0.88 | 0.80 | 1.00 | 1.00 | |||
sah |
Yakut | 0.76 | 0.96 | 0.99 | 0.94 | 1.00 | 1.00 | |||
san |
Sanskrit | 0.65 | 0.96 | 1.00 | 0.65 | 0.81 | 0.97 | 0.80 | 1.00 | 1.00 |
sat |
Santali | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |||
scn |
Sicilian | 0.55 | 0.96 | 1.00 | 0.45 | 0.91 | 0.98 | 0.58 | 1.00 | 1.00 |
sdh |
Southern Kurdish | 0.76 | 0.98 | 1.00 | ||||||
shi |
Tachelhit | 0.93 | 1.00 | 1.00 | ||||||
shn |
Shan | 0.97 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |||
sin |
Sinhala | 1.00 | 1.00 | 1.00 | 0.97 | 0.97 | 0.98 | 1.00 | 1.00 | 1.00 |
skr |
Saraiki | 0.85 | 1.00 | 1.00 | ||||||
slk |
Slovak | 0.63 | 0.94 | 1.00 | 0.41 | 0.79 | 0.95 | 0.80 | 0.99 | 1.00 |
slv |
Slovenian | 0.53 | 0.95 | 1.00 | 0.34 | 0.84 | 0.97 | 0.64 | 0.96 | 1.00 |
sme |
Northern Sami | 0.46 | 0.85 | 0.91 | 0.84 | 1.00 | 1.00 | |||
sna |
Shona | 0.69 | 0.98 | 1.00 | 0.55 | 0.89 | 0.95 | 0.51 | 0.99 | 1.00 |
som |
Somali | 0.78 | 0.99 | 1.00 | 0.61 | 0.92 | 0.96 | 0.71 | 1.00 | 1.00 |
sot |
Southern Sotho | 0.59 | 0.94 | 1.00 | 0.72 | 0.98 | 1.00 | |||
spa |
Spanish | 0.40 | 0.86 | 0.98 | 0.25 | 0.61 | 0.75 | 0.71 | 0.99 | 1.00 |
srd |
Sardinian | 0.72 | 0.98 | 1.00 | 0.52 | 0.86 | 0.97 | 0.72 | 1.00 | 1.00 |
srp |
Serbian | 0.64 | 0.92 | 0.98 | 0.47 | 0.77 | 0.94 | 0.72 | 0.95 | 1.00 |
ssw |
Swati | 0.66 | 0.95 | 1.00 | 0.73 | 0.98 | 1.00 | |||
sun |
Sundanese | 0.37 | 0.76 | 0.94 | 0.14 | 0.81 | 0.93 | 0.66 | 1.00 | 1.00 |
swe |
Swedish | 0.57 | 0.96 | 1.00 | 0.33 | 0.85 | 0.98 | 0.82 | 1.00 | 1.00 |
swg |
Swabian | 0.66 | 0.99 | 1.00 | ||||||
swh |
Swahili | 0.63 | 0.90 | 0.97 | 0.57 | 0.87 | 0.92 | 0.83 | 1.00 | 1.00 |
szl |
Silesian | 0.67 | 0.94 | 0.99 | 0.26 | 0.47 | 0.42 | 0.73 | 0.99 | 1.00 |
tah |
Tahitian | 0.78 | 1.00 | 1.00 | ||||||
tam |
Tamil | 0.94 | 0.99 | 1.00 | 0.93 | 0.98 | 1.00 | 0.96 | 1.00 | 1.00 |
tat |
Tatar | 0.66 | 0.94 | 0.99 | 0.54 | 0.73 | 0.84 | 0.87 | 1.00 | 1.00 |
tcy |
Tulu | 0.73 | 0.91 | 0.98 | 0.83 | 0.95 | 1.00 | |||
tel |
Telugu | 0.99 | 1.00 | 1.00 | 0.97 | 0.97 | 0.98 | 1.00 | 1.00 | 1.00 |
tet |
Tetum | 0.33 | 0.77 | 0.87 | 0.36 | 0.97 | 1.00 | |||
tgl |
Tagalog | 0.59 | 0.86 | 0.95 | 0.47 | 0.71 | 0.81 | 0.85 | 1.00 | 1.00 |
tha |
Thai | 0.99 | 1.00 | 1.00 | 0.95 | 0.95 | 0.95 | 1.00 | 1.00 | 1.00 |
tig |
Tigre | 0.98 | 1.00 | 1.00 | ||||||
tir |
Tigrinya | 0.95 | 1.00 | 1.00 | 0.93 | 1.00 | 1.00 | |||
tiv |
Tiv | 0.71 | 0.99 | 1.00 | ||||||
tlh |
Klingon | 0.93 | 1.00 | 1.00 | ||||||
tok |
Toki Pona | 0.98 | 1.00 | 1.00 | ||||||
ton |
Tonga | 0.81 | 0.98 | 0.99 | 0.73 | 1.00 | 1.00 | |||
tpi |
Tok Pisin | 0.73 | 0.96 | 0.99 | 0.53 | 0.98 | 1.00 | |||
trp |
Kok Borok | 0.74 | 1.00 | 1.00 | ||||||
tsn |
Tswana | 0.54 | 0.92 | 1.00 | 0.48 | 0.81 | 0.98 | 0.61 | 0.96 | 1.00 |
tso |
Tsonga | 0.74 | 0.98 | 1.00 | 0.76 | 0.99 | 1.00 | |||
ttq |
Tawallammat Tamajaq | 0.77 | 1.00 | 1.00 | ||||||
tuk |
Turkmen | 0.72 | 0.99 | 1.00 | 0.67 | 0.97 | 1.00 | 0.88 | 1.00 | 1.00 |
tum |
Tumbuka | 0.64 | 0.96 | 1.00 | 0.60 | 0.98 | 1.00 | |||
tur |
Turkish | 0.55 | 0.88 | 0.97 | 0.38 | 0.78 | 0.91 | 0.86 | 1.00 | 1.00 |
tyv |
Tuvinian | 0.63 | 0.91 | 0.98 | 0.76 | 0.99 | 1.00 | |||
udm |
Udmurt | 0.55 | 0.84 | 0.97 | 0.90 | 1.00 | 1.00 | |||
uig |
Uighur | 0.97 | 1.00 | 1.00 | 0.91 | 0.98 | 0.98 | 1.00 | 1.00 | 1.00 |
ukr |
Ukrainian | 0.80 | 1.00 | 1.00 | 0.64 | 0.93 | 0.99 | 0.90 | 1.00 | 1.00 |
unr |
Mundari | 0.63 | 0.99 | 1.00 | ||||||
urd |
Urdu | 0.72 | 0.97 | 1.00 | 0.58 | 0.81 | 0.91 | 0.90 | 0.99 | 1.00 |
uzn |
Northern Uzbek | 0.57 | 0.94 | 0.99 | 0.32 | 0.82 | 0.93 | 0.85 | 1.00 | 1.00 |
vec |
Venetian | 0.37 | 0.83 | 0.96 | 0.30 | 0.70 | 0.86 | 0.39 | 0.98 | 1.00 |
ven |
Venda | 0.86 | 1.00 | 1.00 | ||||||
vie |
Vietnamese | 0.95 | 1.00 | 1.00 | 0.86 | 0.97 | 0.98 | 0.99 | 1.00 | 1.00 |
vol |
Volapük | 0.50 | 0.94 | 0.98 | 0.92 | 1.00 | 1.00 | |||
vot |
Votic | 0.95 | 1.00 | 1.00 | ||||||
war |
Waray | 0.58 | 0.93 | 0.99 | 0.04 | 0.00 | 0.65 | 0.54 | 0.97 | 1.00 |
wol |
Wolof | 0.71 | 0.97 | 1.00 | 0.63 | 0.92 | 0.96 | 0.63 | 1.00 | 1.00 |
xal |
Kalmyk | 0.86 | 1.00 | 1.00 | ||||||
xho |
Xhosa | 0.56 | 0.87 | 0.99 | 0.58 | 0.87 | 0.96 | 0.58 | 0.94 | 1.00 |
xnr |
Kangri | 0.58 | 0.99 | 1.00 | ||||||
ydd |
Eastern Yiddish | 0.91 | 1.00 | 1.00 | 0.82 | 0.95 | 0.99 | 1.00 | 1.00 | 1.00 |
yor |
Yoruba | 0.85 | 0.91 | 0.99 | 0.52 | 0.63 | 0.71 | 0.98 | 1.00 | 1.00 |
yua |
Yucateco | 0.70 | 1.00 | 1.00 | ||||||
yue |
Yue Chinese | 0.36 | 0.19 | 0.03 | 0.72 | 0.82 | 0.91 | 0.94 | 1.00 | 1.00 |
zap |
Zapotec | 0.77 | 1.00 | 1.00 | ||||||
zgh |
Standard Moroccan Tamazight | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | |||
zsm |
Standard Malay | 0.27 | 0.43 | 0.49 | 0.24 | 0.48 | 0.55 | 0.58 | 0.85 | 0.99 |
zul |
Zulu | 0.55 | 0.87 | 0.99 | 0.56 | 0.94 | 1.00 |
Licence and credit
Code and weights are MIT, © 2026 Draconitic Labs. NOTICE.md gives the credit the training text's licences ask for, and TATOEBA-CONTRIBUTORS.txt names the Tatoeba contributors.
The name is that of the only known moon of the dwarf planet Gonggong. In Chinese myth, Xiangliu is Gonggong's nine-headed serpent minister.
@misc{xiangliu2026,
title = {Xiangliu: a language identifier under 5 MB},
author = {{Draconitic Labs}},
year = {2026},
url = {https://huggingface.co/DraconiticLabs/xiangliu}
}
- Downloads last month
- 30