Kraken base models: OCR/HTR and segmentation
Collection
Hub mirrors of Benjamin Kiessling's kraken base models: PP-OCRv6 line recognisers, BLLA segmentation and Party. Byte-identical to Zenodo. • 5 items • Updated • 10
Llama Party: multilingual page-wise text recognition base model
Hugging Face Hub mirror
This repository mirrors the canonical Zenodo release by Benjamin Kiessling, ALMAnaCH, Inria Paris. The files are byte-identical to the Zenodo release and have not been modified by Small Models for GLAM.
Use from the Hugging Face Hub
Party needs kraken plus the party package from GitHub
(the Party project on PyPI is unrelated).
pip install "kraken>=7.0" huggingface_hub git+https://github.com/mittagessen/party
hf download small-models-for-glam/kraken-party-base model.safetensors --local-dir ./kraken-party-base
# input.xml: a segmented page (ALTO/PAGE), e.g. from `kraken segment -bl`
kraken -f alto -a -i input.xml out.xml ocr -m -B 32 ./kraken-party-base/model.safetensors
Party is page-wise recognition of text-y. It is a replacement for conventional text recognizers in automatic text recognition pipelines that utilize either bounding box or baseline+bounding polygon segmentation methods for layout analysis.
Llama party is a full-page generative text recognizer that has been pretrained on a large corpus of multilingual historical, contemporary, and born-digital document page images, both handwritten and machine-printed.
Architecture
The recognizer is a deep fusion multimodal model consisting of a Swin vision encoder and a tiny Llama (40m parameters) decoder trained with octet tokenization. The network is prompted with the line positions through positional embeddings added to the encoder hidden state. Line prompts can be encoded either as bounding boxes or as baseline+bounding curves, so the model is compatible with both segmentation paradigms.
During training the encoder weights were initialized with an ImageNet-22k pretrained Swin-base from pytorch-image-models, the decoder weights came from a custom Llama 3.2 pretrained on the historical and cultural subset of the Common Corpus tokenized with a ByT5-style octet tokenizer.
Uses
Llama party is a recognition foundation model primarily targeted at automatic text recognition for the humanities. While it produces fairly accurate output on an impressive range of material it is intended to be fine-tuned on some target dataset to ensure compliance with desired transcription guidelines.
Transcription guidelines, Normalization, and Transformations
No attempts have been made to normalize the datasets or to only use data adhering to common transcription guidelines. While some subsets of the corpus are internally consistent, only a very small subset of the languages in the training data only contain datasets from a single source.
Text was normalized to Unicode NFD and whitespace was normalized during training.
Bias, Risks, and Limitations
The training corpus is heavily skewed towards a couple of languages (English, French, German, Latin, and Middle French) and frequently incorporates datasets of esoteric material transcribed for specific purposes. Especially machine-printed and born-digital material lack diversity, so error rates will most likely vary considerably across languages and document type.
Some additional limitations are to be expected:
- For a number of languages the training material is small and/or comes from a single source (e.g. Ancient Greek, Finnish, Georgian, Malayalam, Slovak), resulting in markedly higher error rates that will require fine-tuning.
- Some transcriptions resolved abbreviations while others did not. Inconsistent output is to be expected, in particular for European manuscripts in Latin script.
- As the model predicts 8-bit UTF-8 code units directly the lack of consistent Unicode normalization can cause slightly different code point streams during prediction.
How to Get Started with the Model
Install kraken through pip and the party package from
github and follow the instructions.
Inference is handled through kraken:
$ kraken -f alto -a -i input.xml out.xml ocr -m -B 32 ~/model.safetensors
Training Details
Training Data
The model has been pretrained on a large collection of publicly available ATR datasets, in addition to a number of restricted datasets.
Training Procedure and Hyperparameters
- Training regime: 5 × A40 GPU, BF16 mixed precision, Muon optimizer (with an auxiliary AdamW group for embeddings/heads), batch size 16, gradient accumulation 16, cosine schedule with 10000 iteration warmup, max LR 3e-4, min LR 1e-5, weight decay 1e-5, gradient clipping 1.0, NFD normalization, augmentation enabled, random sampling of bounding box and curve line prompts.
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was evaluated on held-out test splits drawn from the training data sources.
Metrics
| Language | CER |
|---|---|
| Ancient Greek | 40.76% |
| Arabic | 7.34% |
| Catalan | 5.92% |
| Church Slavonic | 7.30% |
| Corsican | 7.48% |
| Czech | 11.03% |
| Dutch | 8.14% |
| English | 7.63% |
| Finnish | 38.16% |
| French | 7.49% |
| Georgian | 20.70% |
| German | 9.10% |
| Hebrew | 12.23% |
| Italian | 2.81% |
| Latin | 7.47% |
| Malayalam | 39.53% |
| Middle Dutch | 13.31% |
| Middle French | 1.47% |
| Multilingual | 6.27% |
| Norwegian | 1.51% |
| Ottoman Turkish | 6.77% |
| Persian | 5.41% |
| Portuguese | 7.53% |
| Russian | 7.99% |
| Shorthand German | 4.22% |
| Slovak | 19.59% |
| Spanish | 6.07% |
| Swedish | 6.10% |
| Syriac | 5.13% |
| Ukrainian | 8.21% |
| Urdu | 3.91% |
| Yiddish | 1.79% |
| Aggregate (micro-average) | 10.16% |
Citation
Model tree for small-models-for-glam/kraken-party-base
Base model
mittagessen/bytellama-43m-cc