mitra-qwen35-2b-embedder-sentence-alignment

A variant of mitra-qwen35-2b-embedder specialised for sentence alignment of Buddhist texts and their translations (e.g. as the scoring model in a vecalign-style aligner). It is trained to give the highest similarity to the exact translation of a sentence, and lower similarity to spans that contain the translation plus neighbouring material.

Use the base embedder for general retrieval. Use this model for alignment.

Training

Both stages start from the same Qwen3.5-2B base as the general embedder: Buddhist-domain continued pretraining followed by stage-2 SFT. Both stages train the same LoRA adapter (r=32, alpha 64, all attention and MLP projections), which is merged into this checkpoint.

  1. Parallel-sentence contrastive stage. 8.8M strictly parallel sentence pairs over 27 language pairs (Sanskrit, Tibetan, Chinese and Pāli with each other and with modern languages), decontaminated against the evaluation sets. Symmetric InfoNCE with in-batch negatives only (512 pairs per step), temperature 0.02, lr 5e-5, 4000 steps.
  2. Span-precision stage. Continues the stage-1 adapter for 1500 steps on runs of consecutive sentence pairs drawn from the same data. Each query's positive is its exact translation. Hard negatives are the same translation padded with the preceding and/or following sentence, on both the target side and the query side. 20% of the units are two-sentence pairs whose single-sentence halves serve as negatives, so the model does not simply prefer shorter spans. Batches hold 256 units, plus all of their hard negatives. Temperature 0.02, lr 5e-5.

Difference from the general embedder. The released mitra-qwen35-2b-embedder (v2) is trained on a mix of retrieval data with mined hard negatives. This model sees only parallel sentence pairs and span-level negatives. It is much better at choosing the exact sentence span. It is weaker on loose, long-range matches such as commentary retrieval and question answering.

Evaluation (vs. mitra-qwen35-2b-embedder v2)

this model v2
Span precision: exact translation beats padded variants (5,100 items, 34 directions) 0.950 0.766
Complete translation beats partial or over-long spans 0.908 0.735
Document alignment, link F1 (vecalign, 18 language pairs, 6 noise variants) 0.942 0.913
Document alignment, hardest variant (merges + distractors + drift), link F1 0.851 0.753
Internal quick suite R@1: multilingual parallels / English→classical 0.857 / 0.954 0.838 / 0.954
Internal quick suite R@1: verse→commentary / question→passage 0.450 / 0.558 0.639 / 0.657

In the document alignment evaluation, the deletion penalty is chosen per model by cross-validation. With this model, a fixed deletion penalty of about 0.26–0.30 (in vecalign's normalized cost units) works much better than vecalign's percentile-based default. Use 0.26 for noisy documents.

Usage

Same prompting and pooling as the general 2B embedder. Source sentences get the instruction, and target sentences are raw text. Tibetan in native script, Sanskrit and Pāli in IAST.

import torch
from transformers import AutoModel, AutoTokenizer

name = "buddhist-nlp/mitra-qwen35-2b-embedder-sentence-alignment"
tok = AutoTokenizer.from_pretrained(name)
tok.padding_side = "left"
model = AutoModel.from_pretrained(name, dtype=torch.bfloat16, device_map="cuda").eval()

def embed(texts, target_lang=None, batch_size=32):
    if target_lang:  # source side
        texts = [f"<instruct>Please find the semantically most similar text in {target_lang}.\n<query>{t}"
                 for t in texts]
    out = []
    with torch.no_grad():
        for i in range(0, len(texts), batch_size):
            enc = tok(texts[i:i+batch_size], padding=True, truncation=True,
                      max_length=512, return_tensors="pt").to(model.device)
            h = model(**enc).last_hidden_state[:, -1]
            out.append(torch.nn.functional.normalize(h.float(), dim=-1).cpu())
    return torch.cat(out)

src = embed(["...Tibetan sentences..."], target_lang="English")
tgt = embed(["...English sentences..."])

EOS is appended by the bundled tokenizer. Pooling is last-token, L2-normalized, dimension 2048.

Citation

If you use this model, please cite the Dharmamitra project (https://dharmamitra.org). A technical report is in preparation.

Downloads last month
90
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including buddhist-nlp/mitra-qwen35-2b-embedder-sentence-alignment