Add more info about zero-shot text classification and validation process

README.md

@@ -16,30 +16,60 @@ pipeline_tag: zero-shot-classification
 
 This is a model with 155M parameters that is build on top of the [USER2-base](https://huggingface.co/deepvk/USER2-base) sentence encoder (149M) and is fine-tuned for zero-shot classification task.
 
+What is Zero‑Shot Classification?
+
+Zero‑shot text classification lets a model assign user‑supplied labels to a text without seeing any training examples for those labels. At inference you simply provide the candidate labels as strings, and the model chooses the most appropriate one.
+
 ## Performance
 
 To evaluate the model, we measure quality on multiclass classification tasks from the `MTEB-rus` benchmark.
 
 **MTEB-rus**
 
-| Model  | Size  | Hidden Dim | Context Length  | Mean(task) | Kinopoisk | Headlines | GRNTI | OECD | Inappropriateness |
-|----------------------------------------------------------------------------------------------:|:-----:|:----------:|:--------------:|:----------:|:--------------:|:-------------:|:----------:|:------------------------:|:-----------------:|
-| `GeRaCl-USER2-base`                      | 155M | 768   | 8192  | 0.65 | 0.61 | 0.80 | 0.63 | 0.48 | 0.71 |
-| `USER2-base`                     | 149M | 768   | 8192 | 0.52 | 0.50 | 0.65 | 0.56 | 0.39 | 0.51 |
-| `USER-bge-m3`                     | 359M | 1024   | 8192 | 0.53 | 0.60 | 0.73 | 0.43 | 0.28 | 0.62 |
-| `multilingual-e5-large-instruct` | 560M | 1024  | 512  | 0.63 | 0.56 | 0.83 | 0.62 | 0.46 | 0.67 |
-| `mDeBERTa-v3-base-mnli-xnli`             | 279M | 768  | 512 | 0.45 | 0.54 | 0.53 | 0.34 | 0.23 | 0.62 |
-| `bge-m3-zeroshot-v2.0`                 | 568M | 1024  | 8192  | 0.60 | 0.65 | 0.72 | 0.53 | 0.41 | 0.67 |
-| `Qwen2.5-1.5B-Instruct`                 | 1,5B | 1536  | 128K  | 0.56 | 0.62 | 0.55 | 0.51 | 0.41 | 0.71 |
-| `Qwen2.5-3B-Instruct`                 | 3B | 2048  | 128K  | 0.63 | 0.63 | 0.74 | 0.60 | 0.43 | 0.75 |
+| Model                            | Size  | Type        | Mean(task) | Kinopoisk <nobr>(3&nbsp;classes)</nobr> | Headliness (6 classes) | GRNTI <nobr>(28&nbsp;classes)</nobr>  | OECD <nobr>(29&nbsp;classes)</nobr>   | Inappropriateness <nobr>(3&nbsp;classes)</nobr> |
+| -------------------------------- | ----- | ----------- | ---------- | --------- | --------- | -------- | -------- | ----------------- |
+| `GeRaCl-USER2-base`              | 155 M | GeRaCl      | **0.65**   | 0.61      | 0.80      | **0.63** | **0.48** | 0.71              |
+| `USER2-base`                     | 149 M | Encoder     | 0.52       | 0.50      | 0.65      | 0.56     | 0.39     | 0.51              |
+| `USER-bge-m3`                    | 359 M | Encoder     | 0.53       | 0.60      | 0.73      | 0.43     | 0.28     | 0.62              |
+| `multilingual-e5-large-instruct` | 560 M | Encoder     | 0.63       | 0.56      | **0.83**  | 0.62     | 0.46     | 0.67              |
+| `mDeBERTa-v3-base-mnli-xnli`     | 279 M | NLI-encoder | 0.45       | 0.54      | 0.53      | 0.34     | 0.23     | 0.62              |
+| `bge-m3-zeroshot-v2.0`           | 568 M | NLI-encoder | 0.60       | **0.65**  | 0.72      | 0.53     | 0.41     | 0.67              |
+| `Qwen2.5-1.5B-Instruct`          | 1.5 B | LLM         | 0.56       | 0.62      | 0.55      | 0.51     | 0.41     | 0.71              |
+| `Qwen2.5-3B-Instruct`            | 3 B   | LLM         | 0.63       | 0.63      | 0.74      | 0.60     | 0.43     | **0.75**          |
 
-## Usage
+**How comparison was performed**
+
+1. NLI‑Encoders were used via 🤗 ```pipeline("zero-shot-classification")```
+
+Models such as mDeBERTa-v3-base-mnli-xnli and bge-m3-zeroshot-v2.0 are pre‑trained on Natural Language Inference corpora.The Hugging Face pipeline converts classification into NLI hypotheses like:
 
-### Prefixes
+Premise: text
 
-This model is based on the USER2-base sentence encoder. This model uses the "classification: " prefix to work on classification tasks.
+Hypothesis: "This text is about {label}."
 
-### Code
+The model scores each (premise, hypothesis) pair independently; the label with the highest entailment probability wins.
+
+2. LLMs prompted for classification
+
+Large‑language models such as Qwen2.5‑1.5B and Qwen2.5‑3B are queried with a simple classification prompt:
+
+```
+PROMPT = """Ниже указан текст. Ты должен присвоить ему один из перечисленных ниже классов.
+
+Текст:
+{}
+
+Классы:
+{}.
+
+Твой ответ должен состоять только из выбранного класса, ничего другого.
+"""
+```
+
+3. GeRaCl architecture. Detailed information about this architecture is located in **Training Detais** section.
+
+
+## Usage
 
 #### Single classification scenario
 
@@ -86,7 +116,7 @@ for i in range(len(labels)):
 
 ## Training details
 
-This is the base version with 155 million parameters, based on [`USER2-base`](https://huggingface.co/deepvk/USER2-base) sentence encoder. This model uses the GLiNER architecture, but it has only one vector of similarity scores instead of a full matrix of similarities.
+This is the base version with 155 million parameters, based on [`USER2-base`](https://huggingface.co/deepvk/USER2-base) sentence encoder. This model uses similar to GLiNER idea, but it has only one vector of similarity scores instead of a full matrix of similarities.
 Compared to the USER2-base model, there are two additional MLP layers. One is for the text embeddings and another is for the classes embeddings. You can see the detailed model's architecture on the picture below.
 
 <img src="assets/architecture.png" alt="GeRaCl architecture" width="600"/>
@@ -99,8 +129,7 @@ The training set is built entirely from splits of the  [`deepvk/GeRaCl_synthethi
 
 | Dataset                     | # Samples |
 |----------------------------:|:----:|
-| [GeRaCl_synthethic_dataset/synthetic_classes_train](https://huggingface.co/datasets/deepvk/GeRaCl_synthethic_dataset/viewer/synthetic_classes_train)                    | 93K |
-| [GeRaCl_synthethic_dataset/synthetic_classes](https://huggingface.co/datasets/deepvk/GeRaCl_synthethic_dataset/viewer/synthetic_classes) (val and test)                    | 6K |
+| [GeRaCl_synthethic_dataset/synthetic_classes_train](https://huggingface.co/datasets/deepvk/GeRaCl_synthethic_dataset/viewer/synthetic_classes_train)                    | 99K |
 | [GeRaCl_synthethic_dataset/ru_mteb_classes](https://huggingface.co/datasets/deepvk/GeRaCl_synthethic_dataset/viewer/ru_mteb_classes/)                    | 52K  |
 | [GeRaCl_synthethic_dataset/ru_mteb_extended_classes](https://huggingface.co/datasets/deepvk/GeRaCl_synthethic_dataset/viewer/ru_mteb_extended_classes)             | 93K  |
 | **Total**                   | 244K |