| --- |
| model_name: transformer_multi_head_robbertv2_lora |
| base_model: pdelobelle/robbert-v2-dutch-base |
| language: nl |
| library_name: transformers |
| tags: |
| - dutch |
| - regression |
| - multi-head |
| - robbert-v2 |
| - lora |
| - text-quality |
| license: mit |
| datasets: |
| - proprietary |
| metrics: |
| - rmse |
| - r2 |
| pipeline_tag: text-classification |
| --- |
| |
| # transformer_multi_head_robbertv2_lora |
|
|
| This is a **multi-head transformer regression model** using **RobBERT-v2** with **LoRA parameter-efficient fine-tuning**, designed to predict **four separate text quality scores** for Dutch texts. |
|
|
| The final **aggregate metric** recomputes a combined score from the four heads and compares it to the actual aggregate, providing robust quality tracking. |
|
|
| --- |
|
|
| ## 📈 Training & Evaluation |
|
|
| | Epoch | Train Loss | Val Loss | RMSE (delta_cola_to_final) | R² (delta_cola_to_final) | RMSE (delta_perplexity_to_final_large) | R² (delta_perplexity_to_final_large) | RMSE (iter_to_final_simplified) | R² (iter_to_final_simplified) | RMSE (robbert_delta_blurb_to_final) | R² (robbert_delta_blurb_to_final) | Mean RMSE | |
| |-------|-------------|-----------|----------------------------|--------------------------|----------------------------------------|--------------------------------------|---------------------------------|---------------------------------|-------------------------------------|-----------------------------------|-----------| |
| | 1 | 0.0363 | 0.0221 | 0.1543 | 0.3456 | 0.1210 | 0.4855 | 0.1765 | 0.7058 | 0.1377 | 0.6308 | 0.1474 | |
| | 2 | 0.0237 | 0.0199 | 0.1549 | 0.3401 | 0.1157 | 0.5297 | 0.1621 | 0.7517 | 0.1257 | 0.6922 | 0.1396 | |
| | 3 | 0.0212 | 0.0187 | 0.1543 | 0.3457 | 0.1074 | 0.5947 | 0.1547 | 0.7739 | 0.1243 | 0.6991 | 0.1352 | |
| | 4 | 0.0201 | 0.0185 | 0.1533 | 0.3544 | 0.1091 | 0.5818 | 0.1531 | 0.7784 | 0.1234 | 0.7032 | 0.1347 | |
| | 5 | 0.0196 | 0.0182 | 0.1508 | 0.3752 | 0.1081 | 0.5896 | 0.1528 | 0.7794 | 0.1233 | 0.7041 | 0.1337 | |
|
|
| **Final aggregate performance** |
| ✅ **Aggregate RMSE:** `0.0872` |
| ✅ **Aggregate R²:** `0.7970` |
|
|
| --- |
|
|
| ## 🧾 Notes |
|
|
| - This model uses **LoRA fine-tuning** to train only ~0.75% of RobBERT-v2’s parameters. |
| - It has **four parallel regression heads** for: |
| - `delta_cola_to_final` |
| - `delta_perplexity_to_final_large` |
| - `iter_to_final_simplified` |
| - `robbert_delta_blurb_to_final` |
| - The final test set results confirm robust performance with individual and aggregate metrics. |
| - Fine-tuned on a proprietary dataset of Dutch text variations. |
| - **Base:** RobBERT-v2 Dutch Base (`pdelobel |
|
|
