README.md

metadata

model_name: transformer_multi_head_robbertv2_lora
base_model: pdelobelle/robbert-v2-dutch-base
language: nl
library_name: transformers
tags:
  - dutch
  - regression
  - multi-head
  - robbert-v2
  - lora
  - text-quality
license: mit
datasets:
  - proprietary
metrics:
  - rmse
  - r2
pipeline_tag: text-classification

transformer_multi_head_robbertv2_lora

This is a multi-head transformer regression model using RobBERT-v2 with LoRA parameter-efficient fine-tuning, designed to predict four separate text quality scores for Dutch texts.

The final aggregate metric recomputes a combined score from the four heads and compares it to the actual aggregate, providing robust quality tracking.

---

📈 Training & Evaluation

Epoch	Train Loss	Val Loss	RMSE (delta_cola_to_final)	R² (delta_cola_to_final)	RMSE (delta_perplexity_to_final_large)	R² (delta_perplexity_to_final_large)	RMSE (iter_to_final_simplified)	R² (iter_to_final_simplified)	RMSE (robbert_delta_blurb_to_final)	R² (robbert_delta_blurb_to_final)	Mean RMSE
1	0.0363	0.0221	0.1543	0.3456	0.1210	0.4855	0.1765	0.7058	0.1377	0.6308	0.1474
2	0.0237	0.0199	0.1549	0.3401	0.1157	0.5297	0.1621	0.7517	0.1257	0.6922	0.1396
3	0.0212	0.0187	0.1543	0.3457	0.1074	0.5947	0.1547	0.7739	0.1243	0.6991	0.1352
4	0.0201	0.0185	0.1533	0.3544	0.1091	0.5818	0.1531	0.7784	0.1234	0.7032	0.1347
5	0.0196	0.0182	0.1508	0.3752	0.1081	0.5896	0.1528	0.7794	0.1233	0.7041	0.1337

Final aggregate performance
✅ Aggregate RMSE: 0.0872
✅ Aggregate R²: 0.7970

---

🧾 Notes

• This model uses LoRA fine-tuning to train only ~0.75% of RobBERT-v2’s parameters.
• It has four parallel regression heads for:
  • delta_cola_to_final
  • delta_perplexity_to_final_large
  • iter_to_final_simplified
  • robbert_delta_blurb_to_final
• The final test set results confirm robust performance with individual and aggregate metrics.
• Fine-tuned on a proprietary dataset of Dutch text variations.
• Base: RobBERT-v2 Dutch Base (`pdelobel