Update README.md

README.md

@@ -1,10 +1,41 @@
----
-title: README
-emoji: 👀
-colorFrom: blue
-colorTo: purple
-sdk: static
-pinned: false
----
-
-Edit this `README.md` markdown file to author your organization card.
+# Should We Still Pretrain Encoders with Masked Language Modeling?
+
+[![arXiv](https://img.shields.io/badge/arXiv-2503.05500-b31b1b.svg?style=for-the-badge)](http://arxiv.org/abs/2507.00994)
+[![GitHub](https://img.shields.io/badge/Code_Repository-100000?style=for-the-badge&logo=github&logoColor=white)](https://github.com/Nicolas-BZRD/EuroBERT/tree/MLM_vs_CLM)
+[![Blog Post](https://img.shields.io/badge/Blog_Post-018EF5?logo=readme&logoColor=fff&style=for-the-badge)](https://huggingface.co/blog/Nicolas-BZRD/encoders-should-not-be-only-pre-trained-with-mlm)
+
+![MLMvsCLM](https://raw.githubusercontent.com/Nicolas-BZRD/EuroBERT/refs/heads/MLM_vs_CLM/docs/images/hf_card_without.png)
+
+## Abstract
+
+Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional encoders on text representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM objective or arise from confounding factors such as model and data scale. In this paper, we address this question through a series of large-scale, carefully controlled pretraining ablations, training a total of 30 models ranging from 210 million to 1 billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trained models are more data-efficient and demonstrate improved fine-tuning stability. Building on these findings, we experimentally show that a biphasic training strategy that sequentially applies CLM and then MLM, achieves optimal performance under a fixed computational training budget. Moreover, we demonstrate that this strategy becomes more appealing  when initializing from readily available pretrained CLM models (from the existing LLM ecosystem), reducing the computational burden needed to train best-in-class encoder models.
+
+## Ressources
+
+- **[Hugging Face Project Page](https://huggingface.co/MLMvsCLM)**: The Hugging Face page that centralizes everything!
+- **[Preprint](https://arxiv.org/abs/2507.00994)**: The paper with all the details!
+- **[Blog](https://huggingface.co/blog/Nicolas-BZRD/encoders-should-not-be-only-pre-trained-with-mlm)**: A blog post summarizing the paper in a 5-minute read.  
+- **[GitHub Codebase](https://github.com/Nicolas-BZRD/EuroBERT/tree/MLM_vs_CLM)**: *Optimus Training Library* — a scalable distributed training framework for training encoders at scale.  
+- **[Model: EuroBERT](https://huggingface.co/EuroBERT)**: The model architecture used in the experiments.
+
+
+## Contact of the first-authors
+
+- Hippolyte Gisserot-Boukhlef : [email protected]
+- Nicolas Boizard : [email protected]
+
+## Citation
+
+If you use this project in your research, please cite the original paper as follows:
+
+```bibtex
+@misc{gisserotboukhlef2025pretrainencodersmaskedlanguage,
+      title={Should We Still Pretrain Encoders with Masked Language Modeling?}, 
+      author={Hippolyte Gisserot-Boukhlef and Nicolas Boizard and Manuel Faysse and Duarte M. Alves and Emmanuel Malherbe and André F. T. Martins and Céline Hudelot and Pierre Colombo},
+      year={2025},
+      eprint={2507.00994},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2507.00994}, 
+}
+```