Update README.md

README.md

@@ -1,8 +1,6 @@
 # Should We Still Pretrain Encoders with Masked Language Modeling?
 
-[![arXiv](https://img.shields.io/badge/arXiv-2503.05500-b31b1b.svg?style=for-the-badge)](http://arxiv.org/abs/2507.00994)
-[![GitHub](https://img.shields.io/badge/Code_Repository-100000?style=for-the-badge&logo=github&logoColor=white)](https://github.com/Nicolas-BZRD/EuroBERT/tree/MLM_vs_CLM)
-[![Blog Post](https://img.shields.io/badge/Blog_Post-018EF5?logo=readme&logoColor=fff&style=for-the-badge)](https://huggingface.co/blog/Nicolas-BZRD/encoders-should-not-be-only-pre-trained-with-mlm)
+This page gathers all the artefacts and references related to the paper “Should We Still Pretrain Encoders with Masked Language Modeling?” (Gisserot-Boukhlef et al.)
 
 ![MLMvsCLM](https://raw.githubusercontent.com/Nicolas-BZRD/EuroBERT/refs/heads/MLM_vs_CLM/docs/images/hf_card_without.png)
 
@@ -10,23 +8,32 @@
 
 Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional encoders on text representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM objective or arise from confounding factors such as model and data scale. In this paper, we address this question through a series of large-scale, carefully controlled pretraining ablations, training a total of 30 models ranging from 210 million to 1 billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trained models are more data-efficient and demonstrate improved fine-tuning stability. Building on these findings, we experimentally show that a biphasic training strategy that sequentially applies CLM and then MLM, achieves optimal performance under a fixed computational training budget. Moreover, we demonstrate that this strategy becomes more appealing  when initializing from readily available pretrained CLM models (from the existing LLM ecosystem), reducing the computational burden needed to train best-in-class encoder models.
 
-## Ressources
+## Resources
 
-- **[Hugging Face Project Page](https://huggingface.co/MLMvsCLM)**: The Hugging Face page that centralizes everything!
-- **[Preprint](https://arxiv.org/abs/2507.00994)**: The paper with all the details!
-- **[Blog](https://huggingface.co/blog/Nicolas-BZRD/encoders-should-not-be-only-pre-trained-with-mlm)**: A blog post summarizing the paper in a 5-minute read.  
-- **[GitHub Codebase](https://github.com/Nicolas-BZRD/EuroBERT/tree/MLM_vs_CLM)**: *Optimus Training Library* — a scalable distributed training framework for training encoders at scale.  
-- **[Model: EuroBERT](https://huggingface.co/EuroBERT)**: The model architecture used in the experiments.
+- **[Preprint](https://arxiv.org/abs/2507.00994)**: For the full details of our work
+- **[Blog post](https://huggingface.co/blog/Nicolas-BZRD/encoders-should-not-be-only-pre-trained-with-mlm)**: A quick overview if you only have 5 minutes
+- **[EuroBERT](https://huggingface.co/EuroBERT)**: The encoder model architecture used in our experiments
+- **[Training codebase](https://github.com/Nicolas-BZRD/EuroBERT/tree/MLM_vs_CLM)**: *Optimus*, our distributed framework for training encoders at scale
+- **[Evaluation codebase](https://github.com/hgissbkh/EncodEval/tree/MLM_vs_CLM)**: *EncodEval*, our framework for evaluating encoder models across a wide range of representation tasks
 
+## Models
 
-## Contact of the first-authors
+We release all the models trained and evaluated in the paper.
+
+* Model names follow the format `[model size]-[objective]-[number of steps]`: e.g., `610m-clm-42k` refers to a 610M-parameter model trained with CLM for 42k steps.
+* For models trained in two stages, names follow the extended format `[model size]-[objective #1]-[number of steps #1]-[objective #2]-[number of steps #2]`, `[number of steps #2]` indicates the total number of training steps: e.g., `610m-clm-10k-mlm40-42k` is a 610M model first trained with CLM for 10k steps, then further trained with MLM (using a 40% masking ratio) for an additional 32k steps, totaling 42k.
+* Models that were continued from a decayed checkpoint use the "dec" prefix for the first step count: e.g., `610m-clm-dec42k-mlm40-64k` represents a 610M model first trained and decayed with CLM for 42k steps, then continued with MLM (40% masking ratio) for 22k more steps, totaling 64k.
+* By default, model names refer to the final checkpoint. Intermediate checkpoints are indicated by appending the step number at the end: e.g., `610m-mlm40-42k-1000` corresponds to checkpoint 1,000 of a 610M model trained with MLM (40% masking) for 42k steps.
+
+
+## First authors' contact information
 
 - Hippolyte Gisserot-Boukhlef : [email protected]
 - Nicolas Boizard : [email protected]
 
 ## Citation
 
-If you use this project in your research, please cite the original paper as follows:
+If you found our work useful, please consider citing our paper:
 
 ```bibtex
 @misc{gisserotboukhlef2025pretrainencodersmaskedlanguage,