Loubna Ben Allal

1- we used the Olmo2 pes2o dataset in allenai/olmo-mix-1124
2- GitHub issues are from HuggingFaceTB/issues-kaggle-notebooks. Pull Requests and Jupyter Notebooks are part of The Stack v2 for which you will need the Software Heritage agreement. Otherwise, for notebooks you could use The Stack v1 jupyter scripts which are a bit similar, except that we remove any special tokens and clean up some Juytext leftovers (like the metadata or # - and # + lines)
3 - Yes Kaggle notebooks are from there

You can find the latest pretraining configs in this folder: https://github.com/huggingface/smollm/blob/main/text/pretraining/smollm3/

We didn't do further processing of the datasets (except for filtering on number of tokens for the context extension phase and formatting Q&A and instruct data for the decay, as explained here). You can find the datasets in this collection
Some notes (highlighted in the configs on the GitHub repo) :

• we use FineWeb2-HQ for all the languages below except Hindi, Thai, Korean for which we use FineWeb2
• we don't use Stack-Edu in phase 1 for code (despite the s3 bucket name), that's StarCoder2Data (stack v2)

Not sure we can share the s3 buckets but all the s3 datasets are already tokenized anyway.
But let us know if you have specific questions about some datasets, we're happy to elaborate.

we used LLama 3.2 tokenizer as is (except for removing bos_token), here are some details about how Meta built the tokenizer (from llama3 paper)

We use a vocabulary with 128K tokens. Our token vocabulary combines 100K tokens from the tiktoken3tokenizer with 28K additional tokens to better support non-English languages. Compared to the Llama
2 tokenizer, our new tokenizer improves compression rates on a sample of English data from 3.17 to
3.94 characters per token. This enables the model to “read” more text for the same amount of training
compute. We also found that adding 28K tokens from select non-English languages improved both
compression ratios and downstream performance, with no impact on English tokenization.

Yes the wandb logs are here: https://wandb.ai/huggingface/SmolLM3-training-logs . They were released along with the intermediate checkpoints https://x.com/eliebakouch/status/1947314495103160458