Upload folder using huggingface_hub

.gitattributes

@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 figures/performance.png filter=lfs diff=lfs merge=lfs -text
+figures/new_logo.png filter=lfs diff=lfs merge=lfs -text
+figures/wechat.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,44 +1,67 @@
 ---
 license: mit
-license_link: https://huggingface.co/rednote-hilab/dots.llm1.inst/blob/main/LICENSE
+license_link: https://huggingface.co/rednote-hilab/dots.llm1.base/blob/main/LICENSE
 pipeline_tag: text-generation
 base_model: rednote-hilab/dots.llm1.base
 tags:
 - chat
 library_name: transformers
-language:
-- en
-- zh
 ---
 
 # dots1
 
+<p align="center">
+    <img src="figures/new_logo.png" width="200"/>
+<p>
+
+<p align="center">
+    &nbsp&nbsp🤗 <a href="https://huggingface.co/rednote-hilab">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp 📑 <a href="https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf">Paper</a> &nbsp&nbsp 
+<br>
+🖥️ <a href="https://huggingface.co/spaces/rednote-hilab/dots-demo">Demo</a>&nbsp&nbsp | &nbsp&nbsp💬 <a href="figures/wechat.png">WeChat (微信)</a>&nbsp&nbsp | &nbsp&nbsp📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>&nbsp&nbsp
+</p>
+
+
+
+
+Visit our Hugging Face (click links above), search checkpoints with names starting with `dots.llm1` or visit the [dots1 collection](https://huggingface.co/collections/rednote-hilab/dotsllm1-68246aaaaba3363374a8aa7c), and you will find all you need! Enjoy!
+
+
+## News
+
+- 2025.06.06: We released the `dots.llm1` series. Check our [report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf) for more details!
+
 
 ## 1. Introduction
 
 
-`dots.llm1` is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. 
-Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B when trained on 11.2T high-quality tokens without synthetic data. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.
+The `dots.llm1` model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models. 
+Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B after pretrained on 11.2T high-quality tokens without synthetic data. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.
 
 
 <p align="center">
   <img width="90%" src="./figures/performance.png">
 </p>
 
-
 ## 2. Model Summary
 
 **This repo contains the base and instruction-tuned `dots.llm1` model**. which has the following features:
 
-- Type: A 14B/142B MoE model trained on 11.2T tokens.
-- Training Stage: Pretraining & Post-training
-- Architecture: Multi-head Attention with QK-Norm in Attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.
+- Type: A MoE model with 14B activated and 142B total parameters trained on 11.2T tokens.
+- Training Stages: Pretraining and SFT.
+- Architecture: Multi-head Attention with QK-Norm in attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.
 - Number of Layers: 62
 - Number of Attention Heads: 32
+- Supported Languages: English, Chinese
 - Context Length: 32,768 tokens
 - License: MIT
 
-For more details, please refer to our [report](dots1_tech_report.pdf).
+The highlights from `dots.llm1` include:
+
+- **Enhanced Data Processing**: We propose a scalable and fine-grained *three-stage* data processing framework designed to generate large-scale, high-quality and diverse data for pretraining.
+- **No Synthetic Data during Pretraining**: *11.2 trillion* high-quality non-synthetic tokens was used in base model pretraining.
+- **Performance and Cost Efficiency**: `dots.llm1` is an open-source model that activates only *14B* parameters at inference, delivering both comprehensive capabilities and high computational efficiency.
+- **Infrastructure**: We introduce an innovative MoE all-to-all communication and computation overlapping recipe based on interleaved 1F1B pipeline scheduling and an efficient grouped GEMM implementation to boost computational efficiency.
+- **Open Accessibility to Model Dynamics**: Intermediate model checkpoints for *every 1T tokens* trained are released, facilitating future research into the learning dynamics of large language models.
 
 ## 3. Example Usage
 
@@ -53,6 +76,42 @@ For more details, please refer to our [report](dots1_tech_report.pdf).
 
 </div>
 
+### Docker (recommended)
+
+
+The docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/rednotehilab/dots1/tags), based on the official images.
+
+You can start a server via vllm.
+
+```shell
+docker run --gpus all \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    -p 8000:8000 \
+    --ipc=host \
+    rednotehilab/dots1:vllm-openai-v0.9.0.1 \
+    --model rednote-hilab/dots.llm1.inst \
+    --tensor-parallel-size 8 \
+    --trust-remote-code \
+    --served-model-name dots1
+```
+
+Then you can verify whether the model is running successfully in the following way.
+
+```shell
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "dots1",
+        "messages": [
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": "Who won the world series in 2020?"}
+        ],
+        "max_tokens": 32,
+        "temperature": 0
+    }'
+```
+
+
 ### Inference with huggingface
 
 #### Text Completion
@@ -106,8 +165,7 @@ python -m sglang.launch_server --model-path dots.llm1.inst --tp 8 --host 0.0.0.0
 An OpenAI-compatible API will be available at `http://localhost:8000/v1`.
 
 ### Inference with vllm
-[vLLM](https://github.com/vllm-project/vllm) is a high-throughput and memory-efficient inference and serving engine for LLMs.
-`vllm>=***` is recommended.
+[vLLM](https://github.com/vllm-project/vllm) is a high-throughput and memory-efficient inference and serving engine for LLMs. `vllm>=***` is recommended.
 
 ```shell
 vllm serve dots.llm1.inst --port 8000 --tensor-parallel-size 8
@@ -116,7 +174,7 @@ An OpenAI-compatible API will be available at `http://localhost:8000/v1`.
 
 ## 4. Evaluation Results
 
-Detailed evaluation results are reported in this [📑 report](dots1_tech_report.pdf).
+Detailed evaluation results are reported in this [📑 report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf).
 
 ## Citation
 

config.json

@@ -28,7 +28,6 @@
   "rope_theta": 10000000,
   "routed_scaling_factor": 2.5,
   "sliding_window": null,
-  "scoring_func": "noaux_tc",
   "tie_word_embeddings": false,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.46.3",