Update README.md
Browse files
README.md
CHANGED
|
@@ -1,21 +1,22 @@
|
|
| 1 |
---
|
| 2 |
-
license: mit
|
| 3 |
library_name: transformers
|
|
|
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
-
|
| 6 |
-
|
|
|
|
| 7 |
---
|
| 8 |
|
| 9 |
**Repository for:**
|
| 10 |
|
| 11 |
**ThinkEdit-deepseek-llama3-8b**
|
| 12 |
|
| 13 |
-
(We also release ThinkEdit versions for ThinkEdit-deepseek-qwen-1.5b and ThinkEdit-deepseek-qwen-
|
| 14 |
|
| 15 |
-
**Authors**: Chung-En Sun, Ge Yan, Tsui-Wei Weng
|
| 16 |
**Paper**: [ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models](https://arxiv.org/abs/2503.22048)
|
| 17 |
|
| 18 |
-
|
| 19 |
|
| 20 |
---
|
| 21 |
|
|
@@ -25,10 +26,10 @@ Reasoning-augmented models sometimes fail by generating **overly short**, abstra
|
|
| 25 |
|
| 26 |
**ThinkEdit** is a lightweight weight-editing method that:
|
| 27 |
|
| 28 |
-
- Identifies ~
|
| 29 |
-
- Edits only ~0.
|
| 30 |
-
- Removes the "short reasoning" direction from their output
|
| 31 |
-
- Boosts performance, especially on cases with short reasoning traces
|
| 32 |
|
| 33 |
---
|
| 34 |
|
|
@@ -36,40 +37,31 @@ Reasoning-augmented models sometimes fail by generating **overly short**, abstra
|
|
| 36 |
|
| 37 |
### 1. Overall Accuracy
|
| 38 |
|
| 39 |
-
| Model | GSM8K
|
| 40 |
-
|
| 41 |
-
| deepseek-qwen-
|
| 42 |
-
| **ThinkEdit-deepseek-qwen-
|
| 43 |
-
| deepseek-
|
| 44 |
-
| **ThinkEdit-deepseek-
|
| 45 |
-
| deepseek-
|
| 46 |
-
| **ThinkEdit-deepseek-
|
|
|
|
|
|
|
| 47 |
|
| 48 |
---
|
| 49 |
|
| 50 |
### 2. Accuracy on Short Reasoning Cases (Top 5% / 10% / 20%)
|
| 51 |
|
| 52 |
-
| Model | GSM8K
|
| 53 |
-
|
| 54 |
-
| deepseek-qwen-
|
| 55 |
-
| **ThinkEdit-deepseek-qwen-
|
| 56 |
-
| deepseek-
|
| 57 |
-
| **ThinkEdit-deepseek-
|
| 58 |
-
| deepseek-
|
| 59 |
-
| **ThinkEdit-deepseek-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
### 3. Reasoning Lengths (Top 5% / 10% / 20% Shortest Responses)
|
| 64 |
-
|
| 65 |
-
| Model | GSM8K | MMLU Elementary Math | MATH-Level1 | MATH-Level5 | MATH-500 |
|
| 66 |
-
| -------------------------------- | -------------------------------- | --------------------------------- | --------------------------------- | ---------------------------------- | --------------------------------- |
|
| 67 |
-
| deepseek-qwen-14b | 76.6 / 86.5 / 99.1 | 65.8 / 72.2 / 80.6 | 93.7 / 114.3 / 188.6 | 628.8 / 858.4 / 1125.9 | 198.7 / 434.3 / 697.0 |
|
| 68 |
-
| **ThinkEdit-deepseek-qwen-14b** | **95.4** / **106.3** / **120.2** | **79.1** / **87.1** / **98.7** | **125.1** / **150.2** / **243.4** | **698.5** / **906.6** / **1157.2** | **270.2** / **492.6** / **733.3** |
|
| 69 |
-
| deepseek-llama3-8b | 73.0 / 83.1 / 96.6 | 371.0 / 438.1 / 518.2 | 80.3 / 97.2 / 130.3 | 617.9 / 854.9 / 1126.5 | 159.5 / 357.5 / 644.5 |
|
| 70 |
-
| **ThinkEdit-deepseek-llama3-8b** | **93.2** / **106.9** / **127.4** | **396.5** / **464.2** / **543.2** | **137.4** / **173.3** / **277.1** | **791.2** / **954.8** / **1185.1** | **305.2** / **506.3** / **737.6** |
|
| 71 |
-
| deepseek-qwen-1.5b | 78.8 / 89.4 / 103.0 | 61.6 / 68.5 / 77.6 | 88.8 / 110.3 / 219.7 | 804.6 / **1017.9** / **1314.0** | 249.7 / 506.5 / 760.7 |
|
| 72 |
-
| **ThinkEdit-deepseek-qwen-1.5b** | **97.2** / **109.4** / **126.3** | **75.9** / **85.0** / **99.5** | **127.9** / **174.1** / **416.4** | **818.0** / 984.5 / 1214.3 | **435.0** / **612.9** / **800.6** |
|
| 73 |
|
| 74 |
---
|
| 75 |
|
|
@@ -81,12 +73,11 @@ The usage of ThinkEdit models is exactly the same as the original deepseek-disti
|
|
| 81 |
|
| 82 |
```bibtex
|
| 83 |
@misc{sun2025thinkedit,
|
| 84 |
-
title={ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models},
|
| 85 |
author={Chung-En Sun and Ge Yan and Tsui-Wei Weng},
|
| 86 |
year={2025},
|
| 87 |
eprint={2503.22048},
|
| 88 |
archivePrefix={arXiv},
|
| 89 |
primaryClass={cs.CL},
|
| 90 |
-
url={https://arxiv.org/abs/2503.22048},
|
| 91 |
}
|
| 92 |
-
```
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
library_name: transformers
|
| 3 |
+
tags: []
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
+
license: mit
|
| 6 |
+
base_model:
|
| 7 |
+
- deepseek-ai/DeepSeek-R1-Distill-Llama-8B
|
| 8 |
---
|
| 9 |
|
| 10 |
**Repository for:**
|
| 11 |
|
| 12 |
**ThinkEdit-deepseek-llama3-8b**
|
| 13 |
|
| 14 |
+
(We also release ThinkEdit versions for ThinkEdit-deepseek-qwen-1.5b, ThinkEdit-deepseek-qwen-14b, and ThinkEdit-deepseek-qwen-32b.)
|
| 15 |
|
| 16 |
+
**Authors**: Chung-En Sun, Ge Yan, Tsui-Wei Weng
|
| 17 |
**Paper**: [ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models](https://arxiv.org/abs/2503.22048)
|
| 18 |
|
| 19 |
+
Github: https://github.com/Trustworthy-ML-Lab/ThinkEdit
|
| 20 |
|
| 21 |
---
|
| 22 |
|
|
|
|
| 26 |
|
| 27 |
**ThinkEdit** is a lightweight weight-editing method that:
|
| 28 |
|
| 29 |
+
- Identifies ~4% of "short reasoning" attention heads
|
| 30 |
+
- Edits only ~0.2% of total parameters
|
| 31 |
+
- Removes the "short reasoning" direction from their output
|
| 32 |
+
- Boosts performance, especially on cases with short reasoning traces
|
| 33 |
|
| 34 |
---
|
| 35 |
|
|
|
|
| 37 |
|
| 38 |
### 1. Overall Accuracy
|
| 39 |
|
| 40 |
+
| Model | GSM8K | MMLU Elementary Math | MATH-Level1 | MATH-Level5 | MATH-500 |
|
| 41 |
+
|---------------------------------|---------------------|----------------------|---------------------|---------------------|---------------------|
|
| 42 |
+
| deepseek-qwen-32b | 92.97 ± 0.39 | 95.93 ± 0.83 | **96.41 ± 0.45** | 91.27 ± 0.53 | **91.62 ± 0.58** |
|
| 43 |
+
| **ThinkEdit-deepseek-qwen-32b** | **95.25 ± 0.25** | **98.02 ± 0.31** | 96.02 ± 0.42 | **91.31 ± 0.50** | 91.60 ± 0.65 |
|
| 44 |
+
| deepseek-qwen-14b | 90.80 ± 0.36 | 95.08 ± 0.65 | 96.32 ± 0.35 | 90.25 ± 0.72 | 91.48 ± 0.55 |
|
| 45 |
+
| **ThinkEdit-deepseek-qwen-14b** | **93.78 ± 0.50** | **96.56 ± 0.84** | **96.38 ± 0.52** | **91.03 ± 0.44** | **91.92 ± 0.63** |
|
| 46 |
+
| deepseek-llama3-8b | 82.26 ± 0.91 | 96.01 ± 0.62 | 93.46 ± 0.84 | 85.49 ± 0.83 | 87.26 ± 1.16 |
|
| 47 |
+
| **ThinkEdit-deepseek-llama3-8b**| **89.44 ± 0.55** | **96.19 ± 0.73** | **94.44 ± 0.31** | **86.49 ± 0.54** | **88.06 ± 1.09** |
|
| 48 |
+
| deepseek-qwen-1.5b | 79.15 ± 1.08 | 68.52 ± 1.56 | 93.00 ± 0.33 | **75.48 ± 0.90** | 82.22 ± 1.29 |
|
| 49 |
+
| **ThinkEdit-deepseek-qwen-1.5b**| **84.56 ± 0.79** | **90.66 ± 0.97** | **93.66 ± 0.62** | 75.05 ± 0.82 | **82.24 ± 0.89** |
|
| 50 |
|
| 51 |
---
|
| 52 |
|
| 53 |
### 2. Accuracy on Short Reasoning Cases (Top 5% / 10% / 20%)
|
| 54 |
|
| 55 |
+
| Model | GSM8K | MMLU Elementary Math | MATH-Level1 | MATH-Level5 | MATH-500 |
|
| 56 |
+
|---------------------------------|---------------------------------|----------------------------------|----------------------------------|----------------------------------|----------------------------------|
|
| 57 |
+
| deepseek-qwen-32b | 98.31 / 97.18 / 96.20 | 97.78 / 97.03 / 95.87 | 100.00 / 100.00 / **98.97** | 93.03 / 96.36 / 97.35 | 86.40 / 92.00 / 94.00 |
|
| 58 |
+
| **ThinkEdit-deepseek-qwen-32b** | **98.92** / **97.71** / **97.83** | **97.78** / **97.57** / **97.20** | **100.00** / **100.00** / 98.74 | **98.03** / **98.64** / **97.99** | **92.00** / **94.40** / **95.80** |
|
| 59 |
+
| deepseek-qwen-14b | **96.31** / 95.65 / 92.93 | 93.89 / **96.22** / 95.60 | 99.52 / **99.30** / 97.70 | 89.39 / 94.32 / 96.25 | 86.40 / 91.40 / 93.50 |
|
| 60 |
+
| **ThinkEdit-deepseek-qwen-14b** | **96.31** / **96.18** / **96.77** | **97.78** / 95.14 / **96.53** | **99.53** / 98.62 / **98.67** | **96.67** / **97.88** / **98.11** | **91.20** / **93.20** / **95.00** |
|
| 61 |
+
| deepseek-llama3-8b | 88.92 / 87.18 / 85.82 | 97.22 / 96.49 / 96.80 | 97.14 / 94.88 / 94.83 | 78.64 / 88.79 / 93.41 | 82.00 / 81.40 / 88.30 |
|
| 62 |
+
| **ThinkEdit-deepseek-llama3-8b**| **97.08** / **95.27** / **93.95** | **97.78** / **98.65** / **97.87** | **100.00** / **99.30** / **98.62** | **95.61** / **96.89** / **97.12** | **92.80** / **93.60** / **94.40** |
|
| 63 |
+
| deepseek-qwen-1.5b | 88.46 / 87.48 / 85.02 | 62.78 / 62.16 / 60.53 | **97.62** / 95.12 / 93.91 | 91.52 / 95.00 / 95.72 | 82.40 / 89.80 / 93.40 |
|
| 64 |
+
| **ThinkEdit-deepseek-qwen-1.5b**| **92.62** / **92.90** / **92.32** | **87.78** / **88.11** / **88.67** | 95.71 / **95.58** / **96.44** | **95.15** / **96.59** / **97.27** | **90.80** / **92.00** / **94.20** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
---
|
| 67 |
|
|
|
|
| 73 |
|
| 74 |
```bibtex
|
| 75 |
@misc{sun2025thinkedit,
|
| 76 |
+
title={ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models},
|
| 77 |
author={Chung-En Sun and Ge Yan and Tsui-Wei Weng},
|
| 78 |
year={2025},
|
| 79 |
eprint={2503.22048},
|
| 80 |
archivePrefix={arXiv},
|
| 81 |
primaryClass={cs.CL},
|
| 82 |
+
url={https://arxiv.org/abs/2503.22048},
|
| 83 |
}
|
|
|