Update README.md
Browse files
README.md
CHANGED
@@ -34,7 +34,7 @@ The B, C, and D classes are derived from the tokens per model ratio from LLaMA,
|
|
34 |
| GerbilLab/GerbilBlender-A-15m | 15m | A-Class | 20 | 280M | 131k | coming soon |
|
35 |
| GerbilLab/GerbilBlender-A-32m | 32m | A-Class | 20 | 640M | 262K | coming soon |
|
36 |
|
37 |
-
Nearly every base model that isn't finetuned for a specific task was trained on the deduplicated Pile dataset. "Blender" models, inspired by UL2 pretraining, are trained equally in fill-in-the-middle, causal modelling, and masked language modelling tasks. Special tokens for these models include:
|
38 |
|
39 |
```
|
40 |
'<fitm_start>', '<multiple_tok_mask>', '<fitm_result>', '<causal>', '<mlm_start>', '<single_tok_mask>', '<mlm_end>'
|
|
|
34 |
| GerbilLab/GerbilBlender-A-15m | 15m | A-Class | 20 | 280M | 131k | coming soon |
|
35 |
| GerbilLab/GerbilBlender-A-32m | 32m | A-Class | 20 | 640M | 262K | coming soon |
|
36 |
|
37 |
+
Nearly every base model that isn't finetuned for a specific task was trained on the deduplicated Pile dataset, and is a Decoder-only model. "Blender" models, inspired by UL2 pretraining, are trained equally in fill-in-the-middle, causal modelling, and masked language modelling tasks. Special tokens for these models include:
|
38 |
|
39 |
```
|
40 |
'<fitm_start>', '<multiple_tok_mask>', '<fitm_result>', '<causal>', '<mlm_start>', '<single_tok_mask>', '<mlm_end>'
|