crumb commited on
Commit
eb8430c
·
1 Parent(s): 061e839

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -34,7 +34,7 @@ The B, C, and D classes are derived from the tokens per model ratio from LLaMA,
34
  | GerbilLab/GerbilBlender-A-15m | 15m | A-Class | 20 | 280M | 131k | coming soon |
35
  | GerbilLab/GerbilBlender-A-32m | 32m | A-Class | 20 | 640M | 262K | coming soon |
36
 
37
- Nearly every base model that isn't finetuned for a specific task was trained on the deduplicated Pile dataset. "Blender" models, inspired by UL2 pretraining, are trained equally in fill-in-the-middle, causal modelling, and masked language modelling tasks. Special tokens for these models include:
38
 
39
  ```
40
  '<fitm_start>', '<multiple_tok_mask>', '<fitm_result>', '<causal>', '<mlm_start>', '<single_tok_mask>', '<mlm_end>'
 
34
  | GerbilLab/GerbilBlender-A-15m | 15m | A-Class | 20 | 280M | 131k | coming soon |
35
  | GerbilLab/GerbilBlender-A-32m | 32m | A-Class | 20 | 640M | 262K | coming soon |
36
 
37
+ Nearly every base model that isn't finetuned for a specific task was trained on the deduplicated Pile dataset, and is a Decoder-only model. "Blender" models, inspired by UL2 pretraining, are trained equally in fill-in-the-middle, causal modelling, and masked language modelling tasks. Special tokens for these models include:
38
 
39
  ```
40
  '<fitm_start>', '<multiple_tok_mask>', '<fitm_result>', '<causal>', '<mlm_start>', '<single_tok_mask>', '<mlm_end>'