manu
/

tok_custom_scratch

Model card Files Files and versions

manu commited on Oct 13, 2023

Commit

b11f4ae

·

1 Parent(s): 5d7f6da

Update README.md

Files changed (1) hide show

README.md +2 -1

README.md CHANGED Viewed

@@ -8,4 +8,5 @@ language:
 BPE Tokenizer fitted on a custom corpus, with digit separation, byte fallback and other features from LlamaTokenizer.
-Only fitted on 100,000 samples (7.5M words).

 BPE Tokenizer fitted on a custom corpus, with digit separation, byte fallback and other features from LlamaTokenizer.
+Only fitted on 100,000 samples (7.5M words).
+# Warning - Dataset was not shuffled so fitted on code only, not usable as is !