Spaces:

Dovakiins
/

qwerrwe

Build error

Nanobit commited on Aug 28, 2023

Commit

4c37bd0

unverified ·

1 Parent(s): f144e98

Fix(tokenizer): Make sure to add pad for CodeLlamaTokenizer (#489)

Files changed (1) hide show

src/axolotl/utils/models.py CHANGED Viewed

@@ -59,6 +59,7 @@ def load_tokenizer(cfg):
         in [
             "LlamaTokenizer",
             "LlamaTokenizerFast",
         ]
         and hasattr(tokenizer, "pad_token")
         and not tokenizer.pad_token

         in [
             "LlamaTokenizer",
             "LlamaTokenizerFast",
+            "CodeLlamaTokenizer",
         ]
         and hasattr(tokenizer, "pad_token")
         and not tokenizer.pad_token