| # LiquidAI/LFM2-Tokenizer | |
| ## Formatted text | |
| ``` | |
| <|startoftext|><|im_start|>system | |
| You are a helpful assistant.<|im_end|> | |
| <|im_start|>user | |
| Hello! How are you?<|im_end|> | |
| <|im_start|>assistant | |
| I'm doing well, thank you!<|im_end|> | |
| <|im_start|>user | |
| What's the weather like?<|im_end|> | |
| <|im_start|>assistant | |
| ``` | |
| ## Special tokens | |
| - bos_token: <|startoftext|> | |
| - eos_token: <|im_end|> | |
| - pad_token: <|pad|> | |
| - sep_token: None | |
| - cls_token: None | |
| - mask_token: None | |
| ## Added special tokens | |
| - "<|pad|>": 0, | |
| - "<|startoftext|>": 1, | |
| - "<|endoftext|>": 2, | |
| - "<|fim_pre|>": 3, | |
| - "<|fim_mid|>": 4, | |
| - "<|fim_suf|>": 5, | |
| - "<|im_start|>": 6, | |
| - "<|im_end|>": 7, | |
| - "<|tool_list_start|>": 8, | |
| - "<|tool_list_end|>": 9, | |
| - "<|tool_call_start|>": 10, | |
| - "<|tool_call_end|>": 11, | |
| - "<|tool_response_start|>": 12, | |
| - "<|tool_response_end|>": 13, | |
| - "<|cot_start|>": 64394, | |
| - "<|cot_end|>": 64395, | |
| - "<|review_start|>": 64396, | |
| - "<|review_end|>": 64397, | |
| - "<|file_start|>": 64398, | |
| - "<|file_end|>": 64399 | |