LiquidAI/LFM2-Tokenizer
Formatted text
<|startoftext|><|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello! How are you?<|im_end|>
<|im_start|>assistant
I'm doing well, thank you!<|im_end|>
<|im_start|>user
What's the weather like?<|im_end|>
<|im_start|>assistant
Special tokens
- bos_token: <|startoftext|>
- eos_token: <|im_end|>
- pad_token: <|pad|>
- sep_token: None
- cls_token: None
- mask_token: None
Added special tokens
- "<|pad|>": 0,
- "<|startoftext|>": 1,
- "<|endoftext|>": 2,
- "<|fim_pre|>": 3,
- "<|fim_mid|>": 4,
- "<|fim_suf|>": 5,
- "<|im_start|>": 6,
- "<|im_end|>": 7,
- "<|tool_list_start|>": 8,
- "<|tool_list_end|>": 9,
- "<|tool_call_start|>": 10,
- "<|tool_call_end|>": 11,
- "<|tool_response_start|>": 12,
- "<|tool_response_end|>": 13,
- "<|cot_start|>": 64394,
- "<|cot_end|>": 64395,
- "<|review_start|>": 64396,
- "<|review_end|>": 64397,
- "<|file_start|>": 64398,
- "<|file_end|>": 64399