LiquidAI/LFM2-Tokenizer

Formatted text

<|startoftext|><|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello! How are you?<|im_end|>
<|im_start|>assistant
I'm doing well, thank you!<|im_end|>
<|im_start|>user
What's the weather like?<|im_end|>
<|im_start|>assistant

Special tokens

• bos_token: <|startoftext|>
• eos_token: <|im_end|>
• pad_token: <|pad|>
• sep_token: None
• cls_token: None
• mask_token: None

Added special tokens

• "<|pad|>": 0,
• "<|startoftext|>": 1,
• "<|endoftext|>": 2,
• "<|fim_pre|>": 3,
• "<|fim_mid|>": 4,
• "<|fim_suf|>": 5,
• "<|im_start|>": 6,
• "<|im_end|>": 7,
• "<|tool_list_start|>": 8,
• "<|tool_list_end|>": 9,
• "<|tool_call_start|>": 10,
• "<|tool_call_end|>": 11,
• "<|tool_response_start|>": 12,
• "<|tool_response_end|>": 13,
• "<|cot_start|>": 64394,
• "<|cot_end|>": 64395,
• "<|review_start|>": 64396,
• "<|review_end|>": 64397,
• "<|file_start|>": 64398,
• "<|file_end|>": 64399