xai-org

/

grok-2

like 862

Follow

xAI 2.21k

Hey!

Still experimental, but I've just exported the tokenizer from the current tiktoken-based SGLang tokenizer.tok.json file to Hugging Face Tokenizers, so that's compatible with Transformers, Transformers.js, and the rest of the Hugging Face libraries leveraging Tokenizers, allowing users to easily play around and experiment with it at https://huggingface.co/alvarobartt/grok-2-tokenizer 🤗