metadata
license: apache-2.0
The Hugging Face fast tokenizer for LLM-jp 13B.
The vocab size is 50,570.
Requirements:
- transformers>=4.34.0
- tokenizers>=0.14.0
- torch
Usage:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v21b3")