README.md

metadata

license: gpl-2.0
language:
  - en
  - ja
tags:
  - tokenizer
  - novelai
  - sentencepiece

NovelAI Tokenizer v1

This repository is exactly the same as NovelAI/nerdstash-tokenizer-v1, but the config has been changed to address the following points (the sp model itself is not changed).

• Load as T5Tokenizer
• Enable to decode digits (In the original, digits are registered as additional_special_tokens, so if skip_special_tokens=True when decoding, the digits are also skipped.)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mkshing/novelai-tokenizer-v1", use_fast=False)

text = "1+1=3"
tokenizer.decode(tokenizer.encode(text), skip_special_tokens=True)
# '1+1=3'