Patenty-Test2-Radix-65536

Patenty-Test2-Radix-65536 / binaryllm_vocab.json

Duplicate from PhysiQuanty/Patent-Test-Radix-65536-AutoTokenizer_FineTune

8595a97 8 months ago

249 Bytes

{
  "base_vocab_size": 65536,
  "vocab_size": 65538,
  "bos_token": "<BOS>",
  "bos_token_id": 65536,
  "eos_token": "<EOS>",
  "eos_token_id": 65537,
  "unk_token": "<EOS>",
  "unk_token_id": 65537,
  "pad_token": "<EOS>",
  "pad_token_id": 65537
}

	{
	"base_vocab_size": 65536,
	"vocab_size": 65538,
	"bos_token": "<BOS>",
	"bos_token_id": 65536,
	"eos_token": "<EOS>",
	"eos_token_id": 65537,
	"unk_token": "<EOS>",
	"unk_token_id": 65537,
	"pad_token": "<EOS>",
	"pad_token_id": 65537
	}