AI & ML interests

None defined yet.

Recent Activity

azminetoushikwasi 
authored 15 papers 18 days ago
wop 
posted an update about 1 month ago
view post
Post
2197
bench-labs/GCTokenizer-v1 , a multilingual tokenizer which does not require a training corpus

bench-labs
developed **GCTokenizer-v1**, which is a multi-lingual tokenizer
Available in four sizes: 32K, 65K, 131K and 262K tokens "S, M, L, XL"
It utilizes an encoding scheme which allows it to handle characters in any language around the world

General (multi lingual)
Consensus (from multiple model tokenizers consensus)
Tokenizer

We included an implementation script too,
built like BPE- it can encode arbitrary text, most of the time, efficiently
wop 
posted an update about 1 month ago