|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
Anthropic's client side tokenizer. |
|
|
|
|
|
Accuracy compared to actual Claude 3 Haiku tokenizer (Claude 3 family has the same tokenizer): |
|
|
|
|
|
```python |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: Hello, world! This is a simple... |
|
|
Actual tokens: 17 |
|
|
Predicted tokens: 10 |
|
|
Accuracy: 58.82% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: The quick brown fox jumps over... |
|
|
Actual tokens: 19 |
|
|
Predicted tokens: 10 |
|
|
Accuracy: 52.63% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: In computer programming, a hel... |
|
|
Actual tokens: 29 |
|
|
Predicted tokens: 21 |
|
|
Accuracy: 72.41% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: Artificial intelligence (AI) i... |
|
|
Actual tokens: 30 |
|
|
Predicted tokens: 24 |
|
|
Accuracy: 80.00% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: The Eiffel Tower is a wrought-... |
|
|
Actual tokens: 56 |
|
|
Predicted tokens: 48 |
|
|
Accuracy: 85.71% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: To be, or not to be, that is t... |
|
|
Actual tokens: 60 |
|
|
Predicted tokens: 50 |
|
|
Accuracy: 83.33% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: In the beginning God created t... |
|
|
Actual tokens: 38 |
|
|
Predicted tokens: 31 |
|
|
Accuracy: 81.58% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: Four score and seven years ago... |
|
|
Actual tokens: 41 |
|
|
Predicted tokens: 34 |
|
|
Accuracy: 82.93% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: I have a dream that one day th... |
|
|
Actual tokens: 51 |
|
|
Predicted tokens: 43 |
|
|
Accuracy: 84.31% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: That's one small step for man,... |
|
|
Actual tokens: 22 |
|
|
Predicted tokens: 14 |
|
|
Accuracy: 63.64% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: Here are the key points about ... |
|
|
Actual tokens: 203 |
|
|
Predicted tokens: 195 |
|
|
Accuracy: 96.06% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: This appears to be an excerpt ... |
|
|
Actual tokens: 179 |
|
|
Predicted tokens: 180 |
|
|
Accuracy: 99.44% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: This is the beginning of the b... |
|
|
Actual tokens: 194 |
|
|
Predicted tokens: 191 |
|
|
Accuracy: 98.45% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: That is the opening lines of t... |
|
|
Actual tokens: 177 |
|
|
Predicted tokens: 163 |
|
|
Accuracy: 92.09% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: That's a powerful and inspirin... |
|
|
Actual tokens: 193 |
|
|
Predicted tokens: 190 |
|
|
Accuracy: 98.45% |
|
|
-------------------------------------------------- |
|
|
Tokenization results saved to __temp.txt.tokens |
|
|
Text: That famous quote is from Neil... |
|
|
Actual tokens: 131 |
|
|
Predicted tokens: 122 |
|
|
Accuracy: 93.13% |
|
|
-------------------------------------------------- |
|
|
Average accuracy: 82.69% |
|
|
``` |
