Spaces:
Sleeping
Sleeping
File size: 455 Bytes
d716035 9aa2b0c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
---
title: LanguageBPE
emoji: 🌖
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 4.37.1
app_file: app.py
pinned: false
license: mit
---
# App to visualize results of BPE tokenizer trained on wikipedia and opus dataset.
## Features
- Input/output text: Write some word(s) and see tokenized output
- Tokenizer: Select tokenizer
## Usage
- Enter sentence in Hindi, English or both languages
- Select tokenizer model
- Hit submit and see result
|