README.md

---
title: LanguageBPE
emoji: 🌖
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 4.37.1
app_file: app.py
pinned: false
license: mit
---

# App to visualize results of BPE tokenizer trained on wikipedia and opus dataset.

## Features
- Input/output text: Write some word(s) and see tokenized output
- Tokenizer: Select tokenizer

## Usage
- Enter sentence in Hindi, English or both languages
- Select tokenizer model
- Hit submit and see result