Token Probability Analyzer

A web application that analyzes token probabilities using various language models. This tool helps visualize and understand how language models predict tokens in a given text sequence.

Features

Support for multiple language models (GPT-2, TinyLlama, etc.)
Token-by-token probability analysis
Percentile scoring for token probabilities
Top-k predictions for each position
Joint and average log likelihood calculations

Setup

Install the required dependencies:

pip install -r requirements.txt

Run the application:

python app.py

Open your browser and navigate to http://localhost:5000

Usage

Select a language model from the dropdown menu
Enter your text in the input field
Click "Analyze" to see the token probabilities and predictions

Technical Details

The application uses Flask for the backend and provides a simple web interface. It leverages the Hugging Face Transformers library to load and run various language models for token probability analysis.