README.md

---
title: InferBench
emoji: 🥇
colorFrom: green
colorTo: indigo
sdk: gradio
app_file: dashboard/app.py
build_command: pip install -r requirements.txt
pinned: true
license: apache-2.0
short_description: A cost/quality/speed Leaderboard for Inference Providers!
sdk_version: 5.19.0
tags:
- leaderboard
---

# InferBench

Evaluate the quality and efficiency of image gen api's.

## Installation

### Install dependencies

Install dependencies with conda like that:
```
conda env create -f environment.yml
```

### Install uv

Install uv with pip like that:

```
uv venv --python 3.12
```

Then activate the environment:

```
source .venv/bin/activate
```

Then install the dependencies with uv:

```
uv sync --all-groups
```

## Usage

Create .env file with all the credentials you will need.

This is how you can generate the images.
```
python sample.py replicate draw_bench genai_bench geneval hps parti
```

This is how you would evaluate the benchmarks once you have all images:
```
python evaluate.py replicate draw_bench genai_bench geneval hps parti
```

## Leaderboard

The leaderboard is [hosted on Hugging Face](https://huggingface.co/spaces/PrunaAI/InferBench/tree/main) with gradio.

To run the dashboard locally, you can use the following command:

```
python dashboard/app.py
```

To deploy the dashboard to Hugging Face, you can use the following commands:

First, add the remote:

```
git remote add hf https://huggingface.co/spaces/PrunaAI/InferBench
```

Then push the changes of your branch to the remote:

```
git push hf $(git rev-parse --abbrev-ref HEAD):main --force
```