---
title: Email Triage OpenEnv
emoji: 📧
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
short_description: OpenEnv email triage benchmark — GRPO fine-tuned agent
tags:
  - openenv
  - agents
  - evaluation
  - email-triage
---

# 📧 Multi-Agent Email Triage Environment

A real-world RL environment that trains LLMs to intelligently triage
a corporate inbox — classifying, routing, prioritizing, and drafting
replies for incoming emails using GRPO reinforcement learning.

---

## 🔗 Deliverables

| Item | Link |
|------|------|
| 🤗 HF Space | *([this repo](https://huggingface.co/spaces/Vansh04092003/multi-agent-email-env2))* |
| 📓 Training Notebook | [Open in Kaggle](https://www.kaggle.com/code/vanshagrawal004/train/edit) |
| 📝 Blog / Writeup | `BLOG.md` in this repo |
| 📊 wandb Training Run | [Live metrics](https://wandb.ai/vansh-agrawal0409-oracle/email-triage-rl/reports/Email-Triage--VmlldzoxNjY4ODQyMw) |

---

## 🧠 What Was Trained

A **Qwen/Qwen2.5-3B-Instruct** model fine-tuned with **GRPO**
(Group Relative Policy Optimization) via HuggingFace TRL.

- **Base model**: `Qwen/Qwen2.5-3B-Instruct`
- **Method**: GRPO with LoRA (r=4, alpha=8, target: q_proj + v_proj)
- **Training steps**: 200
- **Reward signal**: Environment reward (priority + category +
  routing + reply quality + SLA compliance)

---

## 📈 Training Results

### Reward Curve


![image](https://cdn-uploads.huggingface.co/production/uploads/69d552f9a873ec8cafef7164/o8nn3RivN1UkmfUzKo95g.png)


*Red line = 10-step rolling average. Agent improves from 3.47 → 4.10+
over 200 GRPO steps.*

### Before vs After

![Before After](https://cdn-uploads.huggingface.co/production/uploads/69d552f9a873ec8cafef7164/QE0npeQNGtLGcTbjL-PzS.png)

### Key Numbers

| Metric | Baseline (untrained) | After GRPO (200 steps) |
|--------|----------------------|------------------------|
| Avg reward | 3.47 | 4.10+ |
| Best episode reward | 3.65 | 4.41 |
| Worst episode reward | 2.78 | 3.60 |
| Final evaluation score | — | **4.189 / 5.0** |
| Correct routing | ~40% | **100% (5/5)** |
| Null reply drafts | frequent | eliminated |
| Invalid categories | occasional | eliminated |
| Escalation budget abuse | frequent | eliminated |

### Before vs After Routing (Final Evaluation)

| Email | Untrained | Trained | Per-email Reward |
|-------|-----------|---------|-----------------|
| URGENT: My account has been hacked! | general_inquiry → support_tier1 | customer_complaint → support_tier2 ✅ | 0.705 |
| Congratulations! You've WON $1,000,000! | urgent → support_tier1 | spam_phishing → trash ✅ | 0.840 |
| Team lunch this Friday | low → trash | internal_hr → hr ✅ | 0.988 |
| Invoice #INV-2024-0042 - Payment Overdue | medium → support_tier1 | billing_inquiry → billing ✅ | 0.798 |
| Interested in your Enterprise plan | general_inquiry → archive | sales_lead → sales ✅ | 0.858 |
| **Total** | | | **4.189 / ~5.0** |

---

## 🎯 Problem Statement

Every company receives hundreds of emails daily. Manually reading,
classifying, and routing each one is time-consuming and error-prone.
This environment trains an LLM agent to handle this automatically.

This falls under **Theme #3.2 — Personalized Tasks** (World Modeling).

---

## 🌍 Environment Interface

The environment follows the OpenEnv-style interface:

- `reset()` — starts a fresh episode
- `step(action)` — applies action, returns next observation + reward
- `state()` — returns current internal state

### Observation Space

| Field | Description |
|-------|-------------|
| `current_email` | Email to triage (header + body) |
| `inbox` | Full list of emails in episode |
| `processed` | IDs of already handled emails |
| `escalation_budget_remaining` | Remaining `flag_review` capacity |
| `team_queue_remaining` | Capacity per routing destination |
| `active_sla_warnings` | Emails close to SLA deadline |
| `sla_breaches_so_far` | Number of SLA breaches triggered |

### Action Space

| Field | Valid Values |
|-------|-------------|
| `email_id` | Copy exactly from current_email |
| `priority` | `urgent` \| `high` \| `medium` \| `low` \| `spam` |
| `category` | `customer_complaint` \| `billing_inquiry` \| `technical_support` \| `sales_lead` \| `internal_hr` \| `legal_compliance` \| `spam_phishing` \| `general_inquiry` |
| `route_to` | `support_tier1` \| `support_tier2` \| `billing` \| `sales` \| `legal` \| `hr` \| `management` \| `trash` \| `archive` |
| `summary` | ≤280 chars |
| `flag_review` | `true` or `false` (uses escalation budget) |
| `reply_draft` | Professional reply to sender |

### Tasks

| Task | Emails | Difficulty | Expected Score |
|------|--------|------------|----------------|
| easy | 5 | Easy | 0.75–0.99 |
| medium | 8 | Medium | 0.55–0.80 |
| hard | 12 | Hard | 0.35–0.65 |

### Reward Design

Per-email scoring:
- ✅ Correct priority → ~0.2 pts
- ✅ Correct category → ~0.2 pts
- ✅ Correct routing → ~0.3 pts
- ✅ Reply draft quality → ~0.2 pts (20% of score)
- ✅ Appropriate escalation
- ❌ Penalty for SLA breaches
- ❌ Penalty for invalid field values

---

## 🏋️ Training

```python
MODEL_NAME      = "Qwen/Qwen2.5-3B-Instruct"
TRAINING_STEPS  = 200
BATCH_SIZE      = 1
NUM_GENERATIONS = 4
learning_rate   = 2e-6
lora_r          = 4
```

To run training:
```bash
pip install trl transformers datasets peft accelerate bitsandbytes wandb
# Then open the Colab notebook linked above
```

---

## 🔍 Inference

```python
import requests

BASE_URL = "https://Vansh04092003-multi-agent-email-env2.hf.space"

# Start episode
obs = requests.post(f"{BASE_URL}/reset").json()["observation"]

# Take action
action = {
    "email_id":    "e001",
    "priority":    "urgent",
    "category":    "customer_complaint",
    "route_to":    "support_tier2",
    "summary":     "Account hacked, needs immediate lock.",
    "flag_review": True,
    "reply_draft": "We are securing your account immediately."
}
result = requests.post(f"{BASE_URL}/step", json=action).json()
print(f"Reward: {result['reward']}")
```

---

## 📁 Repository Structure

```
├── README.md
├── BLOG.md
├── openenv.yaml
├── training/
│   ├── notebook_training.ipynb
│   ├── train_grpo.py
│   └── plots/
│       ├── reward_curve.png
│       └── before_after.png
└── src/
    └── openenv_email_triage/
        ├── environment.py
        └── models.py
```

---

## 🐳 Local Run

```bash
export HF_TOKEN=your_token_here
uv run server
```

## Docker

```bash
docker build -t email-triage-env .
docker run -p 7860:7860 -e HF_TOKEN=$HF_TOKEN email-triage-env
```

## Validate

```bash
openenv validate
```