File size: 2,353 Bytes
0af0679 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
# LLM Router & Evaluator-Optimizer Workflow
This project demonstrates a simple, modular workflow for orchestrating multiple LLM tasks using OpenAI's API, with a focus on clarity and extensibility for beginners.
## Workflow Overview
1. **User Input**: The user provides a request (e.g., a math problem, translation, or text to summarize).
2. **Router LLM**: A general-purpose LLM analyzes the input and decides which specialized LLM (math, translation, or summarization) should handle it.
3. **Specialized LLMs**: Each task (math, translation, summarization) is handled by a dedicated prompt to the LLM.
4. **Evaluator-Optimizer Loop**:
- The solution from the specialized LLM is evaluated by an evaluator LLM.
- If the evaluator deems the solution incorrect or unhelpful, it provides feedback.
- The generator LLM retries with the feedback, up to 3 attempts.
- If accepted, the result is returned to the user.
## Key Components
- **Router**: Determines the type of task (Math, Translate, Summarize) using a single-word response from the LLM.
- **Specialized LLMs**: Prompts tailored for each task, leveraging OpenAI's chat models.
- **Evaluator-Optimizer**: Uses a Pydantic schema and OpenAI's structured output to validate and refine the solution, ensuring quality and correctness.
## Technologies Used
- Python 3.8+
- [OpenAI Python SDK (v1.91.0+)](https://github.com/openai/openai-python)
- [Pydantic](https://docs.pydantic.dev/)
## Setup
1. **Install dependencies**:
```bash
pip install openai pydantic
```
2. **Set your OpenAI API key**:
```bash
export OPENAI_API_KEY=sk-...
```
3. **Run the script**:
```bash
python main.py
```
## Example Usage
- **Math**: `calculate 9+2`
- **Translate**: `Translate 'Hello, how are you?' to French.`
- **Summarize**: `Summarize: The cat sat on the mat. It was sunny.`
The router will direct your request to the appropriate LLM, and the evaluator will ensure the answer is correct or provide feedback for improvement.
## Notes
- The workflow is designed for learning and can be extended with more tasks or more advanced routing/evaluation logic.
- The evaluator uses OpenAI's structured output (with Pydantic) for robust, type-safe validation.
---
Feel free to experiment and expand this workflow for your own LLM projects!
|