File size: 7,631 Bytes
3a78b26 736bdeb 3a78b26 736bdeb 3a78b26 736bdeb 3a78b26 736bdeb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
# GAIA Agent Development Plan
This document outlines a structured approach to developing an agent that can successfully solve a subset of the GAIA benchmark, focusing on understanding the task, designing the agent architecture, and planning the development process.
**I. Understanding the Task & Data:**
1. **Analyze common_questions.json:**
* **Structure:** Each entry has `task_id`, `Question`, `Level`, `Final answer`, and sometimes `file_name`.
* **Question Types:** Identify patterns:
* Direct information retrieval (e.g., "How many studio albums...").
* Web search required (e.g., "On June 6, 2023, an article...").
* File-based questions (audio, images, code - indicated by `file_name`).
* Logic/reasoning puzzles (e.g., the table-based commutativity question, reversed sentence).
* Multistep questions.
* **Answer Format:** Observe the format of `Final answer` for each type. Note the guidance in `docs/submission_instructions.md` regarding formatting (numbers, few words, comma-separated lists).
* **File Dependencies:** List all unique `file_name` extensions to understand what file processing capabilities are needed (e.g., `.mp3`, `.png`, `.py`, `.xlsx`).
2. **Review Project Context:**
* **Agent Interface:** The agent will need to fit into the `BasicAgent` structure in `app.py` (i.e., an `__init__` and a `__call__(self, question: str) -> str` method).
* **Evaluation:** Keep `docs/testing_recipe.md` and the `normalize` function in mind for how answers will be compared.
* **Model:** The agent will use an LLM (like the Llama 3 model mentioned in `docs/log.md`).
**II. Agent Architecture Design (Conceptual):**
1. **Core Agent Loop (`MyAgent.answer` or `MyAgent.__call__`):**
* **Input:** Question string (and `task_id`/`file_name` if passed separately or parsed from a richer input object).
* **Step 1: Question Analysis & Planning:**
* Use the LLM to understand the question.
* Determine the type of question (web search, file processing, direct knowledge, etc.).
* Identify if any tools are needed.
* Formulate a high-level plan (e.g., "Search web for X, then extract Y from the page").
* **Step 2: Tool Selection & Execution (if needed):**
* Based on the plan, select and invoke appropriate tools.
* Pass necessary parameters to tools (e.g., search query, file path).
* Collect tool outputs.
* **Step 3: Information Synthesis & Answer Generation:**
* Use the LLM to process tool outputs and any retrieved information.
* Generate the final answer string.
* **Step 4: Answer Formatting:**
* Ensure the answer conforms to the expected format (using guidance from common_questions.json examples and `docs/submission_instructions.md`). This might involve specific post-processing rules or prompting the LLM for a specific format.
* **Output:** Return the formatted answer string.
2. **Key Modules/Components:**
* **LLM Interaction Module:**
* Handles communication with the chosen LLM (e.g., GPT4All Llama 3).
* Manages prompt construction (system prompts, user prompts, few-shot examples if useful).
* Parses LLM responses.
* **Tool Library:** A collection of functions/classes that the agent can call.
* `WebSearchTool`:
* Input: Search query.
* Action: Uses a search engine API (or simulates browsing if necessary, though direct API is better).
* Output: List of search results (titles, snippets, URLs) or page content.
* `FileReaderTool`:
* Input: File path (derived from `file_name` and `task_id` to locate/fetch the file).
* Action: Reads content based on file type.
* Text files (`.py`): Read as string.
* Spreadsheets (`.xlsx`): Parse relevant data (requires a library like `pandas` or `openpyxl`).
* Audio files (`.mp3`): Transcribe to text (requires a speech-to-text model/API).
* Image files (`.png`): Describe image content or extract text (requires a vision model/API or OCR).
* Output: Processed content (text, structured data).
* `CodeInterpreterTool` (for `.py` files like in task `f918266a-b3e0-4914-865d-4faa564f1aef`):
* Input: Python code string.
* Action: Executes the code in a sandboxed environment.
* Output: Captured stdout/stderr or final expression value.
* *(Potentially)* `KnowledgeBaseTool`: If there's a way to pre-process or index relevant documents/FAQs for faster lookups (though most GAIA questions imply dynamic information retrieval).
* **File Management/Access:**
* Mechanism to locate/download files associated with `task_id` and `file_name`. The API endpoint `GET /files/{task_id}` from `docs/API.md` is relevant here. For local testing with common_questions.json, ensure these files are available locally.
* **Prompt Engineering Strategy:**
* Develop a set of system prompts to guide the agent's behavior (e.g., "You are a helpful AI assistant designed to answer questions from the GAIA benchmark...").
* Develop task-specific prompts or prompt templates for different question types or tool usage.
* Incorporate answer formatting instructions into prompts.
**III. Development & Testing Strategy:**
1. **Environment Setup:**
* Install necessary Python libraries for LLM interaction, web requests, file processing (e.g., `requests`, `beautifulsoup4` (for web scraping if needed), `pandas`, `Pillow` (for images), speech recognition libraries, etc.).
2. **Iterative Implementation:**
* **Phase 1: Basic LLM Agent:** Start with an agent that only uses the LLM for direct-answer questions (no tools).
* **Phase 2: Web Search Integration:** Implement the `WebSearchTool` and integrate it for questions requiring web lookups.
* **Phase 3: File Handling:**
* Implement `FileReaderTool` for one file type at a time (e.g., start with `.txt` or `.py`, then `.mp3`, `.png`, `.xlsx`).
* Implement `CodeInterpreterTool`.
* **Phase 4: Complex Reasoning & Multi-step:** Refine the planning and synthesis capabilities of the LLM to handle more complex, multi-step questions that might involve multiple tool uses.
3. **Testing:**
* Use `common_questions.json` as the primary test set.
* Adapt the script from `docs/testing_recipe.md` to run your agent against these questions and compare outputs.
* Focus on one question type or `task_id` at a time for debugging.
* Log agent's internal "thoughts" (plan, tool calls, tool outputs) for easier debugging.
**IV. Pre-computation/Pre-analysis (before coding):**
1. **Map Question Types to Tools:** For each question in common_questions.json, manually note which tool(s) would ideally be used. This helps prioritize tool development.
* Example:
* `8e867cd7-cff9-4e6c-867a-ff5ddc2550be` (Mercedes Sosa albums): WebSearchTool
* `cca530fc-4052-43b2-b130-b30968d8aa44` (Chess): FileReaderTool (image) + Vision/Chess Engine Tool (or very advanced LLM vision)
* `99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3` (Pie ingredients): FileReaderTool (audio) + SpeechToText
* `f918266a-b3e0-4914-865d-4faa564f1aef` (Python output): FileReaderTool (code) + CodeInterpreterTool
2. **Define Tool Interfaces:** Specify the exact input/output signature for each planned tool. |