docs/devplan.md

GAIA Agent Development Plan

This document outlines a structured approach to developing an agent that can successfully solve a subset of the GAIA benchmark, focusing on understanding the task, designing the agent architecture, and planning the development process.

I. Understanding the Task & Data:

1. Analyze common_questions.json:

   • Structure: Each entry has task_id, Question, Level, Final answer, and sometimes file_name.
   • Question Types: Identify patterns:
     • Direct information retrieval (e.g., "How many studio albums...").
     • Web search required (e.g., "On June 6, 2023, an article...").
     • File-based questions (audio, images, code - indicated by file_name).
     • Logic/reasoning puzzles (e.g., the table-based commutativity question, reversed sentence).
     • Multistep questions.
   • Answer Format: Observe the format of Final answer for each type. Note the guidance in docs/submission_instructions.md regarding formatting (numbers, few words, comma-separated lists).
   • File Dependencies: List all unique file_name extensions to understand what file processing capabilities are needed (e.g., .mp3, .png, .py, .xlsx).

2. Review Project Context:

   • Agent Interface: The agent will need to fit into the BasicAgent structure in app.py (i.e., an __init__ and a __call__(self, question: str) -> str method).
   • Evaluation: Keep docs/testing_recipe.md and the normalize function in mind for how answers will be compared.
   • Model: The agent will use an LLM (like the Llama 3 model mentioned in docs/log.md).

II. Agent Architecture Design (Conceptual):

1. Core Agent Loop (MyAgent.answer or MyAgent.__call__):

   • Input: Question string (and task_id/file_name if passed separately or parsed from a richer input object).
   • Step 1: Question Analysis & Planning:
     • Use the LLM to understand the question.
     • Determine the type of question (web search, file processing, direct knowledge, etc.).
     • Identify if any tools are needed.
     • Formulate a high-level plan (e.g., "Search web for X, then extract Y from the page").
   • Step 2: Tool Selection & Execution (if needed):
     • Based on the plan, select and invoke appropriate tools.
     • Pass necessary parameters to tools (e.g., search query, file path).
     • Collect tool outputs.
   • Step 3: Information Synthesis & Answer Generation:
     • Use the LLM to process tool outputs and any retrieved information.
     • Generate the final answer string.
   • Step 4: Answer Formatting:
     • Ensure the answer conforms to the expected format (using guidance from common_questions.json examples and docs/submission_instructions.md). This might involve specific post-processing rules or prompting the LLM for a specific format.
   • Output: Return the formatted answer string.

2. Key Modules/Components:

   • LLM Interaction Module:
     • Handles communication with the chosen LLM (e.g., GPT4All Llama 3).
     • Manages prompt construction (system prompts, user prompts, few-shot examples if useful).
     • Parses LLM responses.
   • Tool Library: A collection of functions/classes that the agent can call.
     • WebSearchTool:
       • Input: Search query.
       • Action: Uses a search engine API (or simulates browsing if necessary, though direct API is better).
       • Output: List of search results (titles, snippets, URLs) or page content.
     • FileReaderTool:
       • Input: File path (derived from file_name and task_id to locate/fetch the file).
       • Action: Reads content based on file type.
         • Text files (.py): Read as string.
         • Spreadsheets (.xlsx): Parse relevant data (requires a library like pandas or openpyxl).
         • Audio files (.mp3): Transcribe to text (requires a speech-to-text model/API).
         • Image files (.png): Describe image content or extract text (requires a vision model/API or OCR).
       • Output: Processed content (text, structured data).
     • CodeInterpreterTool (for .py files like in task f918266a-b3e0-4914-865d-4faa564f1aef):
       • Input: Python code string.
       • Action: Executes the code in a sandboxed environment.
       • Output: Captured stdout/stderr or final expression value.
     • (Potentially) KnowledgeBaseTool: If there's a way to pre-process or index relevant documents/FAQs for faster lookups (though most GAIA questions imply dynamic information retrieval).
   • File Management/Access:
     • Mechanism to locate/download files associated with task_id and file_name. The API endpoint GET /files/{task_id} from docs/API.md is relevant here. For local testing with common_questions.json, ensure these files are available locally.
   • Prompt Engineering Strategy:
     • Develop a set of system prompts to guide the agent's behavior (e.g., "You are a helpful AI assistant designed to answer questions from the GAIA benchmark...").
     • Develop task-specific prompts or prompt templates for different question types or tool usage.
     • Incorporate answer formatting instructions into prompts.

III. Development & Testing Strategy:

1. Environment Setup:
   • Install necessary Python libraries for LLM interaction, web requests, file processing (e.g., requests, beautifulsoup4 (for web scraping if needed), pandas, Pillow (for images), speech recognition libraries, etc.).
2. Iterative Implementation:
   • Phase 1: Basic LLM Agent: Start with an agent that only uses the LLM for direct-answer questions (no tools).
   • Phase 2: Web Search Integration: Implement the WebSearchTool and integrate it for questions requiring web lookups.
   • Phase 3: File Handling:
     • Implement FileReaderTool for one file type at a time (e.g., start with .txt or .py, then .mp3, .png, .xlsx).
     • Implement CodeInterpreterTool.
   • Phase 4: Complex Reasoning & Multi-step: Refine the planning and synthesis capabilities of the LLM to handle more complex, multi-step questions that might involve multiple tool uses.
3. Testing:
   • Use common_questions.json as the primary test set.
   • Adapt the script from docs/testing_recipe.md to run your agent against these questions and compare outputs.
   • Focus on one question type or task_id at a time for debugging.
   • Log agent's internal "thoughts" (plan, tool calls, tool outputs) for easier debugging.

IV. Pre-computation/Pre-analysis (before coding):

1. Map Question Types to Tools: For each question in common_questions.json, manually note which tool(s) would ideally be used. This helps prioritize tool development.
   • Example:
     • 8e867cd7-cff9-4e6c-867a-ff5ddc2550be (Mercedes Sosa albums): WebSearchTool
     • cca530fc-4052-43b2-b130-b30968d8aa44 (Chess): FileReaderTool (image) + Vision/Chess Engine Tool (or very advanced LLM vision)
     • 99c9cc74-fdc8-46c6-8f8d-3ce2d3bfeea3 (Pie ingredients): FileReaderTool (audio) + SpeechToText
     • f918266a-b3e0-4914-865d-4faa564f1aef (Python output): FileReaderTool (code) + CodeInterpreterTool
2. Define Tool Interfaces: Specify the exact input/output signature for each planned tool.