finalize hfcontext7 v2

README.md

@@ -1,6 +1,6 @@
 ---
-title: HfContext7
-emoji: 🐠
+title: HFContext7
+emoji: 🤗
 colorFrom: pink
 colorTo: yellow
 sdk: gradio
@@ -9,8 +9,161 @@ app_file: app.py
 pinned: false
 tags:
   - mcp-server-track
+  - Agents-MCP-Hackathon
 license: apache-2.0
 short_description: Latest 🤗 documentation for LLMs and AI code editors
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🐠 HfContext7 MCP Server
+
+<p align="center">
+  <em>Real-time HuggingFace Documentation for AI Coding Assistants and LLMs</em>
+</p>
+
+
+## 🚀 What is HfContext7?
+---
+**HfContext7** is a specialized Model Context Protocol (MCP) server designed to provide AI coding assistants and Large Language Models (LLMs) with **real-time, up-to-date documentation** from the HuggingFace ecosystem. 
+
+Inspired by the groundbreaking [Context7 MCP Server](https://github.com/upstash/context7), HfContext7 specifically targets the rapidly evolving HuggingFace libraries, ensuring your AI assistant always has the latest and most accurate information.
+
+---
+
+## ❌ The Problem We Solve
+
+The HuggingFace ecosystem evolves at lightning speed. New APIs, features, and best practices emerge constantly, making it challenging for LLMs trained on static datasets to keep up. This leads to:
+
+- ❌ Outdated code examples based on old training data
+- ❌ Hallucinated APIs that no longer exist or never existed
+- ❌ Generic answers that don't reflect current HuggingFace best practices
+- ❌ Confusion between similar HuggingFace libraries (Transformers, Diffusers, PEFT, etc.)
+
+---
+
+## ✅ How HfContext7 Solves It
+
+HfContext7 MCP server solves these issues by:
+
+- **Real-time Documentation**: Fetching the latest HuggingFace documentation directly from official sources.
+- **Semantic Search**: Leveraging advanced embeddings and vector search (powered by Milvus and OpenAI embeddings) to retrieve highly relevant documentation snippets.
+- **Seamless Integration**: Easily integrates with popular AI coding assistants (Cursor, Claude Desktop, Windsurf, etc.) via MCP.
+
+Simply add `use hfcontext7` to your prompt:
+
+```txt
+Create a LoRA fine-tuning script for Llama with PEFT. use hfcontext7
+```
+
+```txt
+Set up a Gradio interface with Diffusers for image generation. use hfcontext7
+```
+
+HfContext7 instantly provides your AI assistant with accurate, up-to-date HuggingFace documentation and code examples.
+
+---
+
+## 📚 Supported HuggingFace Libraries (28+)
+
+HfContext7 supports a wide range of HuggingFace libraries, including:
+
+- **Transformers** – State-of-the-art NLP models
+- **Diffusers** – Diffusion models for image/audio generation
+- **PEFT** – Parameter-Efficient Fine-Tuning (LoRA, etc.)
+- **TRL** – Transformer Reinforcement Learning
+- **Datasets** – Access and share datasets
+- **Accelerate** – Simplified distributed training
+- **Text Generation Inference (TGI)** – High-performance inference
+- **Optimum** – Hardware-optimized transformers
+- **AutoTrain** – No-code training platform
+- **bitsandbytes** – 8-bit optimizers and quantization
+
+...and many more! (Full list available in `repos_config.json`)
+
+---
+
+## 🛠️ Available Tools
+
+HfContext7 provides essential tools for AI coding assistants:
+
+- **`list_huggingface_resources_names`**: Lists all available HuggingFace resources in the documentation database.
+- **`get_huggingface_documentation`**: Retrieves relevant documentation for a specific topic, optionally filtered by resource names.
+
+---
+
+## ⚙️ Quick Start
+
+### 1. Clone and Install
+
+```bash
+git clone <repo-url>
+cd hfcontext7
+pip install -r requirements.txt
+```
+
+### 2. Set OpenAI API Key
+
+```bash
+echo "OPENAI_API_KEY=your_key_here" > .env
+```
+
+### 3. Build Documentation Database
+
+```bash
+python make_docs.py
+python make_rag_db.py
+```
+
+### 4. Run the Server
+
+```bash
+python app.py
+```
+
+---
+
+## 🔌 MCP Client Setup
+
+### Cursor & Claude Desktop Example
+
+```json
+{
+  "mcpServers": {
+    "hfcontext7": {
+      "command": "python",
+      "args": ["/path/to/hfcontext7/app.py"],
+      "env": {
+        "OPENAI_API_KEY": "your_openai_api_key"
+      }
+    }
+  }
+}
+```
+
+---
+
+## 💡 How It Works
+
+HfContext7 MCP server workflow:
+
+1. **Crawls** official HuggingFace documentation repositories.
+2. **Organizes** documentation using semantic embeddings (OpenAI embeddings + Milvus vector DB).
+3. **Serves** relevant documentation snippets directly into your AI assistant's context via MCP.
+4. **Updates** easily—just re-run the build scripts to refresh documentation.
+
+---
+
+## 🌟 Inspired by Context7
+
+This project was heavily inspired by the incredible [Context7 MCP Server](https://github.com/upstash/context7) by Upstash, which revolutionized how LLMs access general development documentation. While Context7 provides broad coverage across many frameworks, HfContext7 focuses specifically on the HuggingFace ecosystem, providing deeper, more specialized knowledge for AI/ML development.
+
+---
+
+## 📄 License
+
+Apache 2.0
+
+---
+
+<p align="center">
+  <strong>Stop fighting outdated HuggingFace examples. Get the latest docs in every prompt. 🚀</strong>
+</p>

Relevant README 1.md

@@ -0,0 +1,311 @@
+<h1 align="center">Crawl4AI RAG MCP Server</h1>
+
+<p align="center">
+  <em>Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants</em>
+</p>
+
+A powerful implementation of the [Model Context Protocol (MCP)](https://modelcontextprotocol.io) integrated with [Crawl4AI](https://crawl4ai.com) and [Supabase](https://supabase.com/) for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.
+
+With this MCP server, you can <b>scrape anything</b> and then <b>use that knowledge anywhere</b> for RAG.
+
+The primary goal is to bring this MCP server into [Archon](https://github.com/coleam00/Archon) as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama.
+
+## Overview
+
+This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the [Mem0 MCP server template](https://github.com/coleam00/mcp-mem0/) I provided on my channel previously.
+
+The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality:
+- **Contextual Embeddings** for enriched semantic understanding
+- **Hybrid Search** combining vector and keyword search
+- **Agentic RAG** for specialized code example extraction
+- **Reranking** for improved result relevance using cross-encoder models
+
+See the [Configuration section](#configuration) below for details on how to enable and configure these strategies.
+
+## Vision
+
+The Crawl4AI RAG MCP server is just the beginning. Here's where we're headed:
+
+1. **Integration with Archon**: Building this system directly into [Archon](https://github.com/coleam00/Archon) to create a comprehensive knowledge engine for AI coding assistants to build better AI agents.
+
+2. **Multiple Embedding Models**: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy.
+
+3. **Advanced RAG Strategies**: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic "naive lookups" and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon.
+
+4. **Enhanced Chunking Strategy**: Implementing a Context 7-inspired chunking approach that focuses on examples and creates distinct, semantically meaningful sections for each chunk, improving retrieval precision.
+
+5. **Performance Optimization**: Increasing crawling and indexing speed to make it more realistic to "quickly" index new documentation to then leverage it within the same prompt in an AI coding assistant.
+
+## Features
+
+- **Smart URL Detection**: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)
+- **Recursive Crawling**: Follows internal links to discover content
+- **Parallel Processing**: Efficiently crawls multiple pages simultaneously
+- **Content Chunking**: Intelligently splits content by headers and size for better processing
+- **Vector Search**: Performs RAG over crawled content, optionally filtering by data source for precision
+- **Source Retrieval**: Retrieve sources available for filtering to guide the RAG process
+
+## Tools
+
+The server provides essential web crawling and search tools:
+
+### Core Tools (Always Available)
+
+1. **`crawl_single_page`**: Quickly crawl a single web page and store its content in the vector database
+2. **`smart_crawl_url`**: Intelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
+3. **`get_available_sources`**: Get a list of all available sources (domains) in the database
+4. **`perform_rag_query`**: Search for relevant content using semantic search with optional source filtering
+
+### Conditional Tools
+
+5. **`search_code_examples`** (requires `USE_AGENTIC_RAG=true`): Search specifically for code examples and their summaries from crawled documentation. This tool provides targeted code snippet retrieval for AI coding assistants.
+
+## Prerequisites
+
+- [Docker/Docker Desktop](https://www.docker.com/products/docker-desktop/) if running the MCP server as a container (recommended)
+- [Python 3.12+](https://www.python.org/downloads/) if running the MCP server directly through uv
+- [Supabase](https://supabase.com/) (database for RAG)
+- [OpenAI API key](https://platform.openai.com/api-keys) (for generating embeddings)
+
+## Installation
+
+### Using Docker (Recommended)
+
+1. Clone this repository:
+   ```bash
+   git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
+   cd mcp-crawl4ai-rag
+   ```
+
+2. Build the Docker image:
+   ```bash
+   docker build -t mcp/crawl4ai-rag --build-arg PORT=8051 .
+   ```
+
+3. Create a `.env` file based on the configuration section below
+
+### Using uv directly (no Docker)
+
+1. Clone this repository:
+   ```bash
+   git clone https://github.com/coleam00/mcp-crawl4ai-rag.git
+   cd mcp-crawl4ai-rag
+   ```
+
+2. Install uv if you don't have it:
+   ```bash
+   pip install uv
+   ```
+
+3. Create and activate a virtual environment:
+   ```bash
+   uv venv
+   .venv\Scripts\activate
+   # on Mac/Linux: source .venv/bin/activate
+   ```
+
+4. Install dependencies:
+   ```bash
+   uv pip install -e .
+   crawl4ai-setup
+   ```
+
+5. Create a `.env` file based on the configuration section below
+
+## Database Setup
+
+Before running the server, you need to set up the database with the pgvector extension:
+
+1. Go to the SQL Editor in your Supabase dashboard (create a new project first if necessary)
+
+2. Create a new query and paste the contents of `crawled_pages.sql`
+
+3. Run the query to create the necessary tables and functions
+
+## Configuration
+
+Create a `.env` file in the project root with the following variables:
+
+```
+# MCP Server Configuration
+HOST=0.0.0.0
+PORT=8051
+TRANSPORT=sse
+
+# OpenAI API Configuration
+OPENAI_API_KEY=your_openai_api_key
+
+# LLM for summaries and contextual embeddings
+MODEL_CHOICE=gpt-4.1-nano
+
+# RAG Strategies (set to "true" or "false", default to "false")
+USE_CONTEXTUAL_EMBEDDINGS=false
+USE_HYBRID_SEARCH=false
+USE_AGENTIC_RAG=false
+USE_RERANKING=false
+
+# Supabase Configuration
+SUPABASE_URL=your_supabase_project_url
+SUPABASE_SERVICE_KEY=your_supabase_service_key
+```
+
+### RAG Strategy Options
+
+The Crawl4AI RAG MCP server supports four powerful RAG strategies that can be enabled independently:
+
+#### 1. **USE_CONTEXTUAL_EMBEDDINGS**
+When enabled, this strategy enhances each chunk's embedding with additional context from the entire document. The system passes both the full document and the specific chunk to an LLM (configured via `MODEL_CHOICE`) to generate enriched context that gets embedded alongside the chunk content.
+
+- **When to use**: Enable this when you need high-precision retrieval where context matters, such as technical documentation where terms might have different meanings in different sections.
+- **Trade-offs**: Slower indexing due to LLM calls for each chunk, but significantly better retrieval accuracy.
+- **Cost**: Additional LLM API calls during indexing.
+
+#### 2. **USE_HYBRID_SEARCH**
+Combines traditional keyword search with semantic vector search to provide more comprehensive results. The system performs both searches in parallel and intelligently merges results, prioritizing documents that appear in both result sets.
+
+- **When to use**: Enable this when users might search using specific technical terms, function names, or when exact keyword matches are important alongside semantic understanding.
+- **Trade-offs**: Slightly slower search queries but more robust results, especially for technical content.
+- **Cost**: No additional API costs, just computational overhead.
+
+#### 3. **USE_AGENTIC_RAG**
+Enables specialized code example extraction and storage. When crawling documentation, the system identifies code blocks (≥300 characters), extracts them with surrounding context, generates summaries, and stores them in a separate vector database table specifically designed for code search.
+
+- **When to use**: Essential for AI coding assistants that need to find specific code examples, implementation patterns, or usage examples from documentation.
+- **Trade-offs**: Significantly slower crawling due to code extraction and summarization, requires more storage space.
+- **Cost**: Additional LLM API calls for summarizing each code example.
+- **Benefits**: Provides a dedicated `search_code_examples` tool that AI agents can use to find specific code implementations.
+
+#### 4. **USE_RERANKING**
+Applies cross-encoder reranking to search results after initial retrieval. Uses a lightweight cross-encoder model (`cross-encoder/ms-marco-MiniLM-L-6-v2`) to score each result against the original query, then reorders results by relevance.
+
+- **When to use**: Enable this when search precision is critical and you need the most relevant results at the top. Particularly useful for complex queries where semantic similarity alone might not capture query intent.
+- **Trade-offs**: Adds ~100-200ms to search queries depending on result count, but significantly improves result ordering.
+- **Cost**: No additional API costs - uses a local model that runs on CPU.
+- **Benefits**: Better result relevance, especially for complex queries. Works with both regular RAG search and code example search.
+
+### Recommended Configurations
+
+**For general documentation RAG:**
+```
+USE_CONTEXTUAL_EMBEDDINGS=false
+USE_HYBRID_SEARCH=true
+USE_AGENTIC_RAG=false
+USE_RERANKING=true
+```
+
+**For AI coding assistant with code examples:**
+```
+USE_CONTEXTUAL_EMBEDDINGS=true
+USE_HYBRID_SEARCH=true
+USE_AGENTIC_RAG=true
+USE_RERANKING=true
+```
+
+**For fast, basic RAG:**
+```
+USE_CONTEXTUAL_EMBEDDINGS=false
+USE_HYBRID_SEARCH=true
+USE_AGENTIC_RAG=false
+USE_RERANKING=false
+```
+
+## Running the Server
+
+### Using Docker
+
+```bash
+docker run --env-file .env -p 8051:8051 mcp/crawl4ai-rag
+```
+
+### Using Python
+
+```bash
+uv run src/crawl4ai_mcp.py
+```
+
+The server will start and listen on the configured host and port.
+
+## Integration with MCP Clients
+
+### SSE Configuration
+
+Once you have the server running with SSE transport, you can connect to it using this configuration:
+
+```json
+{
+  "mcpServers": {
+    "crawl4ai-rag": {
+      "transport": "sse",
+      "url": "http://localhost:8051/sse"
+    }
+  }
+}
+```
+
+> **Note for Windsurf users**: Use `serverUrl` instead of `url` in your configuration:
+> ```json
+> {
+>   "mcpServers": {
+>     "crawl4ai-rag": {
+>       "transport": "sse",
+>       "serverUrl": "http://localhost:8051/sse"
+>     }
+>   }
+> }
+> ```
+>
+> **Note for Docker users**: Use `host.docker.internal` instead of `localhost` if your client is running in a different container. This will apply if you are using this MCP server within n8n!
+
+### Stdio Configuration
+
+Add this server to your MCP configuration for Claude Desktop, Windsurf, or any other MCP client:
+
+```json
+{
+  "mcpServers": {
+    "crawl4ai-rag": {
+      "command": "python",
+      "args": ["path/to/crawl4ai-mcp/src/crawl4ai_mcp.py"],
+      "env": {
+        "TRANSPORT": "stdio",
+        "OPENAI_API_KEY": "your_openai_api_key",
+        "SUPABASE_URL": "your_supabase_url",
+        "SUPABASE_SERVICE_KEY": "your_supabase_service_key"
+      }
+    }
+  }
+}
+```
+
+### Docker with Stdio Configuration
+
+```json
+{
+  "mcpServers": {
+    "crawl4ai-rag": {
+      "command": "docker",
+      "args": ["run", "--rm", "-i", 
+               "-e", "TRANSPORT", 
+               "-e", "OPENAI_API_KEY", 
+               "-e", "SUPABASE_URL", 
+               "-e", "SUPABASE_SERVICE_KEY", 
+               "mcp/crawl4ai"],
+      "env": {
+        "TRANSPORT": "stdio",
+        "OPENAI_API_KEY": "your_openai_api_key",
+        "SUPABASE_URL": "your_supabase_url",
+        "SUPABASE_SERVICE_KEY": "your_supabase_service_key"
+      }
+    }
+  }
+}
+```
+
+## Building Your Own Server
+
+This implementation provides a foundation for building more complex MCP servers with web crawling capabilities. To build your own:
+
+1. Add your own tools by creating methods with the `@mcp.tool()` decorator
+2. Create your own lifespan function to add your own dependencies
+3. Modify the `utils.py` file for any helper functions you need
+4. Extend the crawling capabilities by adding more specialized crawlers

Relevant README 2.md

@@ -0,0 +1,618 @@
+# Context7 MCP - Up-to-date Code Docs For Any Prompt
+
+[![Website](https://img.shields.io/badge/Website-context7.com-blue)](https://context7.com) [![smithery badge](https://smithery.ai/badge/@upstash/context7-mcp)](https://smithery.ai/server/@upstash/context7-mcp) [<img alt="Install in VS Code (npx)" src="https://img.shields.io/badge/VS_Code-VS_Code?style=flat-square&label=Install%20Context7%20MCP&color=0098FF">](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%7B%22name%22%3A%22context7%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40upstash%2Fcontext7-mcp%40latest%22%5D%7D)
+
+[![繁體中文](https://img.shields.io/badge/docs-繁體中文-yellow)](./docs/README.zh-TW.md) [![簡體中文](https://img.shields.io/badge/docs-簡體中文-yellow)](./docs/README.zh-CN.md) [![한국어 문서](https://img.shields.io/badge/docs-한국어-green)](./docs/README.ko.md) [![Documentación en Español](https://img.shields.io/badge/docs-Español-orange)](./docs/README.es.md) [![Documentation en Français](https://img.shields.io/badge/docs-Français-blue)](./docs/README.fr.md) [![Documentação em Português (Brasil)](<https://img.shields.io/badge/docs-Português%20(Brasil)-purple>)](./docs/README.pt-BR.md) [![Documentazione in italiano](https://img.shields.io/badge/docs-Italian-red)](./docs/README.it.md) [![Dokumentasi Bahasa Indonesia](https://img.shields.io/badge/docs-Bahasa%20Indonesia-pink)](./docs/README.id-ID.md) [![Dokumentation auf Deutsch](https://img.shields.io/badge/docs-Deutsch-darkgreen)](./docs/README.de.md) [![Документация на русском языке](https://img.shields.io/badge/docs-Русский-darkblue)](./docs/README.ru.md) [![Türkçe Doküman](https://img.shields.io/badge/docs-Türkçe-blue)](./docs/README.tr.md) [![Arabic Documentation](https://img.shields.io/badge/docs-Arabic-white)](./docs/README.ar.md)
+
+## ❌ Without Context7
+
+LLMs rely on outdated or generic information about the libraries you use. You get:
+
+- ❌ Code examples are outdated and based on year-old training data
+- ❌ Hallucinated APIs don't even exist
+- ❌ Generic answers for old package versions
+
+## ✅ With Context7
+
+Context7 MCP pulls up-to-date, version-specific documentation and code examples straight from the source — and places them directly into your prompt.
+
+Add `use context7` to your prompt in Cursor:
+
+```txt
+Create a basic Next.js project with app router. use context7
+```
+
+```txt
+Create a script to delete the rows where the city is "" given PostgreSQL credentials. use context7
+```
+
+Context7 fetches up-to-date code examples and documentation right into your LLM's context.
+
+- 1️⃣ Write your prompt naturally
+- 2️⃣ Tell the LLM to `use context7`
+- 3️⃣ Get working code answers
+
+No tab-switching, no hallucinated APIs that don't exist, no outdated code generations.
+
+## 📚 Adding Projects
+
+Check out our [project addition guide](./docs/adding-projects.md) to learn how to add (or update) your favorite libraries to Context7.
+
+## 🛠️ Installation
+
+### Requirements
+
+- Node.js >= v18.0.0
+- Cursor, Windsurf, Claude Desktop or another MCP Client
+
+<details>
+<summary><b>Installing via Smithery</b></summary>
+
+To install Context7 MCP Server for any client automatically via [Smithery](https://smithery.ai/server/@upstash/context7-mcp):
+
+```bash
+npx -y @smithery/cli@latest install @upstash/context7-mcp --client <CLIENT_NAME> --key <YOUR_SMITHERY_KEY>
+```
+
+You can find your Smithery key in the [Smithery.ai webpage](https://smithery.ai/server/@upstash/context7-mcp).
+
+</details>
+
+<details>
+<summary><b>Install in Cursor</b></summary>
+
+Go to: `Settings` -> `Cursor Settings` -> `MCP` -> `Add new global MCP server`
+
+Pasting the following configuration into your Cursor `~/.cursor/mcp.json` file is the recommended approach. You may also install in a specific project by creating `.cursor/mcp.json` in your project folder. See [Cursor MCP docs](https://docs.cursor.com/context/model-context-protocol) for more info.
+
+> Since Cursor 1.0, you can click the install button below for instant one-click installation.
+
+#### Cursor Remote Server Connection
+
+[![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/install-mcp?name=context7&config=eyJ1cmwiOiJodHRwczovL21jcC5jb250ZXh0Ny5jb20vbWNwIn0%3D)
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "url": "https://mcp.context7.com/mcp"
+    }
+  }
+}
+```
+
+#### Cursor Local Server Connection
+
+[![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/install-mcp?name=context7&config=eyJjb21tYW5kIjoibnB4IC15IEB1cHN0YXNoL2NvbnRleHQ3LW1jcCJ9)
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+<details>
+<summary>Alternative: Use Bun</summary>
+
+[![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/install-mcp?name=context7&config=eyJjb21tYW5kIjoiYnVueCAteSBAdXBzdGFzaC9jb250ZXh0Ny1tY3AifQ%3D%3D)
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "bunx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary>Alternative: Use Deno</summary>
+
+[![Install MCP Server](https://cursor.com/deeplink/mcp-install-dark.svg)](https://cursor.com/install-mcp?name=context7&config=eyJjb21tYW5kIjoiZGVubyBydW4gLS1hbGxvdy1lbnYgLS1hbGxvdy1uZXQgbnBtOkB1cHN0YXNoL2NvbnRleHQ3LW1jcCJ9)
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "deno",
+      "args": ["run", "--allow-env", "--allow-net", "npm:@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+</details>
+
+<details>
+<summary><b>Install in Windsurf</b></summary>
+
+Add this to your Windsurf MCP config file. See [Windsurf MCP docs](https://docs.windsurf.com/windsurf/mcp) for more info.
+
+#### Windsurf Remote Server Connection
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "serverUrl": "https://mcp.context7.com/sse"
+    }
+  }
+}
+```
+
+#### Windsurf Local Server Connection
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Install in VS Code</b></summary>
+
+[<img alt="Install in VS Code (npx)" src="https://img.shields.io/badge/VS_Code-VS_Code?style=flat-square&label=Install%20Context7%20MCP&color=0098FF">](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%7B%22name%22%3A%22context7%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40upstash%2Fcontext7-mcp%40latest%22%5D%7D)
+[<img alt="Install in VS Code Insiders (npx)" src="https://img.shields.io/badge/VS_Code_Insiders-VS_Code_Insiders?style=flat-square&label=Install%20Context7%20MCP&color=24bfa5">](https://insiders.vscode.dev/redirect?url=vscode-insiders%3Amcp%2Finstall%3F%7B%22name%22%3A%22context7%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40upstash%2Fcontext7-mcp%40latest%22%5D%7D)
+
+Add this to your VS Code MCP config file. See [VS Code MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers) for more info.
+
+#### VS Code Remote Server Connection
+
+```json
+"mcp": {
+  "servers": {
+    "context7": {
+      "type": "http",
+      "url": "https://mcp.context7.com/mcp"
+    }
+  }
+}
+```
+
+#### VS Code Local Server Connection
+
+```json
+"mcp": {
+  "servers": {
+    "context7": {
+      "type": "stdio",
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Install in Zed</b></summary>
+
+It can be installed via [Zed Extensions](https://zed.dev/extensions?query=Context7) or you can add this to your Zed `settings.json`. See [Zed Context Server docs](https://zed.dev/docs/assistant/context-servers) for more info.
+
+```json
+{
+  "context_servers": {
+    "Context7": {
+      "command": {
+        "path": "npx",
+        "args": ["-y", "@upstash/context7-mcp"]
+      },
+      "settings": {}
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Install in Claude Code</b></summary>
+
+Run this command. See [Claude Code MCP docs](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/tutorials#set-up-model-context-protocol-mcp) for more info.
+
+#### Claude Code Remote Server Connection
+
+```sh
+claude mcp add --transport sse context7 https://mcp.context7.com/sse
+```
+
+#### Claude Code Local Server Connection
+
+```sh
+claude mcp add context7 -- npx -y @upstash/context7-mcp
+```
+
+</details>
+
+<details>
+<summary><b>Install in Claude Desktop</b></summary>
+
+Add this to your Claude Desktop `claude_desktop_config.json` file. See [Claude Desktop MCP docs](https://modelcontextprotocol.io/quickstart/user) for more info.
+
+```json
+{
+  "mcpServers": {
+    "Context7": {
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Install in BoltAI</b></summary>
+
+Open the "Settings" page of the app, navigate to "Plugins," and enter the following JSON:
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+Once saved, enter in the chat `get-library-docs` followed by your Context7 documentation ID (e.g., `get-library-docs /nuxt/ui`). More information is available on [BoltAI's Documentation site](https://docs.boltai.com/docs/plugins/mcp-servers). For BoltAI on iOS, [see this guide](https://docs.boltai.com/docs/boltai-mobile/mcp-servers).
+
+</details>
+
+<details>
+<summary><b>Using Docker</b></summary>
+
+If you prefer to run the MCP server in a Docker container:
+
+1. **Build the Docker Image:**
+
+   First, create a `Dockerfile` in the project root (or anywhere you prefer):
+
+   <details>
+   <summary>Click to see Dockerfile content</summary>
+
+   ```Dockerfile
+   FROM node:18-alpine
+
+   WORKDIR /app
+
+   # Install the latest version globally
+   RUN npm install -g @upstash/context7-mcp
+
+   # Expose default port if needed (optional, depends on MCP client interaction)
+   # EXPOSE 3000
+
+   # Default command to run the server
+   CMD ["context7-mcp"]
+   ```
+
+   </details>
+
+   Then, build the image using a tag (e.g., `context7-mcp`). **Make sure Docker Desktop (or the Docker daemon) is running.** Run the following command in the same directory where you saved the `Dockerfile`:
+
+   ```bash
+   docker build -t context7-mcp .
+   ```
+
+2. **Configure Your MCP Client:**
+
+   Update your MCP client's configuration to use the Docker command.
+
+   _Example for a cline_mcp_settings.json:_
+
+   ```json
+   {
+     "mcpServers": {
+       "Сontext7": {
+         "autoApprove": [],
+         "disabled": false,
+         "timeout": 60,
+         "command": "docker",
+         "args": ["run", "-i", "--rm", "context7-mcp"],
+         "transportType": "stdio"
+       }
+     }
+   }
+   ```
+
+   _Note: This is an example configuration. Please refer to the specific examples for your MCP client (like Cursor, VS Code, etc.) earlier in this README to adapt the structure (e.g., `mcpServers` vs `servers`). Also, ensure the image name in `args` matches the tag used during the `docker build` command._
+
+</details>
+
+<details>
+<summary><b>Install in Windows</b></summary>
+
+The configuration on Windows is slightly different compared to Linux or macOS (_`Cline` is used in the example_). The same principle applies to other editors; refer to the configuration of `command` and `args`.
+
+```json
+{
+  "mcpServers": {
+    "github.com/upstash/context7-mcp": {
+      "command": "cmd",
+      "args": ["/c", "npx", "-y", "@upstash/context7-mcp@latest"],
+      "disabled": false,
+      "autoApprove": []
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Install in Augment Code</b></summary>
+
+To configure Context7 MCP in Augment Code, follow these steps:
+
+1. Press Cmd/Ctrl Shift P or go to the hamburger menu in the Augment panel
+2. Select Edit Settings
+3. Under Advanced, click Edit in settings.json
+4. Add the server configuration to the `mcpServers` array in the `augment.advanced` object
+
+```json
+"augment.advanced": {
+    "mcpServers": [
+        {
+            "name": "context7",
+            "command": "npx",
+            "args": ["-y", "@upstash/context7-mcp"]
+        }
+    ]
+}
+```
+
+Once the MCP server is added, restart your editor. If you receive any errors, check the syntax to make sure closing brackets or commas are not missing.
+
+</details>
+
+<details>
+<summary><b>Install in Roo Code</b></summary>
+
+Add this to your Roo Code MCP configuration file. See [Roo Code MCP docs](https://docs.roocode.com/features/mcp/using-mcp-in-roo) for more info.
+
+#### Roo Code Remote Server Connection
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "type": "streamable-http",
+      "url": "https://mcp.context7.com/mcp"
+    }
+  }
+}
+```
+
+#### Roo Code Local Server Connection
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Install in Zencoder</b></summary>
+
+To configure Context7 MCP in Zencoder, follow these steps:
+
+1. Go to the Zencoder menu (...)
+2. From the dropdown menu, select Agent tools
+3. Click on the Add custom MCP
+4. Add the name and server configuration from below, and make sure to hit the Install button
+
+```json
+{
+    "command": "npx",
+    "args": [
+        "-y",
+        "@upstash/context7-mcp@latest"
+    ]
+}
+```
+
+Once the MCP server is added, you can easily continue using it.
+
+</details>
+
+## 🔧 Environment Variables
+
+The Context7 MCP server supports the following environment variables:
+
+- `DEFAULT_MINIMUM_TOKENS`: Set the minimum token count for documentation retrieval (default: 10000)
+
+Example configuration with environment variables:
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "@upstash/context7-mcp"],
+      "env": {
+        "DEFAULT_MINIMUM_TOKENS": "6000"
+      }
+    }
+  }
+}
+```
+
+## 🔨 Available Tools
+
+Context7 MCP provides the following tools that LLMs can use:
+
+- `resolve-library-id`: Resolves a general library name into a Context7-compatible library ID.
+
+  - `libraryName` (required): The name of the library to search for
+
+- `get-library-docs`: Fetches documentation for a library using a Context7-compatible library ID.
+  - `context7CompatibleLibraryID` (required): Exact Context7-compatible library ID (e.g., `/mongodb/docs`, `/vercel/next.js`)
+  - `topic` (optional): Focus the docs on a specific topic (e.g., "routing", "hooks")
+  - `tokens` (optional, default 10000): Max number of tokens to return. Values less than the configured `DEFAULT_MINIMUM_TOKENS` value or the default value of 10000 are automatically increased to that value.
+
+## 💻 Development
+
+Clone the project and install dependencies:
+
+```bash
+bun i
+```
+
+Build:
+
+```bash
+bun run build
+```
+
+<details>
+<summary><b>Local Configuration Example</b></summary>
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["tsx", "/path/to/folder/context7-mcp/src/index.ts"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>Testing with MCP Inspector</b></summary>
+
+```bash
+npx -y @modelcontextprotocol/inspector npx @upstash/context7-mcp
+```
+
+</details>
+
+## 🚨 Troubleshooting
+
+<details>
+<summary><b>Module Not Found Errors</b></summary>
+
+If you encounter `ERR_MODULE_NOT_FOUND`, try using `bunx` instead of `npx`:
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "bunx",
+      "args": ["-y", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+This often resolves module resolution issues in environments where `npx` doesn't properly install or resolve packages.
+
+</details>
+
+<details>
+<summary><b>ESM Resolution Issues</b></summary>
+
+For errors like `Error: Cannot find module 'uriTemplate.js'`, try the `--experimental-vm-modules` flag:
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "--node-options=--experimental-vm-modules", "@upstash/[email protected]"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>TLS/Certificate Issues</b></summary>
+
+Use the `--experimental-fetch` flag to bypass TLS-related problems:
+
+```json
+{
+  "mcpServers": {
+    "context7": {
+      "command": "npx",
+      "args": ["-y", "--node-options=--experimental-fetch", "@upstash/context7-mcp"]
+    }
+  }
+}
+```
+
+</details>
+
+<details>
+<summary><b>General MCP Client Errors</b></summary>
+
+1. Try adding `@latest` to the package name
+2. Use `bunx` as an alternative to `npx`
+3. Consider using `deno` as another alternative
+4. Ensure you're using Node.js v18 or higher for native fetch support
+
+</details>
+
+## ⚠️ Disclaimer
+
+Context7 projects are community-contributed and while we strive to maintain high quality, we cannot guarantee the accuracy, completeness, or security of all library documentation. Projects listed in Context7 are developed and maintained by their respective owners, not by Context7. If you encounter any suspicious, inappropriate, or potentially harmful content, please use the "Report" button on the project page to notify us immediately. We take all reports seriously and will review flagged content promptly to maintain the integrity and safety of our platform. By using Context7, you acknowledge that you do so at your own discretion and risk.
+
+## 🤝 Connect with Us
+
+Stay updated and join our community:
+
+- 📢 Follow us on [X](https://x.com/contextai) for the latest news and updates
+- 🌐 Visit our [Website](https://context7.com)
+- 💬 Join our [Discord Community](https://upstash.com/discord)
+
+## 📺 Context7 In Media
+
+- [Better Stack: "Free Tool Makes Cursor 10x Smarter"](https://youtu.be/52FC3qObp9E)
+- [Cole Medin: "This is Hands Down the BEST MCP Server for AI Coding Assistants"](https://www.youtube.com/watch?v=G7gK8H6u7Rs)
+- [Income Stream Surfers: "Context7 + SequentialThinking MCPs: Is This AGI?"](https://www.youtube.com/watch?v=-ggvzyLpK6o)
+- [Julian Goldie SEO: "Context7: New MCP AI Agent Update"](https://www.youtube.com/watch?v=CTZm6fBYisc)
+- [JeredBlu: "Context 7 MCP: Get Documentation Instantly + VS Code Setup"](https://www.youtube.com/watch?v=-ls0D-rtET4)
+- [Income Stream Surfers: "Context7: The New MCP Server That Will CHANGE AI Coding"](https://www.youtube.com/watch?v=PS-2Azb-C3M)
+- [AICodeKing: "Context7 + Cline & RooCode: This MCP Server Makes CLINE 100X MORE EFFECTIVE!"](https://www.youtube.com/watch?v=qZfENAPMnyo)
+- [Sean Kochel: "5 MCP Servers For Vibe Coding Glory (Just Plug-In & Go)"](https://www.youtube.com/watch?v=LqTQi8qexJM)
+
+## ⭐ Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=upstash/context7&type=Date)](https://www.star-history.com/#upstash/context7&Date)
+
+## 📄 License
+
+MIT

app.py

@@ -2,80 +2,166 @@ import gradio as gr
 import os
 import json
 import subprocess
-import tempfile
 import dotenv
 import shutil
+import uuid
+
+from schemas import Response
+from openai import OpenAI
 from pathlib import Path
-from string import Template
 from pymilvus import MilvusClient, model
+from repo2txt import make_tree
+from utils import copy_search_results, create_documentation_string, choice_prompt
 
 _ = dotenv.load_dotenv()
 
 subprocess.run(["python3", "make_docs.py"])
 subprocess.run(["python3", "make_rag_db.py"])
 
-template = Template("""\
----
-File: $file_path
----
-                    
-$file_content""")
-
 client = MilvusClient("milvus.db")
 embedding_fn = model.dense.OpenAIEmbeddingFunction(
-    model_name='text-embedding-3-small', # Specify the model name
-    api_key=os.environ.get('OPENAI_API_KEY'), # Provide your OpenAI API key
-    dimensions=1536 # Set the embedding dimensionality
+    model_name="text-embedding-3-large",
+    api_key=os.environ.get("OPENAI_API_KEY"),
+    dimensions=3072,
 )
 
+oai_client = OpenAI()
+
 
 def list_huggingface_resources_names() -> list[str]:
     """List all the names of the libraries, services, and other resources available within the HuggingFace ecosystem.
-    
+
     Returns:
         A list of libraries, services, and other resources available within the HuggingFace ecosystem
     """
-    with open('repos_config.json', 'r') as f:
+    with open("repos_config.json", "r") as f:
         repos = json.load(f)
 
-    print([repo['title'] for repo in repos])
+    print([repo["title"] for repo in repos])
+
+    return [repo["title"] for repo in repos]
+
+
+def search_documents(query, resource_names=None, topk=50):
+    """Search for relevant documents in the Milvus database."""
+    query_vectors = embedding_fn.encode_queries([query])
+
+    search_params = {
+        "collection_name": "hf_docs",
+        "data": query_vectors,
+        "limit": topk,
+        "output_fields": ["text", "file_path", "resource"],
+    }
+
+    if resource_names:
+        if len(resource_names) == 1:
+            search_params["filter"] = f"resource == '{resource_names[0]}'"
+        else:
+            resource_list = "', '".join(resource_names)
+            search_params["filter"] = f"resource in ['{resource_list}']"
 
-    return [repo['title'] for repo in repos]
+    return client.search(**search_params)
 
 
 def get_huggingface_documentation(topic: str, resource_names: list[str] = []) -> str:
     """Get the documentation for the given topic and resource names.
-    
+
     Args:
         topic: Focus the docs on a specific topic (e.g. "Anthropic Provider Chat UI", "LoRA methods PEFT" or "TGI on Intel GPUs")
-        resource_names: A list of relevant resource names to the topic
-        
+        resource_names: A list of relevant resource names to the topic. Must be as specific as possible. Empty list means all resources.
+
     Returns:
         A string of documentation for the given topic and resource names
     """
-    print(resource_names)
-    query_vectors = embedding_fn.encode_queries([topic])
-    res = client.search(collection_name="hf_docs", data=query_vectors, limit=3, output_fields=["text", "file_path"])
-    print(res)
+    try:
+        # Search for relevant documents
+        query_vectors = embedding_fn.encode_queries([topic])
+
+        search_params = {
+            "collection_name": "hf_docs",
+            "data": query_vectors,
+            "limit": 50,
+            "output_fields": ["text", "file_path", "resource"],
+        }
+
+        if resource_names:
+            if len(resource_names) == 1:
+                search_params["filter"] = f"resource == '{resource_names[0]}'"
+            else:
+                resource_list = "', '".join(resource_names)
+                search_params["filter"] = f"resource in ['{resource_list}']"
+
+        search_results = client.search(**search_params)
+
+        # Create temporary folder and copy files
+        temp_folder = str(uuid.uuid4())
+        copy_search_results(search_results, temp_folder)
+
+        # Generate directory tree
+        tree_structure = make_tree(Path(temp_folder) / "docs")
+
+        # Get relevant file IDs using GPT-4
+        response = oai_client.responses.parse(
+            model="gpt-4o",
+            input=[
+                {
+                    "role": "user",
+                    "content": choice_prompt.substitute(
+                        question=topic, tree_structure=tree_structure
+                    ),
+                }
+            ],
+            text_format=Response,
+        )
+
+        file_ids = response.output_parsed.file_ids
+
+        # Create the documentation string using the file IDs and template
+        documentation_string = create_documentation_string(file_ids, temp_folder)
+
+        # Clean up temporary folder
+        shutil.rmtree(temp_folder, ignore_errors=True)
+
+        return documentation_string
+
+    except Exception as e:
+        return f"Error generating documentation: {str(e)}"
+
+
+def load_readme() -> str:
+    """Load and return the README content, skipping YAML frontmatter."""
+    try:
+        with open("README.md", "r", encoding="utf-8") as f:
+            content = f.read()
 
-    docs_paths = [res[0][i]['file_path'] for i in range(len(res[0]))]
-    print(docs_paths)
+        # Skip YAML frontmatter if it exists
+        if content.startswith("---"):
+            # Find the second '---' line
+            lines = content.split("\n")
+            start_index = 0
+            dash_count = 0
 
-    documentation = ""
-    for path in docs_paths:
-        with open(path, 'r') as f:
-            content = f.read()
-            documentation += template.substitute(file_path=path.replace('docs/', ''), file_content=content) + "\n\n"
+            for i, line in enumerate(lines):
+                if line.strip() == "---":
+                    dash_count += 1
+                    if dash_count == 2:
+                        start_index = i + 1
+                        break
+
+            # Join the lines after the frontmatter
+            content = "\n".join(lines[start_index:])
+
+        return content
+    except FileNotFoundError:
+        return "README.md not found"
 
-    print(documentation.strip())
-    return documentation.strip()
 
 list_resources_demo = gr.Interface(
     fn=list_huggingface_resources_names,
     inputs=[],
     outputs="json",
     title="HuggingFace Ecosystem Explorer",
-    description="Explore the names of the libraries, services, and other resources available within the HuggingFace ecosystem"
+    description="Explore the names of the libraries, services, and other resources available within the HuggingFace ecosystem",
 )
 
 get_docs_demo = gr.Interface(
@@ -84,11 +170,15 @@ get_docs_demo = gr.Interface(
     outputs="text",
 )
 
+# Create README tab with Markdown component
+with gr.Blocks() as readme_tab:
+    gr.Markdown(load_readme())
+
 # Create tabbed interface
 demo = gr.TabbedInterface(
-    [list_resources_demo, get_docs_demo],
-    ["List Resources", "Get Documentation"],
-    title="HuggingFace Ecosystem Documentation Explorer",
-)   
+    [readme_tab, list_resources_demo, get_docs_demo],
+    ["Quickstart", "List Resources", "Get Documentation"],
+    title="OpenHFContext7 MCP - Up-to-date Code Docs For Any Prompt",
+)
 
-demo.launch(mcp_server=True)
+demo.launch(mcp_server=True)

make_docs.py

@@ -51,11 +51,7 @@ def clone_repo(repo_url: str, dir_to_clone: str, target_dir: str) -> bool:
     sparse_init = run_command(["git", "sparse-checkout", "init", "--no-cone"], cwd=target_dir)
     if not sparse_init: return False
     
-    # Set sparse checkout patterns to only include the specified directory. Pattern explanation:
-    # '/*' - include all files at root level
-    # '!/*' - exclude all files at root level (overrides previous)
-    # f'/{dir_to_clone}/' - include the specific directory
-    # f'/{dir_to_clone}/**' - include everything under that directory
+    # Set sparse checkout patterns to only include the specified directory
     sparse_patterns = ['/*', '!/*', f'/{dir_to_clone}/', f'/{dir_to_clone}/**']
     sparse_set = run_command(["git", "sparse-checkout", "set", "--no-cone"] + sparse_patterns, cwd=target_dir)
     if not sparse_set: return False
@@ -72,17 +68,25 @@ def clone_repo(repo_url: str, dir_to_clone: str, target_dir: str) -> bool:
     return True
 
 
-def save_section_to_disk(section: Dict, file_path: Path, raw_docs_path: Path):
-
-    title = section["title"]
+def save_section_to_disk(section: Dict, file_path: Path, raw_docs_path: Path, prefix: str, index: int):
+    """
+    Recursively saves a documentation section to disk with hierarchical numbering.
+    """
+    current_number = f"{prefix}{index}"
+    numbered_title = f"{current_number}. {section['title']}"
 
     if "sections" in section:
-        file_path = file_path / title
-        os.makedirs(file_path, exist_ok=True)
-        for subsection in section["sections"]:
-            save_section_to_disk(subsection, file_path, raw_docs_path)
+        # This is a directory
+        new_dir_path = file_path / numbered_title
+        os.makedirs(new_dir_path, exist_ok=True)
+        
+        # The new prefix for children adds the current number, e.g., "1.1."
+        new_prefix = f"{current_number}."
+        for i, subsection in enumerate(section["sections"], 1):
+            save_section_to_disk(subsection, new_dir_path, raw_docs_path, new_prefix, i)
             
     else:
+        # This is a file
         try:
             local_path = raw_docs_path / f"{section['local']}.md"
 
@@ -90,7 +94,9 @@ def save_section_to_disk(section: Dict, file_path: Path, raw_docs_path: Path):
                 local_path = raw_docs_path / f"{section['local']}.mdx"
             assert local_path.exists(), f"File {local_path} does not exist"
 
-            shutil.copy(local_path, file_path / f"{title}{local_path.suffix}")
+            # Create the numbered filename
+            new_filename = f"{numbered_title}{local_path.suffix}"
+            shutil.copy(local_path, file_path / new_filename)
         
         except Exception as e:
             # TODO: Not many cases, but handle symlinks, missing files, and other edge cases
@@ -99,20 +105,23 @@ def save_section_to_disk(section: Dict, file_path: Path, raw_docs_path: Path):
 
 def make_docs(repos: Dict, args: Dict):
 
-    for repo in tqdm(repos, desc="Consolidating 🤗 Documentation"):
+    for repo_index, repo in enumerate(tqdm(repos, desc="Consolidating 🤗 Documentation"), 1):
         save_repo_docs_path = Path(f"{args.repos_dir}/{repo['repo_url'].split('/')[-1]}")
         clone_repo(repo["repo_url"], repo["subfolder"], str(save_repo_docs_path))
         
         repo_docs_path = save_repo_docs_path / repo["subfolder"]
         toctree = parse_toctree_yaml(repo_docs_path / "_toctree.yml")
 
-        # print(toctree)
+        # Create the top-level numbered directory for the repo, e.g., "1. Accelerate"
+        repo_title = f"{repo_index}. {repo['title']}"
+        repo_output_path = Path(args.docs_dir) / repo_title
+        os.makedirs(repo_output_path, exist_ok=True)
 
-        save_doc_path = Path(f"{args.docs_dir}/{repo['title']}")
-        os.makedirs(save_doc_path, exist_ok=True)
-
-        for block in toctree:
-            save_section_to_disk(block, save_doc_path, repo_docs_path)
+        # The initial prefix for numbering is the repo index, e.g., "1."
+        prefix = f"{repo_index}."
+        for block_index, block in enumerate(toctree, 1):
+            # Start the recursive saving with the initial prefix and the block's index
+            save_section_to_disk(block, repo_output_path, repo_docs_path, prefix, block_index)
 
         shutil.rmtree(save_repo_docs_path)
 
@@ -128,5 +137,7 @@ if __name__ == "__main__":
     with open("repos_config.json", "r") as f:
         repos = json.load(f)
 
-    # shutil.rmtree(args.docs_dir)
-    make_docs(repos, args)
+    if os.path.exists(args.docs_dir):
+        shutil.rmtree(args.docs_dir)
+
+    make_docs(repos, args)

make_rag_db.py

@@ -1,5 +1,7 @@
 import os
 import argparse
+import json
+import re
 from typing import Dict
 import dotenv
 from pathlib import Path
@@ -18,6 +20,14 @@ def create_collection(client: MilvusClient, collection_name: str, dimension: int
         dimension=dimension,
     )
 
+
+def clean_filename(s):
+    s = re.sub(r'\d+(?:\.\d+)*\.\s*', '', s) # Remove hierarchical numbering (e.g., "28.", "28.1.")
+    s = re.sub(r'[^\w\s/.-]', '', s)         # Remove emojis
+    s = re.sub(r'\s+', ' ', s)               # Clean up extra spaces
+    return s.strip()
+
+
 def main(args: Dict):
     client = MilvusClient("milvus.db")
 
@@ -29,17 +39,25 @@ def main(args: Dict):
 
     create_collection(client, args.collection_name, args.dimension)
 
-    docs = Path(args.docs_dir)
-    md_file_paths  = list(docs.rglob('*.md'))
-    mdx_file_paths = list(docs.rglob('*.mdx'))
-    all_file_paths = md_file_paths + mdx_file_paths
-    
+    with open(args.repos_config_path, "r") as f:
+        repos = json.load(f)
+
     docs, payloads = [], []
-    for file in tqdm(all_file_paths):
-        embed_string = str(file).replace('docs/', '').replace('.mdx', '').replace('.md', '').replace('/', ' ')
+    for i, repo in enumerate(repos, 1):
 
-        docs.append(embed_string)
-        payloads.append({'file_path': str(file)})
+        docs_path = Path('docs') / f"{i}. {repo['title']}"
+        md_file_paths  = list(docs_path.rglob('*.md'))
+        mdx_file_paths = list(docs_path.rglob('*.mdx'))
+        all_file_paths = md_file_paths + mdx_file_paths
+
+        # print(all_file_paths[:5])
+
+        for file in all_file_paths:
+            embed_string = str(file).replace('docs/', '').replace('.mdx', '').replace('.md', '').replace('/', ' ')
+            embed_string = clean_filename(embed_string)
+
+            docs.append(embed_string)
+            payloads.append({'file_path': str(file), 'resource': repo['title']})
 
     vectors = embedding_fn.encode_documents(docs)
 
@@ -54,9 +72,14 @@ def main(args: Dict):
 if __name__ == "__main__":
     parser = argparse.ArgumentParser()
     parser.add_argument("--collection_name", type=str, default="hf_docs")
-    parser.add_argument("--model_name", type=str, default="text-embedding-3-small")
-    parser.add_argument("--dimension", type=int, default=1536)
+    parser.add_argument("--model_name", type=str, default="text-embedding-3-large")
+    parser.add_argument("--dimension", type=int, default=3072)
     parser.add_argument("--docs_dir", type=str, default="docs")
+    parser.add_argument("--repos_config_path", type=str, default="repos_config.json")
     args = parser.parse_args()
 
+    if Path('milvus.db').exists():
+        print("Removing existing Milvus database...")
+        os.remove('milvus.db')
+
     main(args)

postBuild

@@ -1,2 +0,0 @@
-python3 make_docs.py
-python3 make_rag_db.py

repo2txt.py

@@ -5,47 +5,21 @@ This version only includes the functionality to document the structure of a repo
 """
 
 import os
-import argparse
 
-def parse_args():
-    """
-    Parse command-line arguments for the script.
 
-    Returns:
-        argparse.Namespace: An object containing the parsed command-line arguments.
-    """
-    parser = argparse.ArgumentParser(
-        description='Document the structure of a repository containing .md and .mdx files.',
-        epilog='Example usage:\n  python repo2txt.py -r /path/to/repo -o output.txt',
-        formatter_class=argparse.RawDescriptionHelpFormatter
-    )
-
-    parser.add_argument('-r', '--repo_path', default=os.getcwd(),
-                        help='Path to the directory to process. Defaults to the current directory.')
-    parser.add_argument('-o', '--output_file', default='output.txt',
-                        help='Name for the output text file. Defaults to "output.txt".')
-
-    return parser.parse_args()
-
-
-def should_ignore(item_path, output_file_path):
+def should_ignore(item_path):
     """
     Determine if a given item should be ignored.
     Only includes .md and .mdx files, ignores hidden files and directories.
 
     Args:
         item_path (str): The path of the item (file or directory) to check.
-        output_file_path (str): The path of the output file being written to.
 
     Returns:
         bool: True if the item should be ignored, False otherwise.
     """
     item_name = os.path.basename(item_path)
     
-    # Ignore the output file itself
-    if os.path.abspath(item_path) == os.path.abspath(output_file_path):
-        return True
-
     # Ignore hidden files and directories
     if item_name.startswith('.'):
         return True
@@ -58,24 +32,26 @@ def should_ignore(item_path, output_file_path):
     # Include directories (they will be traversed)
     return False
 
-
-def write_tree(dir_path, output_file, output_file_path, prefix="", is_root=True):
+def make_tree(dir_path, prefix="", is_root=True):
     """
-    Recursively write the directory tree to the output file.
+    Recursively generate the directory tree as a string.
 
     Args:
         dir_path (str): The path of the directory to document.
-        output_file (file object): The file object to write to.
-        output_file_path (str): The path of the output file being written to.
         prefix (str): Prefix string for line indentation and structure.
         is_root (bool): Flag to indicate if the current directory is the root.
+    
+    Returns:
+        str: The tree structure as a string.
     """
+    tree_string = ""
+    
     if is_root:
-        output_file.write("└── ./\n")
+        tree_string += "└── ./\n"
         # Add the actual directory name as a child of ./
         actual_dir_name = os.path.basename(dir_path)
         if actual_dir_name:
-            output_file.write(f"    └── {actual_dir_name}\n")
+            tree_string += f"    └── {actual_dir_name}\n"
             prefix = "        "
         else:
             prefix = "    "
@@ -84,7 +60,7 @@ def write_tree(dir_path, output_file, output_file_path, prefix="", is_root=True)
     try:
         items = os.listdir(dir_path)
     except PermissionError:
-        return
+        return tree_string
     
     items.sort()
     
@@ -92,7 +68,7 @@ def write_tree(dir_path, output_file, output_file_path, prefix="", is_root=True)
     filtered_items = []
     for item in items:
         item_path = os.path.join(dir_path, item)
-        if not should_ignore(item_path, output_file_path):
+        if not should_ignore(item_path):
             filtered_items.append(item)
     
     num_items = len(filtered_items)
@@ -103,75 +79,10 @@ def write_tree(dir_path, output_file, output_file_path, prefix="", is_root=True)
         new_prefix = "└── " if is_last_item else "├── "
         child_prefix = "    " if is_last_item else "│   "
 
-        output_file.write(f"{prefix}{new_prefix}{item}\n")
+        tree_string += f"{prefix}{new_prefix}{item}\n"
 
         if os.path.isdir(item_path):
             next_prefix = prefix + child_prefix
-            write_tree(item_path, output_file, output_file_path, next_prefix, is_root=False)
-
-
-def write_file_content(file_path, output_file):
-    """
-    Write the contents of a given file to the output file.
-
-    Args:
-        file_path (str): Path of the file to read.
-        output_file (file object): The file object to write the contents to.
-    """
-    try:
-        with open(file_path, 'r', encoding='utf-8', errors='ignore') as file:
-            for line in file:
-                output_file.write(line)
-    except Exception as e:
-        output_file.write(f"Error reading file: {e}\n")
-
-
-def write_file_contents_in_order(dir_path, output_file, output_file_path, repo_path):
-    """
-    Recursively document the contents of .md and .mdx files in directory order.
-
-    Args:
-        dir_path (str): The path of the directory to start documenting from.
-        output_file (file object): The file object to write the contents to.
-        output_file_path (str): The path of the output file being written to.
-        repo_path (str): The root path of the repository for relative path calculation.
-    """
-    try:
-        items = os.listdir(dir_path)
-    except PermissionError:
-        return
+            tree_string += make_tree(item_path, next_prefix, is_root=False)
     
-    items = sorted(item for item in items if not should_ignore(os.path.join(dir_path, item), output_file_path))
-
-    for item in items:
-        item_path = os.path.join(dir_path, item)
-        relative_path = os.path.relpath(item_path, start=repo_path)
-
-        if os.path.isdir(item_path):
-            write_file_contents_in_order(item_path, output_file, output_file_path, repo_path)
-        elif os.path.isfile(item_path):
-            output_file.write(f"\n\n---\nFile: /{relative_path}\n---\n\n")
-            write_file_content(item_path, output_file)
-
-
-def main():
-    """
-    Main function to execute the script logic.
-    """
-    args = parse_args()
-
-    # Check if the provided directory path is valid
-    if not os.path.isdir(args.repo_path):
-        print(f"Error: The specified directory does not exist: {args.repo_path}")
-        return
-
-    with open(args.output_file, 'w', encoding='utf-8') as output_file:
-        output_file.write("Directory Structure:\n\n")
-        write_tree(args.repo_path, output_file, args.output_file, "", is_root=True)
-        write_file_contents_in_order(args.repo_path, output_file, args.output_file, args.repo_path)
-
-    print(f"Documentation generated successfully: {args.output_file}")
-
-
-if __name__ == "__main__":
-    main()
+    return tree_string

requirements.txt

@@ -1,5 +1,6 @@
-pymilvus==2.5.10
-pymilvus_model==0.3.2
-python-dotenv==1.1.0
-PyYAML==6.0.2
-tqdm==4.65.0
+pymilvus
+pymilvus_model
+python-dotenv
+PyYAML
+tqdm
+openai

schemas.py

@@ -0,0 +1,4 @@
+from pydantic import BaseModel
+
+class Response(BaseModel):
+    file_ids: list[str]

utils.py

@@ -0,0 +1,68 @@
+import shutil
+from pathlib import Path
+from string import Template
+
+
+doc_template = Template("""
+
+---
+File: $file_path
+---
+
+$file_content
+
+""".strip())
+
+
+choice_prompt = Template("""
+
+The user has asked the following question: $question
+
+The goal is get the user the 3 most relevant documentation files to answer the question.
+
+Here is the tree structure of the documentation. Your task is to return the numeric ids \
+associated with the 3 most relevant .md and .mdx files.
+
+<tree>
+$tree_structure
+</tree>
+
+Sample response: ["1.3.2", "11.4.12", "7.12.11"]
+Top 3 file ids:
+
+""".strip())
+
+
+def copy_search_results(search_results, dest_folder):
+    """Copy files from search results to destination folder."""
+    for item in search_results[0]:
+        file_path = item['entity']['file_path']
+        dest_path = Path(dest_folder) / file_path
+        
+        dest_path.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copy2(file_path, dest_path)
+
+
+def create_documentation_string(file_ids, temp_folder):
+    """Create documentation string from file IDs using the template."""
+    documentation_parts = []
+    
+    for file_id in file_ids:
+        # Find the corresponding file in the temp folder
+        docs_path = Path(temp_folder) / "docs"
+        for file_path in docs_path.rglob("*.md*"):
+            if file_id in str(file_path):
+                try:
+                    with open(file_path, 'r', encoding='utf-8') as f:
+                        content = f.read()
+                    
+                    formatted_doc = doc_template.substitute(
+                        file_path=str(file_path.relative_to(docs_path)),
+                        file_content=content
+                    )
+                    documentation_parts.append(formatted_doc)
+                    break
+                except Exception as e:
+                    print(f"Error reading file {file_path}: {e}")
+    
+    return "\n\n".join(documentation_parts)