Spaces:
Running
Running
Delete CLAUDE.md
Browse files
CLAUDE.md
DELETED
@@ -1,296 +0,0 @@
|
|
1 |
-
# CLAUDE.md
|
2 |
-
|
3 |
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
4 |
-
|
5 |
-
## Project Overview
|
6 |
-
|
7 |
-
Chat UI Helper is a Gradio-based tool for generating and configuring chat interfaces for HuggingFace Spaces. It creates deployable packages with custom assistants, web scraping capabilities, and optional vector RAG functionality.
|
8 |
-
|
9 |
-
## Core Architecture
|
10 |
-
|
11 |
-
### Main Application Flow (`app.py`)
|
12 |
-
The application follows a three-tab Gradio interface pattern:
|
13 |
-
1. **Configuration Tab**: Space setup, assistant configuration, tool settings
|
14 |
-
2. **Sandbox Preview Tab**: Interactive testing with real OpenRouter API integration
|
15 |
-
3. **Support Docs Tab**: Comprehensive guidance and templates via `support_docs.py`
|
16 |
-
|
17 |
-
### Template Generation System
|
18 |
-
- `SPACE_TEMPLATE` (lines 130-710): Complete HuggingFace Space template with export functionality
|
19 |
-
- `generate_zip()` function (lines 869-935): Orchestrates package creation with all dependencies
|
20 |
-
- Key template variables: `{system_prompt}`, `{model}`, `{enable_vector_rag}`, `{api_key_var}`, `{grounding_urls}`, `{enable_dynamic_urls}`, `{enable_web_search}`
|
21 |
-
|
22 |
-
### Preview Sandbox Architecture
|
23 |
-
- Real OpenRouter API integration in preview mode (`preview_chat_response()` line 1185)
|
24 |
-
- URL context testing with dynamic add/remove functionality
|
25 |
-
- Configuration-aware responses using exact model and parameters from user configuration
|
26 |
-
- Fallback messaging when `OPENROUTER_API_KEY` environment variable not set
|
27 |
-
- Legacy tuple format compatibility for Gradio 4.44.1 ChatInterface
|
28 |
-
- Comprehensive debugging with enhanced error handling and API response validation
|
29 |
-
|
30 |
-
### Document Processing Pipeline (RAG)
|
31 |
-
- **RAGTool** (`rag_tool.py`): Main orchestrator with 10MB file size validation
|
32 |
-
- **DocumentProcessor** (`document_processor.py`): PDF/DOCX/TXT/MD parsing with semantic chunking (800 chars, 100 overlap)
|
33 |
-
- **VectorStore** (`vector_store.py`): FAISS-based similarity search and base64 serialization
|
34 |
-
|
35 |
-
### Web Scraping Architecture
|
36 |
-
Simple HTTP + BeautifulSoup approach with crawl4ai integration:
|
37 |
-
- `enhanced_fetch_url_content()` (lines 79-128): Enhanced requests with timeout and user-agent headers
|
38 |
-
- Content cleaning: Removes scripts, styles, navigation elements
|
39 |
-
- Content limits: ~4000 character truncation for context management
|
40 |
-
- URL content caching: `get_cached_grounding_context()` (line 1441) prevents redundant fetches
|
41 |
-
- `extract_urls_from_text()` (line 51): Regex-based URL extraction for dynamic fetching
|
42 |
-
|
43 |
-
## Development Commands
|
44 |
-
|
45 |
-
### Environment Setup
|
46 |
-
**Important**: This application requires Python ≥3.10 for Gradio 5.x compatibility.
|
47 |
-
|
48 |
-
```bash
|
49 |
-
# Recommended: Use Python 3.11+ environment
|
50 |
-
python3.11 -m venv venv311
|
51 |
-
source venv311/bin/activate # or venv311\Scripts\activate on Windows
|
52 |
-
pip install -r requirements.txt
|
53 |
-
```
|
54 |
-
|
55 |
-
### Running the Application
|
56 |
-
```bash
|
57 |
-
# With virtual environment activated
|
58 |
-
python app.py
|
59 |
-
```
|
60 |
-
|
61 |
-
### Testing Commands
|
62 |
-
```bash
|
63 |
-
# Test vector database functionality (requires all RAG dependencies)
|
64 |
-
python test_vector_db.py
|
65 |
-
|
66 |
-
# Test RAG fixes and error handling
|
67 |
-
python test_rag_fix.py
|
68 |
-
|
69 |
-
# Test OpenRouter API key validation
|
70 |
-
python test_api_key.py
|
71 |
-
|
72 |
-
# Test minimal Gradio functionality (for debugging)
|
73 |
-
python test_minimal.py
|
74 |
-
|
75 |
-
# Test preview functionality components
|
76 |
-
python test_preview.py
|
77 |
-
|
78 |
-
# Test individual RAG components
|
79 |
-
python -c "from test_vector_db import test_document_processing; test_document_processing()"
|
80 |
-
python -c "from test_vector_db import test_vector_store; test_vector_store()"
|
81 |
-
python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
|
82 |
-
```
|
83 |
-
|
84 |
-
### Pre-Test Setup for RAG Components
|
85 |
-
```bash
|
86 |
-
# Create test document for vector database testing
|
87 |
-
echo "This is a test document for RAG functionality testing." > test_document.txt
|
88 |
-
|
89 |
-
# Verify all dependencies are installed
|
90 |
-
python -c "import sentence_transformers, faiss, fitz; print('RAG dependencies available')"
|
91 |
-
```
|
92 |
-
|
93 |
-
## Key Dependencies and Versions
|
94 |
-
|
95 |
-
### Required Dependencies
|
96 |
-
- **Gradio ≥4.44.1**: Main UI framework (5.37.0 recommended for Python ≥3.10)
|
97 |
-
- **requests ≥2.32.3**: HTTP requests for web content fetching
|
98 |
-
- **beautifulsoup4 ≥4.12.3**: HTML parsing for web scraping
|
99 |
-
- **python-dotenv ≥1.0.0**: Environment variable management
|
100 |
-
|
101 |
-
### Optional RAG Dependencies
|
102 |
-
- **sentence-transformers ≥2.2.2**: Text embeddings
|
103 |
-
- **faiss-cpu ==1.7.4**: Vector similarity search
|
104 |
-
- **PyMuPDF ≥1.23.0**: PDF text extraction
|
105 |
-
- **python-docx ≥0.8.11**: DOCX document processing
|
106 |
-
- **numpy ==1.26.4**: Numerical operations
|
107 |
-
|
108 |
-
### Optional Web Search Dependencies
|
109 |
-
- **crawl4ai ≥0.2.0**: Advanced web crawling for web search functionality
|
110 |
-
- **aiohttp ≥3.8.0**: Async HTTP client for crawl4ai
|
111 |
-
|
112 |
-
## Configuration Patterns
|
113 |
-
|
114 |
-
### Conditional Dependency Loading
|
115 |
-
```python
|
116 |
-
try:
|
117 |
-
from rag_tool import RAGTool
|
118 |
-
HAS_RAG = True
|
119 |
-
except ImportError:
|
120 |
-
HAS_RAG = False
|
121 |
-
RAGTool = None
|
122 |
-
```
|
123 |
-
This pattern allows graceful degradation when optional vector dependencies are unavailable.
|
124 |
-
|
125 |
-
### Template Variable Substitution
|
126 |
-
Generated spaces use these key substitutions:
|
127 |
-
- `{system_prompt}`: Combined assistant configuration
|
128 |
-
- `{grounding_urls}`: Static URL list for context
|
129 |
-
- `{enable_dynamic_urls}`: Runtime URL fetching capability
|
130 |
-
- `{enable_vector_rag}`: Document search integration
|
131 |
-
- `{enable_web_search}`: Web search integration via crawl4ai
|
132 |
-
- `{rag_data_json}`: Serialized embeddings and chunks
|
133 |
-
- `{api_key_var}`: Customizable API key environment variable name
|
134 |
-
|
135 |
-
### Access Control Pattern
|
136 |
-
- Environment variable `SPACE_ACCESS_CODE` for student access control
|
137 |
-
- Global state management for session-based access in generated spaces
|
138 |
-
- Security-first approach storing credentials as HuggingFace Spaces secrets
|
139 |
-
|
140 |
-
### RAG Integration Workflow
|
141 |
-
1. Documents uploaded through Gradio File component with conditional visibility (`HAS_RAG` flag)
|
142 |
-
2. Processed via DocumentProcessor (PDF/DOCX/TXT/MD support) in `process_documents()` function
|
143 |
-
3. Chunked and embedded using sentence-transformers (800 chars, 100 overlap)
|
144 |
-
4. FAISS index created and serialized to base64 for deployment portability
|
145 |
-
5. Embedded in generated template via `{rag_data_json}` template variable
|
146 |
-
|
147 |
-
## Implementation Notes
|
148 |
-
|
149 |
-
### Research Template System (Simplified)
|
150 |
-
- **Simple Toggle**: `toggle_research_assistant()` function (line 1704) provides simple on/off functionality
|
151 |
-
- **Direct System Prompt**: Enables predefined academic research prompt with DOI verification and LibKey integration
|
152 |
-
- **Auto-Enable Dynamic URLs**: Research template automatically enables dynamic URL fetching for academic sources
|
153 |
-
- **Template Content**: Academic inquiry focus with DOI-verified sources, fact-checking, and proper citation requirements
|
154 |
-
|
155 |
-
### State Management Across Tabs
|
156 |
-
- Extensive use of `gr.State()` for maintaining session data
|
157 |
-
- Cross-tab functionality through shared state variables (`sandbox_state`, `preview_config_state`)
|
158 |
-
- URL content caching to prevent redundant web requests (`url_content_cache` global variable)
|
159 |
-
- Preview debugging with comprehensive error handling and API response validation
|
160 |
-
|
161 |
-
### Gradio Compatibility and Message Format Handling
|
162 |
-
- **Target Version**: Gradio 5.37.0 (requires Python ≥3.10)
|
163 |
-
- **Legacy Support**: Gradio 4.44.1 compatibility with JSON schema workarounds
|
164 |
-
- **Message Format**: Preview uses legacy tuple format `[user_msg, bot_msg]` for ChatInterface compatibility
|
165 |
-
- **Generated Spaces**: Use modern dictionary format `{"role": "user", "content": "..."}` for OpenRouter API
|
166 |
-
|
167 |
-
### Security Considerations
|
168 |
-
- Never embed API keys or access codes in generated templates
|
169 |
-
- Environment variable pattern for all sensitive configuration (`{api_key_var}` template variable)
|
170 |
-
- Input validation for uploaded files and URL processing
|
171 |
-
- Content length limits for web scraping operations
|
172 |
-
|
173 |
-
## Tool Configuration Changes
|
174 |
-
|
175 |
-
### Code Execution Functionality Removed
|
176 |
-
**Important**: Code execution functionality has been completely removed from the application. Do not attempt to re-add it.
|
177 |
-
|
178 |
-
- All `enable_code_execution` parameters and checkboxes have been removed
|
179 |
-
- The `toggle_code_execution` function has been removed
|
180 |
-
- Code execution logic in preview and generation functions has been removed
|
181 |
-
- Generated spaces no longer support code execution capabilities
|
182 |
-
|
183 |
-
### Web Search Integration
|
184 |
-
- **Enable Web Search**: Checkbox to enable web search functionality using crawl4ai
|
185 |
-
- **Technology**: Uses crawl4ai library with DuckDuckGo for search results
|
186 |
-
- **Implementation**: Integrated in both preview mode and generated spaces
|
187 |
-
- **Fallback**: Simple HTTP requests if crawl4ai is not available
|
188 |
-
|
189 |
-
## Testing Infrastructure
|
190 |
-
|
191 |
-
### Current Test Structure
|
192 |
-
- `test_vector_db.py`: Comprehensive RAG component testing
|
193 |
-
- `test_api_key.py`: OpenRouter API validation
|
194 |
-
- `test_minimal.py`: Basic Gradio functionality debugging
|
195 |
-
- `test_preview.py`: Preview functionality component testing
|
196 |
-
|
197 |
-
### Test Dependencies
|
198 |
-
RAG testing requires: `sentence-transformers`, `faiss-cpu`, `PyMuPDF`, `python-docx`
|
199 |
-
Core testing requires: `gradio`, `requests`, `beautifulsoup4`, `python-dotenv`
|
200 |
-
|
201 |
-
### Testing Status
|
202 |
-
- **Functional**: Four main test files covering core functionality
|
203 |
-
- **Usage**: Run individual Python test modules directly
|
204 |
-
- **Coverage**: Basic component testing, no automated integration tests
|
205 |
-
|
206 |
-
## Known Issues and Compatibility
|
207 |
-
|
208 |
-
### RAG Processing "Connection errored out" Issue
|
209 |
-
- **Issue**: Server crashes or hangs during document processing with "Connection errored out" error
|
210 |
-
- **Root Cause**: Memory-intensive embedding model download/initialization causing server timeout
|
211 |
-
- **Symptoms**:
|
212 |
-
- `stream.ts:185 Method not implemented.`
|
213 |
-
- `Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING`
|
214 |
-
- Server becomes unresponsive during RAG document processing
|
215 |
-
- **Solutions**:
|
216 |
-
1. **Use smaller batch sizes**: Reduced from 32 to 16 chunks per batch
|
217 |
-
2. **Improved error handling**: Better feedback for network/memory issues
|
218 |
-
3. **CPU-only processing**: Force CPU usage to avoid GPU/multiprocessing conflicts
|
219 |
-
4. **Environment variables**: Set `TOKENIZERS_PARALLELISM=false` to prevent multiprocessing issues
|
220 |
-
5. **Smaller model**: Default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `all-MiniLM-L6-v2`
|
221 |
-
- **Testing**: Run `python test_rag_fix.py` to verify RAG functionality
|
222 |
-
- **Prevention**: Process documents one at a time, use smaller files (<5MB)
|
223 |
-
|
224 |
-
### Gradio 4.44.1 JSON Schema Bug
|
225 |
-
- **Issue**: TypeError in `json_schema_to_python_type` prevents app startup in some environments
|
226 |
-
- **Symptom**: "argument of type 'bool' is not iterable" error during API schema generation
|
227 |
-
- **Workaround**: Individual component functions work correctly
|
228 |
-
- **Solution**: Upgrade to Gradio 5.x for full compatibility
|
229 |
-
|
230 |
-
### Python Version Requirements
|
231 |
-
- **Minimum**: Python 3.9 (for Gradio 4.44.1)
|
232 |
-
- **Recommended**: Python 3.11+ (for Gradio 5.x and optimal performance)
|
233 |
-
|
234 |
-
## Common Claude Code Anti-Patterns to Avoid
|
235 |
-
|
236 |
-
### Message Format Reversion
|
237 |
-
**❌ Don't revert to:** New dictionary format in preview functions
|
238 |
-
```python
|
239 |
-
# WRONG - breaks Gradio 4.44.1 ChatInterface
|
240 |
-
history.append({"role": "user", "content": message})
|
241 |
-
history.append({"role": "assistant", "content": response})
|
242 |
-
```
|
243 |
-
**✅ Keep:** Legacy tuple format for preview compatibility
|
244 |
-
```python
|
245 |
-
# CORRECT - works with current Gradio ChatInterface
|
246 |
-
history.append([message, response])
|
247 |
-
```
|
248 |
-
|
249 |
-
### Template Variable Substitution
|
250 |
-
**❌ Don't change:** Template string escaping patterns in `SPACE_TEMPLATE`
|
251 |
-
- Keep double backslashes: `\\n\\n` (becomes `\n\n` after Python string processing)
|
252 |
-
- Keep double braces: `{{variable}}` (becomes `{variable}` after format())
|
253 |
-
- **Reason**: Template undergoes two levels of processing (Python format + HuggingFace deployment)
|
254 |
-
|
255 |
-
### Code Execution Re-Addition
|
256 |
-
**❌ Don't re-add:** Code execution functionality has been intentionally removed
|
257 |
-
- Do not add `enable_code_execution` parameters back to functions
|
258 |
-
- Do not create code execution UI components
|
259 |
-
- Do not add code execution logic to preview or generation workflows
|
260 |
-
- **Reason**: Code execution functionality was removed by design
|
261 |
-
|
262 |
-
### Conditional Dependency Loading
|
263 |
-
**❌ Don't remove:** `HAS_RAG` flag and conditional imports
|
264 |
-
```python
|
265 |
-
# WRONG - breaks installations without vector dependencies
|
266 |
-
from rag_tool import RAGTool
|
267 |
-
```
|
268 |
-
**✅ Keep:** Graceful degradation pattern
|
269 |
-
```python
|
270 |
-
# CORRECT - allows app to work without optional dependencies
|
271 |
-
try:
|
272 |
-
from rag_tool import RAGTool
|
273 |
-
HAS_RAG = True
|
274 |
-
except ImportError:
|
275 |
-
HAS_RAG = False
|
276 |
-
RAGTool = None
|
277 |
-
```
|
278 |
-
|
279 |
-
### URL Management and Preview Functionality
|
280 |
-
**❌ Don't remove:** Dynamic URL add/remove functionality or real API integration in preview
|
281 |
-
- Keep `add_urls()`, `remove_urls()`, `add_chat_urls()`, `remove_chat_urls()` functions
|
282 |
-
- Maintain URL count state management with `gr.State()`
|
283 |
-
- Keep actual OpenRouter API calls in preview mode when `OPENROUTER_API_KEY` is set
|
284 |
-
- **Reason**: Users expect scalable URL input interface and realistic preview testing
|
285 |
-
|
286 |
-
## Development-Only Utilities
|
287 |
-
|
288 |
-
### MCP Servers
|
289 |
-
- **Gradio Docs**: Available at https://gradio-docs-mcp.hf.space/gradio_api/mcp/sse
|
290 |
-
- Use `gradio_docs.py` utility for development assistance
|
291 |
-
- **CRITICAL**: Do NOT import in main application - this is for development tooling only
|
292 |
-
|
293 |
-
Usage for development:
|
294 |
-
```bash
|
295 |
-
python -c "from gradio_docs import gradio_docs; print(gradio_docs.search_docs('ChatInterface'))"
|
296 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|