milwright commited on
Commit
1920e6d
·
verified ·
1 Parent(s): 0b0106d

Delete CLAUDE.md

Browse files
Files changed (1) hide show
  1. CLAUDE.md +0 -296
CLAUDE.md DELETED
@@ -1,296 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- Chat UI Helper is a Gradio-based tool for generating and configuring chat interfaces for HuggingFace Spaces. It creates deployable packages with custom assistants, web scraping capabilities, and optional vector RAG functionality.
8
-
9
- ## Core Architecture
10
-
11
- ### Main Application Flow (`app.py`)
12
- The application follows a three-tab Gradio interface pattern:
13
- 1. **Configuration Tab**: Space setup, assistant configuration, tool settings
14
- 2. **Sandbox Preview Tab**: Interactive testing with real OpenRouter API integration
15
- 3. **Support Docs Tab**: Comprehensive guidance and templates via `support_docs.py`
16
-
17
- ### Template Generation System
18
- - `SPACE_TEMPLATE` (lines 130-710): Complete HuggingFace Space template with export functionality
19
- - `generate_zip()` function (lines 869-935): Orchestrates package creation with all dependencies
20
- - Key template variables: `{system_prompt}`, `{model}`, `{enable_vector_rag}`, `{api_key_var}`, `{grounding_urls}`, `{enable_dynamic_urls}`, `{enable_web_search}`
21
-
22
- ### Preview Sandbox Architecture
23
- - Real OpenRouter API integration in preview mode (`preview_chat_response()` line 1185)
24
- - URL context testing with dynamic add/remove functionality
25
- - Configuration-aware responses using exact model and parameters from user configuration
26
- - Fallback messaging when `OPENROUTER_API_KEY` environment variable not set
27
- - Legacy tuple format compatibility for Gradio 4.44.1 ChatInterface
28
- - Comprehensive debugging with enhanced error handling and API response validation
29
-
30
- ### Document Processing Pipeline (RAG)
31
- - **RAGTool** (`rag_tool.py`): Main orchestrator with 10MB file size validation
32
- - **DocumentProcessor** (`document_processor.py`): PDF/DOCX/TXT/MD parsing with semantic chunking (800 chars, 100 overlap)
33
- - **VectorStore** (`vector_store.py`): FAISS-based similarity search and base64 serialization
34
-
35
- ### Web Scraping Architecture
36
- Simple HTTP + BeautifulSoup approach with crawl4ai integration:
37
- - `enhanced_fetch_url_content()` (lines 79-128): Enhanced requests with timeout and user-agent headers
38
- - Content cleaning: Removes scripts, styles, navigation elements
39
- - Content limits: ~4000 character truncation for context management
40
- - URL content caching: `get_cached_grounding_context()` (line 1441) prevents redundant fetches
41
- - `extract_urls_from_text()` (line 51): Regex-based URL extraction for dynamic fetching
42
-
43
- ## Development Commands
44
-
45
- ### Environment Setup
46
- **Important**: This application requires Python ≥3.10 for Gradio 5.x compatibility.
47
-
48
- ```bash
49
- # Recommended: Use Python 3.11+ environment
50
- python3.11 -m venv venv311
51
- source venv311/bin/activate # or venv311\Scripts\activate on Windows
52
- pip install -r requirements.txt
53
- ```
54
-
55
- ### Running the Application
56
- ```bash
57
- # With virtual environment activated
58
- python app.py
59
- ```
60
-
61
- ### Testing Commands
62
- ```bash
63
- # Test vector database functionality (requires all RAG dependencies)
64
- python test_vector_db.py
65
-
66
- # Test RAG fixes and error handling
67
- python test_rag_fix.py
68
-
69
- # Test OpenRouter API key validation
70
- python test_api_key.py
71
-
72
- # Test minimal Gradio functionality (for debugging)
73
- python test_minimal.py
74
-
75
- # Test preview functionality components
76
- python test_preview.py
77
-
78
- # Test individual RAG components
79
- python -c "from test_vector_db import test_document_processing; test_document_processing()"
80
- python -c "from test_vector_db import test_vector_store; test_vector_store()"
81
- python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
82
- ```
83
-
84
- ### Pre-Test Setup for RAG Components
85
- ```bash
86
- # Create test document for vector database testing
87
- echo "This is a test document for RAG functionality testing." > test_document.txt
88
-
89
- # Verify all dependencies are installed
90
- python -c "import sentence_transformers, faiss, fitz; print('RAG dependencies available')"
91
- ```
92
-
93
- ## Key Dependencies and Versions
94
-
95
- ### Required Dependencies
96
- - **Gradio ≥4.44.1**: Main UI framework (5.37.0 recommended for Python ≥3.10)
97
- - **requests ≥2.32.3**: HTTP requests for web content fetching
98
- - **beautifulsoup4 ≥4.12.3**: HTML parsing for web scraping
99
- - **python-dotenv ≥1.0.0**: Environment variable management
100
-
101
- ### Optional RAG Dependencies
102
- - **sentence-transformers ≥2.2.2**: Text embeddings
103
- - **faiss-cpu ==1.7.4**: Vector similarity search
104
- - **PyMuPDF ≥1.23.0**: PDF text extraction
105
- - **python-docx ≥0.8.11**: DOCX document processing
106
- - **numpy ==1.26.4**: Numerical operations
107
-
108
- ### Optional Web Search Dependencies
109
- - **crawl4ai ≥0.2.0**: Advanced web crawling for web search functionality
110
- - **aiohttp ≥3.8.0**: Async HTTP client for crawl4ai
111
-
112
- ## Configuration Patterns
113
-
114
- ### Conditional Dependency Loading
115
- ```python
116
- try:
117
- from rag_tool import RAGTool
118
- HAS_RAG = True
119
- except ImportError:
120
- HAS_RAG = False
121
- RAGTool = None
122
- ```
123
- This pattern allows graceful degradation when optional vector dependencies are unavailable.
124
-
125
- ### Template Variable Substitution
126
- Generated spaces use these key substitutions:
127
- - `{system_prompt}`: Combined assistant configuration
128
- - `{grounding_urls}`: Static URL list for context
129
- - `{enable_dynamic_urls}`: Runtime URL fetching capability
130
- - `{enable_vector_rag}`: Document search integration
131
- - `{enable_web_search}`: Web search integration via crawl4ai
132
- - `{rag_data_json}`: Serialized embeddings and chunks
133
- - `{api_key_var}`: Customizable API key environment variable name
134
-
135
- ### Access Control Pattern
136
- - Environment variable `SPACE_ACCESS_CODE` for student access control
137
- - Global state management for session-based access in generated spaces
138
- - Security-first approach storing credentials as HuggingFace Spaces secrets
139
-
140
- ### RAG Integration Workflow
141
- 1. Documents uploaded through Gradio File component with conditional visibility (`HAS_RAG` flag)
142
- 2. Processed via DocumentProcessor (PDF/DOCX/TXT/MD support) in `process_documents()` function
143
- 3. Chunked and embedded using sentence-transformers (800 chars, 100 overlap)
144
- 4. FAISS index created and serialized to base64 for deployment portability
145
- 5. Embedded in generated template via `{rag_data_json}` template variable
146
-
147
- ## Implementation Notes
148
-
149
- ### Research Template System (Simplified)
150
- - **Simple Toggle**: `toggle_research_assistant()` function (line 1704) provides simple on/off functionality
151
- - **Direct System Prompt**: Enables predefined academic research prompt with DOI verification and LibKey integration
152
- - **Auto-Enable Dynamic URLs**: Research template automatically enables dynamic URL fetching for academic sources
153
- - **Template Content**: Academic inquiry focus with DOI-verified sources, fact-checking, and proper citation requirements
154
-
155
- ### State Management Across Tabs
156
- - Extensive use of `gr.State()` for maintaining session data
157
- - Cross-tab functionality through shared state variables (`sandbox_state`, `preview_config_state`)
158
- - URL content caching to prevent redundant web requests (`url_content_cache` global variable)
159
- - Preview debugging with comprehensive error handling and API response validation
160
-
161
- ### Gradio Compatibility and Message Format Handling
162
- - **Target Version**: Gradio 5.37.0 (requires Python ≥3.10)
163
- - **Legacy Support**: Gradio 4.44.1 compatibility with JSON schema workarounds
164
- - **Message Format**: Preview uses legacy tuple format `[user_msg, bot_msg]` for ChatInterface compatibility
165
- - **Generated Spaces**: Use modern dictionary format `{"role": "user", "content": "..."}` for OpenRouter API
166
-
167
- ### Security Considerations
168
- - Never embed API keys or access codes in generated templates
169
- - Environment variable pattern for all sensitive configuration (`{api_key_var}` template variable)
170
- - Input validation for uploaded files and URL processing
171
- - Content length limits for web scraping operations
172
-
173
- ## Tool Configuration Changes
174
-
175
- ### Code Execution Functionality Removed
176
- **Important**: Code execution functionality has been completely removed from the application. Do not attempt to re-add it.
177
-
178
- - All `enable_code_execution` parameters and checkboxes have been removed
179
- - The `toggle_code_execution` function has been removed
180
- - Code execution logic in preview and generation functions has been removed
181
- - Generated spaces no longer support code execution capabilities
182
-
183
- ### Web Search Integration
184
- - **Enable Web Search**: Checkbox to enable web search functionality using crawl4ai
185
- - **Technology**: Uses crawl4ai library with DuckDuckGo for search results
186
- - **Implementation**: Integrated in both preview mode and generated spaces
187
- - **Fallback**: Simple HTTP requests if crawl4ai is not available
188
-
189
- ## Testing Infrastructure
190
-
191
- ### Current Test Structure
192
- - `test_vector_db.py`: Comprehensive RAG component testing
193
- - `test_api_key.py`: OpenRouter API validation
194
- - `test_minimal.py`: Basic Gradio functionality debugging
195
- - `test_preview.py`: Preview functionality component testing
196
-
197
- ### Test Dependencies
198
- RAG testing requires: `sentence-transformers`, `faiss-cpu`, `PyMuPDF`, `python-docx`
199
- Core testing requires: `gradio`, `requests`, `beautifulsoup4`, `python-dotenv`
200
-
201
- ### Testing Status
202
- - **Functional**: Four main test files covering core functionality
203
- - **Usage**: Run individual Python test modules directly
204
- - **Coverage**: Basic component testing, no automated integration tests
205
-
206
- ## Known Issues and Compatibility
207
-
208
- ### RAG Processing "Connection errored out" Issue
209
- - **Issue**: Server crashes or hangs during document processing with "Connection errored out" error
210
- - **Root Cause**: Memory-intensive embedding model download/initialization causing server timeout
211
- - **Symptoms**:
212
- - `stream.ts:185 Method not implemented.`
213
- - `Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING`
214
- - Server becomes unresponsive during RAG document processing
215
- - **Solutions**:
216
- 1. **Use smaller batch sizes**: Reduced from 32 to 16 chunks per batch
217
- 2. **Improved error handling**: Better feedback for network/memory issues
218
- 3. **CPU-only processing**: Force CPU usage to avoid GPU/multiprocessing conflicts
219
- 4. **Environment variables**: Set `TOKENIZERS_PARALLELISM=false` to prevent multiprocessing issues
220
- 5. **Smaller model**: Default model changed from `sentence-transformers/all-MiniLM-L6-v2` to `all-MiniLM-L6-v2`
221
- - **Testing**: Run `python test_rag_fix.py` to verify RAG functionality
222
- - **Prevention**: Process documents one at a time, use smaller files (<5MB)
223
-
224
- ### Gradio 4.44.1 JSON Schema Bug
225
- - **Issue**: TypeError in `json_schema_to_python_type` prevents app startup in some environments
226
- - **Symptom**: "argument of type 'bool' is not iterable" error during API schema generation
227
- - **Workaround**: Individual component functions work correctly
228
- - **Solution**: Upgrade to Gradio 5.x for full compatibility
229
-
230
- ### Python Version Requirements
231
- - **Minimum**: Python 3.9 (for Gradio 4.44.1)
232
- - **Recommended**: Python 3.11+ (for Gradio 5.x and optimal performance)
233
-
234
- ## Common Claude Code Anti-Patterns to Avoid
235
-
236
- ### Message Format Reversion
237
- **❌ Don't revert to:** New dictionary format in preview functions
238
- ```python
239
- # WRONG - breaks Gradio 4.44.1 ChatInterface
240
- history.append({"role": "user", "content": message})
241
- history.append({"role": "assistant", "content": response})
242
- ```
243
- **✅ Keep:** Legacy tuple format for preview compatibility
244
- ```python
245
- # CORRECT - works with current Gradio ChatInterface
246
- history.append([message, response])
247
- ```
248
-
249
- ### Template Variable Substitution
250
- **❌ Don't change:** Template string escaping patterns in `SPACE_TEMPLATE`
251
- - Keep double backslashes: `\\n\\n` (becomes `\n\n` after Python string processing)
252
- - Keep double braces: `{{variable}}` (becomes `{variable}` after format())
253
- - **Reason**: Template undergoes two levels of processing (Python format + HuggingFace deployment)
254
-
255
- ### Code Execution Re-Addition
256
- **❌ Don't re-add:** Code execution functionality has been intentionally removed
257
- - Do not add `enable_code_execution` parameters back to functions
258
- - Do not create code execution UI components
259
- - Do not add code execution logic to preview or generation workflows
260
- - **Reason**: Code execution functionality was removed by design
261
-
262
- ### Conditional Dependency Loading
263
- **❌ Don't remove:** `HAS_RAG` flag and conditional imports
264
- ```python
265
- # WRONG - breaks installations without vector dependencies
266
- from rag_tool import RAGTool
267
- ```
268
- **✅ Keep:** Graceful degradation pattern
269
- ```python
270
- # CORRECT - allows app to work without optional dependencies
271
- try:
272
- from rag_tool import RAGTool
273
- HAS_RAG = True
274
- except ImportError:
275
- HAS_RAG = False
276
- RAGTool = None
277
- ```
278
-
279
- ### URL Management and Preview Functionality
280
- **❌ Don't remove:** Dynamic URL add/remove functionality or real API integration in preview
281
- - Keep `add_urls()`, `remove_urls()`, `add_chat_urls()`, `remove_chat_urls()` functions
282
- - Maintain URL count state management with `gr.State()`
283
- - Keep actual OpenRouter API calls in preview mode when `OPENROUTER_API_KEY` is set
284
- - **Reason**: Users expect scalable URL input interface and realistic preview testing
285
-
286
- ## Development-Only Utilities
287
-
288
- ### MCP Servers
289
- - **Gradio Docs**: Available at https://gradio-docs-mcp.hf.space/gradio_api/mcp/sse
290
- - Use `gradio_docs.py` utility for development assistance
291
- - **CRITICAL**: Do NOT import in main application - this is for development tooling only
292
-
293
- Usage for development:
294
- ```bash
295
- python -c "from gradio_docs import gradio_docs; print(gradio_docs.search_docs('ChatInterface'))"
296
- ```