Organize repository structure: move files to appropriate directories

.gitignore

@@ -29,22 +29,17 @@ Thumbs.db
 # Generated files
 *.zip
 
-# Test files
-test_*.py
-*_test.py
-
-# Documentation/notes (keep only essential docs)
-*_SUMMARY.md
-WEB_SEARCH_IMPLEMENTATION.md
-simplified_documentation.md
-file_upload_proposal.md
-TEST_PROCEDURE.md
+# Organized directories
+docs/
+tests/
+development/
+temp/
 
 # Gradio cache
 .gradio/
 flagged/
 
-# Development images
+# Development images (except secret.png)
 *.png
 !secret.png
 *.jpg
@@ -52,12 +47,4 @@ flagged/
 *.gif
 
 # Claude local files
-claude.local.md
-.claude/
-
-# Config files
-*.aiconfig.json
-
-# Sample/temp files
-test_*.txt
-*_sample.txt
+.claude/

TEST_PROCEDURE.md

@@ -1,285 +0,0 @@
-# Chat UI Helper - Comprehensive Test Procedure
-
-This document outlines a systematic test procedure for validating the Chat UI Helper application after new commits. This procedure ensures all components function correctly and can be iterated upon as the project evolves.
-
-## Pre-Test Setup
-
-### Environment Verification
-```bash
-# Verify Python environment
-python --version  # Should be 3.8+
-
-# Install/update dependencies
-pip install -r requirements.txt
-
-# Verify optional dependencies status
-python -c "
-try:
-    import sentence_transformers, faiss, fitz, docx
-    print('✅ All RAG dependencies available')
-except ImportError as e:
-    print(f'⚠️  Optional RAG dependencies missing: {e}')
-"
-```
-
-### Test Data Preparation
-```bash
-# Ensure test document exists
-echo "This is a test document for RAG functionality testing." > test_document.txt
-
-# Create test directory structure if needed
-mkdir -p test_outputs
-```
-
-## Test Categories
-
-### 1. Core Application Tests
-
-#### 1.1 Application Startup
-```bash
-# Test basic application launch
-python app.py &
-APP_PID=$!
-sleep 10
-curl -f http://localhost:7860 > /dev/null && echo "✅ App started successfully" || echo "❌ App failed to start"
-kill $APP_PID
-```
-
-#### 1.2 Gradio Interface Validation
-- [ ] Application loads without errors
-- [ ] Two tabs visible: "Spaces Configuration" and "Chat Support"
-- [ ] All form fields render correctly
-- [ ] Template selection works (Custom vs Research Assistant)
-- [ ] File upload components appear when RAG is enabled
-
-### 2. Vector RAG Component Tests
-
-#### 2.1 Individual Component Testing
-```bash
-# Test document processing
-python -c "from test_vector_db import test_document_processing; test_document_processing()"
-
-# Test vector store functionality
-python -c "from test_vector_db import test_vector_store; test_vector_store()"
-
-# Test full RAG pipeline
-python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
-```
-
-#### 2.2 RAG Integration Tests
-- [ ] Document upload accepts PDF, DOCX, TXT, MD files
-- [ ] File size validation (10MB limit) works
-- [ ] Documents are processed and chunked correctly
-- [ ] Vector embeddings are generated
-- [ ] Similarity search returns relevant results
-- [ ] RAG data serializes/deserializes properly for templates
-
-### 3. Space Generation Tests
-
-#### 3.1 Basic Space Creation
-- [ ] Generate space with minimal configuration
-- [ ] Verify all required files are created (app.py, requirements.txt, README.md, config.json)
-- [ ] Check generated app.py syntax is valid
-- [ ] Verify requirements.txt has correct dependencies
-- [ ] Ensure README.md contains proper deployment instructions
-
-#### 3.2 Advanced Feature Testing
-- [ ] Generate space with URL grounding enabled
-- [ ] Generate space with vector RAG enabled
-- [ ] Generate space with access code protection
-- [ ] Test template substitution works correctly
-- [ ] Verify environment variable security pattern
-
-### 4. Web Scraping Tests
-
-#### 4.1 Mock vs Production Mode
-```bash
-# Test in mock mode (lines 14-18 in app.py)
-# Verify placeholder content is returned
-
-# Test in production mode
-# Verify actual web content is fetched via HTTP requests
-```
-
-#### 4.2 URL Processing
-- [ ] Valid URLs are processed correctly
-- [ ] Invalid URLs are handled gracefully
-- [ ] Content extraction works for different site types
-- [ ] Rate limiting and error handling work
-
-### 5. Security and Configuration Tests
-
-#### 5.1 Environment Variable Handling
-- [ ] API keys are not embedded in generated templates
-- [ ] Access codes use environment variable pattern
-- [ ] Sensitive data is properly excluded from version control
-
-#### 5.2 Input Validation
-- [ ] File upload validation works
-- [ ] URL validation prevents malicious inputs
-- [ ] Content length limits are enforced
-- [ ] XSS prevention in user inputs
-
-### 6. Chat Support Tests
-
-#### 6.1 OpenRouter Integration
-- [ ] Chat responds when API key is configured
-- [ ] Proper error message when API key is missing
-- [ ] Message history formatting works correctly
-- [ ] URL grounding provides relevant context
-
-#### 6.2 Gradio 5.x Compatibility
-- [ ] Message format uses `type="messages"`
-- [ ] ChatInterface renders correctly
-- [ ] User/assistant message distinction works
-- [ ] Chat history persists during session
-
-## Automated Test Execution
-
-### Quick Test Suite
-```bash
-#!/bin/bash
-# quick_test.sh - Run essential tests
-
-echo "🔍 Running Quick Test Suite..."
-
-# 1. Syntax check
-python -m py_compile app.py && echo "✅ app.py syntax valid" || echo "❌ app.py syntax error"
-
-# 2. Import test
-python -c "import app; print('✅ App imports successfully')" 2>/dev/null || echo "❌ Import failed"
-
-# 3. RAG component test (if available)
-if python -c "from rag_tool import RAGTool" 2>/dev/null; then
-    python test_vector_db.py && echo "✅ RAG tests passed" || echo "❌ RAG tests failed"
-else
-    echo "⚠️  RAG components not available"
-fi
-
-# 4. Template generation test
-python -c "
-import app
-result = app.generate_zip('Test Space', 'Test Description', 'Test Role', 'Test Audience', 'Test Tasks', '', [], '', '', 'gpt-3.5-turbo', 0.7, 4000, [], False, False, None)
-assert result[0].endswith('.zip'), 'ZIP generation failed'
-print('✅ Space generation works')
-"
-
-echo "🎉 Quick test suite completed!"
-```
-
-### Full Test Suite
-```bash
-#!/bin/bash
-# full_test.sh - Comprehensive testing
-
-echo "🔍 Running Full Test Suite..."
-
-# Run all component tests
-./quick_test.sh
-
-# Additional integration tests
-echo "🧪 Running integration tests..."
-
-# Test with different configurations
-# Test error handling
-# Test edge cases
-# Performance tests
-
-echo "📊 Generating test report..."
-# Generate detailed test report
-```
-
-## Regression Test Checklist
-
-After each commit, verify:
-
-- [ ] All existing functionality still works
-- [ ] New features don't break existing features
-- [ ] Generated spaces deploy successfully to HuggingFace
-- [ ] Documentation is updated appropriately
-- [ ] Dependencies are correctly specified
-- [ ] Security patterns are maintained
-
-## Performance Benchmarks
-
-### Metrics to Track
-- Application startup time
-- Space generation time
-- Document processing time (for various file sizes)
-- Memory usage during RAG operations
-- API response times
-
-### Benchmark Commands
-```bash
-# Startup time
-time python -c "import app; print('App loaded')"
-
-# Space generation time
-time python -c "
-import app
-app.generate_zip('Benchmark', 'Test', 'Role', 'Audience', 'Tasks', '', [], '', '', 'gpt-3.5-turbo', 0.7, 4000, [], False, False, None)
-"
-
-# RAG processing time
-time python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
-```
-
-## Test Data Management
-
-### Sample Test Files
-- `test_document.txt` - Basic text document
-- `sample.pdf` - PDF document for upload testing
-- `sample.docx` - Word document for testing
-- `sample.md` - Markdown document for testing
-
-### Test Configuration Profiles
-- Minimal configuration (basic chat only)
-- Research assistant template
-- Full-featured (RAG + URL grounding + access control)
-- Edge case configurations
-
-## Continuous Integration Integration
-
-### GitHub Actions Integration
-```yaml
-# .github/workflows/test.yml
-name: Test Chat UI Helper
-on: [push, pull_request]
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: '3.9'
-      - name: Install dependencies
-        run: pip install -r requirements.txt
-      - name: Run test suite
-        run: ./quick_test.sh
-```
-
-## Future Test Enhancements
-
-### Planned Additions
-- [ ] Automated UI testing with Selenium
-- [ ] Load testing for generated spaces
-- [ ] Cross-browser compatibility testing
-- [ ] Mobile responsiveness testing
-- [ ] Accessibility testing
-- [ ] Multi-language content testing
-
-### Test Coverage Goals
-- [ ] 90%+ code coverage for core components
-- [ ] All user workflows tested end-to-end
-- [ ] Error conditions properly tested
-- [ ] Performance regression detection
-
----
-
-**Last Updated**: 2025-07-13
-**Version**: 1.0
-**Maintained by**: Development Team
-
-This test procedure should be updated whenever new features are added or existing functionality is modified.

claude.local.md

@@ -1,314 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-Chat UI Helper is a Gradio-based tool for generating and configuring chat interfaces for HuggingFace Spaces. It creates deployable packages with custom assistants, web scraping capabilities, and optional vector RAG functionality.
-
-## Core Architecture
-
-### Main Application Flow (`app.py`)
-The application follows a three-tab Gradio interface pattern:
-1. **Configuration Tab**: Space setup, assistant configuration, tool settings (lines 1267-1589)
-2. **Sandbox Preview Tab**: Interactive testing with real OpenRouter API integration (lines 1591-1699) 
-3. **Support Docs Tab**: Comprehensive guidance and templates via `support_docs.py`
-
-### Template Generation System
-- `SPACE_TEMPLATE` (lines 50-347): Complete HuggingFace Space template with export functionality and legacy tuple format compatibility
-- `generate_zip()` function (lines 562-652): Orchestrates package creation with all dependencies
-- Key template variables: `{system_prompt}`, `{model}`, `{enable_vector_rag}`, `{api_key_var}`, `{grounding_urls}`, `{enable_dynamic_urls}`
-
-### Preview Sandbox Architecture (Enhanced)
-- Real OpenRouter API integration in preview mode (`preview_chat_response()` line 855)
-- URL context testing with dynamic add/remove functionality
-- Configuration-aware responses using exact model and parameters from user configuration
-- Fallback messaging when `OPENROUTER_API_KEY` environment variable not set
-- Legacy tuple format compatibility for Gradio 4.44.1 ChatInterface
-- **Comprehensive Debugging**: Enhanced error handling with detailed API response validation (lines 928-955)
-  - Empty response detection and logging
-  - API structure validation (choices, message, content)
-  - Request payload debugging for troubleshooting
-  - Timeout handling (30 seconds) for API requests
-
-### Document Processing Pipeline (RAG)
-- **RAGTool** (`rag_tool.py`): Main orchestrator with 10MB file size validation (lines 19-79)
-- **DocumentProcessor** (`document_processor.py`): PDF/DOCX/TXT/MD parsing with semantic chunking (800 chars, 100 overlap)
-- **VectorStore** (`vector_store.py`): FAISS-based similarity search and base64 serialization
-
-### Web Scraping Architecture
-Simple HTTP + BeautifulSoup approach (replacing previous Crawl4AI):
-- `fetch_url_content()` (lines 390-415): Basic requests with timeout and user-agent headers
-- Content cleaning: Removes scripts, styles, navigation elements
-- Content limits: ~4000 character truncation for context management
-- URL content caching: `get_cached_grounding_context()` (line 1019) prevents redundant fetches
-- `extract_urls_from_text()` (line 44): Regex-based URL extraction for dynamic fetching
-
-## Development-Only Utilities
-
-### MCP Servers
-- **Gradio Docs**: Available at https://gradio-docs-mcp.hf.space/gradio_api/mcp/sse
-- Use `gradio_docs.py` utility for development assistance
-- **CRITICAL**: Do NOT import in main application - this is for development tooling only
-
-Usage for development:
-```bash
-python -c "from gradio_docs import gradio_docs; print(gradio_docs.search_docs('ChatInterface'))"
-```
-
-## Development Commands
-
-### Environment Setup
-**Important**: This application requires Python ≥3.10 for Gradio 5.x compatibility.
-
-```bash
-# Recommended: Use Python 3.11+ environment
-python3.11 -m venv venv311
-source venv311/bin/activate  # or venv311\Scripts\activate on Windows
-pip install -r requirements.txt
-```
-
-### Running the Application
-```bash
-# With virtual environment activated
-python app.py
-```
-
-### Testing Commands
-```bash
-# Test vector database functionality (requires all RAG dependencies)
-python test_vector_db.py
-
-# Test OpenRouter API key validation
-python test_api_key.py
-
-# Test minimal Gradio functionality (for debugging)
-python test_minimal.py
-
-# Test preview functionality components (new)
-python test_preview.py
-
-# Test individual RAG components
-python -c "from test_vector_db import test_document_processing; test_document_processing()"
-python -c "from test_vector_db import test_vector_store; test_vector_store()"
-python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
-```
-
-### Pre-Test Setup for RAG Components
-```bash
-# Create test document for vector database testing
-echo "This is a test document for RAG functionality testing." > test_document.txt
-
-# Verify all dependencies are installed
-python -c "import sentence_transformers, faiss, fitz; print('RAG dependencies available')"
-```
-
-### Key Dependencies and Versions
-
-#### Required Dependencies
-- **Gradio ≥4.44.1**: Main UI framework (5.37.0 recommended for Python ≥3.10)
-- **requests ≥2.32.3**: HTTP requests for web content fetching
-- **beautifulsoup4 ≥4.12.3**: HTML parsing for web scraping
-- **python-dotenv ≥1.0.0**: Environment variable management
-
-#### Optional RAG Dependencies
-- **sentence-transformers ≥2.2.2**: Text embeddings
-- **faiss-cpu ==1.7.4**: Vector similarity search
-- **PyMuPDF ≥1.23.0**: PDF text extraction
-- **python-docx ≥0.8.11**: DOCX document processing
-- **numpy ==1.26.4**: Numerical operations
-
-## Configuration Patterns
-
-### Conditional Dependency Loading
-```python
-try:
-    from rag_tool import RAGTool
-    HAS_RAG = True
-except ImportError:
-    HAS_RAG = False
-    RAGTool = None
-```
-This pattern allows graceful degradation when optional vector dependencies are unavailable.
-
-### Template Variable Substitution
-Generated spaces use these key substitutions:
-- `{system_prompt}`: Combined assistant configuration
-- `{grounding_urls}`: Static URL list for context
-- `{enable_dynamic_urls}`: Runtime URL fetching capability
-- `{enable_vector_rag}`: Document search integration
-- `{rag_data_json}`: Serialized embeddings and chunks
-- `{api_key_var}`: Customizable API key environment variable name
-
-### Access Control Pattern
-- Environment variable `SPACE_ACCESS_CODE` for student access control
-- Global state management for session-based access in generated spaces
-- Security-first approach storing credentials as HuggingFace Spaces secrets
-
-### RAG Integration Workflow
-1. Documents uploaded through Gradio File component with conditional visibility (`HAS_RAG` flag)
-2. Processed via DocumentProcessor (PDF/DOCX/TXT/MD support) in `process_documents()` function
-3. Chunked and embedded using sentence-transformers (800 chars, 100 overlap)
-4. FAISS index created and serialized to base64 for deployment portability
-5. Embedded in generated template via `{rag_data_json}` template variable
-
-## Implementation Notes
-
-### Research Template System (Simplified)
-- **Simple Toggle**: `toggle_research_assistant()` function (line 1225) now provides simple on/off functionality
-- **Direct System Prompt**: Enables predefined academic research prompt with DOI verification and LibKey integration
-- **Auto-Enable Dynamic URLs**: Research template automatically enables dynamic URL fetching for academic sources
-- **Template Content**: Academic inquiry focus with DOI-verified sources, fact-checking, and proper citation requirements
-- **Note**: Previous complex field system (Role and Purpose, Intended Audience, Key Tasks, Additional Context) has been removed for simplified architecture
-
-### State Management Across Tabs
-- Extensive use of `gr.State()` for maintaining session data
-- Cross-tab functionality through shared state variables (`sandbox_state`, `preview_config_state`)
-- URL content caching to prevent redundant web requests (`url_content_cache` global variable)
-- Preview debugging with comprehensive error handling and API response validation
-
-### Gradio Compatibility and Message Format Handling
-- **Target Version**: Gradio 5.37.0 (requires Python ≥3.10)
-- **Legacy Support**: Gradio 4.44.1 compatibility with JSON schema workarounds
-- **Message Format**: Preview uses legacy tuple format `[user_msg, bot_msg]` for ChatInterface compatibility
-- **Generated Spaces**: Use modern dictionary format `{"role": "user", "content": "..."}` for OpenRouter API
-
-### Security Considerations
-- Never embed API keys or access codes in generated templates
-- Environment variable pattern for all sensitive configuration (`{api_key_var}` template variable)
-- Input validation for uploaded files and URL processing
-- Content length limits for web scraping operations
-
-## Testing Infrastructure
-
-### Current Test Structure
-- `test_vector_db.py`: Comprehensive RAG component testing (196 lines)
-- `test_api_key.py`: OpenRouter API validation (85 lines)  
-- `test_minimal.py`: Basic Gradio functionality debugging (20 lines)
-- `test_preview.py`: Preview functionality component testing (URL extraction, fetching, chat response)
-
-### Test Dependencies
-RAG testing requires: `sentence-transformers`, `faiss-cpu`, `PyMuPDF`, `python-docx`
-
-Core testing requires: `gradio`, `requests`, `beautifulsoup4`, `python-dotenv`
-
-### Testing Status
-- **Functional**: Three main test files covering core functionality
-- **Missing**: Automated test scripts referenced in TEST_PROCEDURE.md (`quick_test.sh`, `full_test.sh`) are documented but not implemented
-- **Usage**: Run individual Python test modules directly
-
-## File Structure Notes
-
-### Generated Space Structure
-All generated HuggingFace Spaces follow consistent structure:
-1. Configuration section with environment variable loading
-2. Web scraping functions (simple HTTP requests with BeautifulSoup)
-3. RAG context retrieval (if enabled)
-4. OpenRouter API integration with conversation history
-5. Gradio ChatInterface with access control
-
-### Development Files Not For Production
-- `gradio_docs.py`: MCP server integration (development only)
-- `test_*.py`: Testing utilities
-- `TEST_PROCEDURE.md`: Comprehensive testing methodology
-- `file_upload_proposal.md`: Technical architecture proposals
-
-## Known Issues and Compatibility
-
-### Gradio 4.44.1 JSON Schema Bug
-- **Issue**: TypeError in `json_schema_to_python_type` prevents app startup in some environments
-- **Symptom**: "argument of type 'bool' is not iterable" error during API schema generation
-- **Workaround**: Individual component functions work correctly (as verified by `test_preview.py`)
-- **Solution**: Upgrade to Gradio 5.x for full compatibility, or wait for Gradio 4.x patch
-
-### Message Format Compatibility
-- **Preview Mode**: Uses legacy tuple format `[user_msg, bot_msg]` for Gradio 4.44.1 ChatInterface
-- **Generated Spaces**: Use modern dictionary format for OpenRouter API compatibility
-- **Cross-Version Support**: Template generation handles both formats appropriately
-
-### Python Version Requirements
-- **Minimum**: Python 3.9 (for Gradio 4.44.1)
-- **Recommended**: Python 3.11+ (for Gradio 5.x and optimal performance)
-
-## Common Claude Code Anti-Patterns to Avoid
-
-### Message Format Reversion
-**❌ Don't revert to:** New dictionary format in preview functions
-```python
-# WRONG - breaks Gradio 4.44.1 ChatInterface
-history.append({"role": "user", "content": message})
-history.append({"role": "assistant", "content": response})
-```
-**✅ Keep:** Legacy tuple format for preview compatibility
-```python
-# CORRECT - works with current Gradio ChatInterface
-history.append([message, response])
-```
-
-### Template Variable Substitution
-**❌ Don't change:** Template string escaping patterns in `SPACE_TEMPLATE`
-- Keep double backslashes: `\\n\\n` (becomes `\n\n` after Python string processing)
-- Keep double braces: `{{variable}}` (becomes `{variable}` after format())
-- **Reason**: Template undergoes two levels of processing (Python format + HuggingFace deployment)
-
-### Research Template Function Signature
-**✅ Current Implementation:** Simplified function signature for research template
-```python
-# CURRENT - simplified toggle with direct system prompt management
-def toggle_research_assistant(enable_research):
-    if enable_research:
-        return (gr.update(value=combined_prompt), gr.update(value=True))
-    else:
-        return (gr.update(value=""), gr.update(value=False))
-```
-**❌ Don't revert to:** Complex field management patterns that are no longer needed
-- The research template no longer uses separate fields for role, audience, tasks, context
-- Current implementation directly manages system prompt and dynamic URL setting only
-
-### Import Organization Anti-Patterns
-**❌ Don't move:** `extract_urls_from_text()` back into template string
-- Function must remain in main app code (line 44) for preview functionality
-- Template version is for generated spaces only
-
-### URL Management Simplification
-**❌ Don't remove:** Dynamic URL add/remove functionality
-- Keep `add_urls()`, `remove_urls()`, `add_chat_urls()`, `remove_chat_urls()` functions
-- Maintain URL count state management with `gr.State()`
-- **Reason**: Users expect scalable URL input interface
-
-### Preview Functionality Degradation
-**❌ Don't revert to:** Simple mock responses in preview
-```python
-# WRONG - provides no real testing value
-def preview_chat_response(message, history, config_data):
-    return "", history + [[message, "Mock response"]]
-```
-**✅ Keep:** Real API integration with comprehensive debugging
-- Actual OpenRouter API calls when `OPENROUTER_API_KEY` is set
-- URL context fetching and processing
-- Configuration-aware responses using exact user settings
-- Comprehensive debugging for empty responses and API errors (lines 928-955)
-
-### Research Template Simplification
-**✅ Current Implementation:** Simplified research template system
-- Simple toggle functionality without complex field management
-- Direct system prompt injection for academic research use cases
-- Auto-enables dynamic URL fetching for academic sources
-- **Reason**: Simplified architecture reduces maintenance complexity while preserving core functionality
-
-### Conditional Dependency Loading
-**❌ Don't remove:** `HAS_RAG` flag and conditional imports
-```python
-# WRONG - breaks installations without vector dependencies
-from rag_tool import RAGTool
-```
-**✅ Keep:** Graceful degradation pattern
-```python
-# CORRECT - allows app to work without optional dependencies
-try:
-    from rag_tool import RAGTool
-    HAS_RAG = True
-except ImportError:
-    HAS_RAG = False
-    RAGTool = None
-```

file_upload_proposal.md

@@ -1,144 +0,0 @@
-# File Upload System Proposal for Faculty Course Materials
-
-Based on your existing architecture, here's a comprehensive proposal for implementing file uploads with efficient parsing and deployment preservation:
-
-## Core Architecture Design
-
-### 1. File Processing Pipeline
-```
-Upload → Parse → Chunk → Vector Store → RAG Integration → Deployment Package
-```
-
-### 2. File Storage Structure
-```
-/course_materials/
-├── raw_files/           # Original uploaded files
-├── processed/           # Parsed text content
-├── embeddings/          # Vector representations
-└── metadata.json        # File tracking & metadata
-```
-
-## Implementation Components
-
-### File Upload Handler (app.py:352-408 enhancement)
-- Add `gr.File(file_types=[".pdf", ".docx", ".txt", ".md"])` component
-- Support multiple file uploads with `file_count="multiple"`
-- Implement file validation and size limits (10MB per file)
-
-### Document Parser Service (new: `document_parser.py`)
-- **PDF**: PyMuPDF for text extraction with layout preservation
-- **DOCX**: python-docx for structured content
-- **TXT/MD**: Direct text processing with metadata extraction
-- **Auto-detection**: File type identification and appropriate parser routing
-
-### RAG Integration (enhancement to existing web scraping system)
-- **Chunking Strategy**: Semantic chunking (500-1000 tokens with 100-token overlap)
-- **Embeddings**: sentence-transformers/all-MiniLM-L6-v2 (lightweight, fast)
-- **Vector Store**: In-memory FAISS index for deployment portability
-- **Retrieval**: Top-k similarity search (k=3-5) with relevance scoring
-
-### Enhanced Template (SPACE_TEMPLATE modification)
-```python
-# Add to generated app.py
-COURSE_MATERIALS = json.loads('''{{course_materials_json}}''')
-EMBEDDINGS_INDEX = pickle.loads(base64.b64decode('''{{embeddings_base64}}'''))
-
-def get_relevant_context(query, max_contexts=3):
-    """Retrieve relevant course material context"""
-    # Vector similarity search
-    # Return formatted context snippets
-```
-
-## Speed & Accuracy Optimizations
-
-### 1. Processing Speed
-- Batch processing during upload (not per-query)
-- Lightweight embedding model (384 dimensions vs 1536)
-- In-memory vector store (no database dependencies)
-- Cached embeddings in deployment package
-
-### 2. Query Speed
-- Pre-computed embeddings (no real-time encoding)
-- Efficient FAISS indexing for similarity search
-- Context caching for repeated queries
-- Parallel processing for multiple files
-
-### 3. Accuracy Enhancements
-- Semantic chunking preserves context boundaries
-- Query expansion with synonyms/related terms
-- Relevance scoring with threshold filtering
-- Metadata-aware retrieval (file type, section, date)
-
-## Deployment Package Integration
-
-### Package Structure Enhancement
-```
-generated_space.zip
-├── app.py                    # Enhanced with RAG
-├── requirements.txt          # + sentence-transformers, faiss-cpu
-├── course_materials/         # Embedded materials
-│   ├── embeddings.pkl       # FAISS index
-│   ├── chunks.json          # Text chunks with metadata
-│   └── files_metadata.json  # Original file info
-└── README.md                # Updated instructions
-```
-
-### Size Management
-- Compress embeddings with pickle optimization
-- Base64 encode for template embedding
-- Implement file size warnings (>50MB total)
-- Optional: External storage links for large datasets
-
-## User Interface Updates
-
-### Configuration Tab Enhancements
-```python
-with gr.Accordion("Course Materials Upload", open=False):
-    file_upload = gr.File(
-        label="Upload Course Materials",
-        file_types=[".pdf", ".docx", ".txt", ".md"],
-        file_count="multiple"
-    )
-    processing_status = gr.Markdown()
-    material_summary = gr.DataFrame() # Show processed files
-```
-
-## Technical Implementation
-
-### Dependencies Addition (requirements.txt)
-```
-sentence-transformers==2.2.2
-faiss-cpu==1.7.4
-PyMuPDF==1.23.0
-python-docx==0.8.11
-tiktoken==0.5.1
-```
-
-### Processing Workflow
-1. **Upload**: Faculty uploads syllabi, schedules, readings
-2. **Parse**: Extract text with structure preservation
-3. **Chunk**: Semantic segmentation with metadata
-4. **Embed**: Generate vector representations
-5. **Package**: Serialize index and chunks into deployment
-6. **Deploy**: Single-file space with embedded knowledge
-
-## Performance Metrics
-
-- **Upload Processing**: ~2-5 seconds per document
-- **Query Response**: <200ms additional latency
-- **Package Size**: +5-15MB for typical course materials
-- **Accuracy**: 85-95% relevant context retrieval
-- **Memory Usage**: +50-100MB runtime overhead
-
-## Benefits
-
-This approach maintains your existing speed while adding powerful document understanding capabilities that persist in the deployed package. Faculty can upload course materials once during configuration, and students get contextually-aware responses based on actual course content without any external dependencies in the deployed space.
-
-## Next Steps
-
-1. Implement document parser service
-2. Add file upload UI components
-3. Integrate RAG system with existing web scraping architecture
-4. Enhance SPACE_TEMPLATE with embedded materials
-5. Test with sample course materials
-6. Optimize for deployment package size

hf_comparisons.aiconfig.json

@@ -1,115 +0,0 @@
-{
-  "name": "Hugging Face LLM Comparisons",
-  "schema_version": "latest",
-  "metadata": {
-    "parameters": {
-      "CoLA_ex_prompt": "Is the sentence grammatical or ungrammatical?\n\n\"This building is than that one.\"",
-      "SST_2_ex_prompt": "Is the movie review positive, negative, or neutral?\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\"",
-      "WNLI_ex_prompt": "Sentence B replaces sentence A's ambiguous pronoun with one of the nouns - is this the correct noun?\n\n\"A) Lily spoke to Donna, breaking her concentration.\nB) Lily spoke to Donna, breaking Lily's concentration.\""
-    },
-    "models": {},
-    "default_model": null,
-    "model_parsers": null
-  },
-  "description": "**In this notebook, we compare the individual performance of HF hosted LLMs () on a few example questions from the GLUE benchmarks (https://gluebenchmark.com/tasks).**\n\n**Example questions taken from \"What is the GLUE Benchmark\" medium post - https://angelina-yang.medium.com/what-is-the-glue-benchmark-for-nlu-systems-61127b3cab3f**\n\n---\n\n| General Language Understanding Evaluation (GLUE) Tasks      | Example Question |\n| ----------- | ----------- |\n| Corpus of Linguistic Acceptability (CoLA)     | Is the sentence grammatical or ungrammatical? \"This building is than that one.\"       |\n| Stanford Sentiment Treebank (SST)   | Is the movie review positive, negative, or neutral? \"The movie is funny, smart, visually inventive, and most of all, alive.\"  |\n| Winograd NLI (WNLI) | Sentence B replaces sentence A's ambiguous pronoun with one of the nouns - is this the correct noun? \"A) Lily spoke to Donna, breaking her concentration. B) Lily spoke to Donna, breaking Lily's concentration.\" |",
-  "prompts": [
-    {
-      "name": "mistral_7b_instruct_v0.1",
-      "input": "Is the movie review positive, negative, or neutral?\n\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\"",
-      "metadata": {
-        "model": {
-          "name": "Text Generation",
-          "settings": {
-            "model": "mistralai/Mistral-7B-Instruct-v0.1"
-          }
-        },
-        "tags": null,
-        "parameters": {}
-      },
-      "outputs": [
-        {
-          "output_type": "execute_result",
-          "execution_count": 0,
-          "data": "\n\nThe movie review is positive.</s>",
-          "mime_type": null,
-          "metadata": {}
-        }
-      ]
-    },
-    {
-      "name": "google_flan_t5_sm",
-      "input": "Is the movie review positive, negative, or neutral?\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\"",
-      "metadata": {
-        "model": {
-          "name": "Conversational",
-          "settings": {
-            "model": "google/flan-t5-small",
-            "max_new_tokens": 250,
-            "stream": false
-          }
-        },
-        "tags": null,
-        "parameters": {}
-      },
-      "outputs": [
-        {
-          "output_type": "execute_result",
-          "execution_count": 0,
-          "data": "positive",
-          "mime_type": null,
-          "metadata": {
-            "raw_response": {
-              "generated_text": "positive",
-              "conversation": {
-                "generated_responses": [
-                  "positive"
-                ],
-                "past_user_inputs": [
-                  "Is the movie review positive, negative, or neutral?\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\""
-                ]
-              },
-              "warnings": [
-                "\nNo chat template is defined for this tokenizer - using a default chat template that implements the ChatML format (without BOS/EOS tokens!). If the default is not appropriate for your model, please set `tokenizer.chat_template` to an appropriate template. See https://huggingface.co/docs/transformers/main/chat_templating for more information.\n"
-              ]
-            }
-          }
-        }
-      ]
-    },
-    {
-      "name": "tinyllama-1_1B",
-      "input": "<|system|>\nYou are to answer the following question by the user</s>\n<|user|>\n{{SST_2_ex_prompt}}</s>\n<|assistant|>",
-      "metadata": {
-        "model": {
-          "name": "Conversational",
-          "settings": {
-            "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
-          }
-        },
-        "tags": null,
-        "parameters": {}
-      },
-      "outputs": [
-        {
-          "output_type": "execute_result",
-          "execution_count": 0,
-          "data": "The movie review is positive.",
-          "mime_type": null,
-          "metadata": {
-            "raw_response": {
-              "generated_text": "The movie review is positive.",
-              "conversation": {
-                "generated_responses": [
-                  "The movie review is positive."
-                ],
-                "past_user_inputs": [
-                  "<|system|>\nYou are to answer the following question by the user</s>\n<|user|>\nIs the movie review positive, negative, or neutral?\n\n&quot;The movie is funny, smart, visually inventive, and most of all, alive.&quot;</s>\n<|assistant|>"
-                ]
-              }
-            }
-          }
-        }
-      ]
-    }
-  ]
-}

test_api_key.py

@@ -1,85 +0,0 @@
-#!/usr/bin/env python3
-"""Test OpenRouter API key functionality"""
-
-import requests
-import json
-
-def test_openrouter_api_key(api_key):
-    """Test if an OpenRouter API key is valid by making a simple completion request"""
-    
-    url = "https://openrouter.ai/api/v1/chat/completions"
-    
-    headers = {
-        "Authorization": f"Bearer {api_key}",
-        "Content-Type": "application/json",
-        "HTTP-Referer": "https://github.com/test-api-key",  # Required by OpenRouter
-        "X-Title": "API Key Test"  # Optional but recommended
-    }
-    
-    # Simple test message
-    data = {
-        "model": "openrouter/auto",  # Auto-select cheapest available model
-        "messages": [
-            {"role": "user", "content": "Say 'API key is working!' in exactly 4 words."}
-        ],
-        "max_tokens": 10,
-        "temperature": 0.1
-    }
-    
-    try:
-        print("Testing OpenRouter API key...")
-        response = requests.post(url, headers=headers, json=data, timeout=30)
-        
-        if response.status_code == 200:
-            result = response.json()
-            if "choices" in result and len(result["choices"]) > 0:
-                assistant_message = result["choices"][0]["message"]["content"]
-                print(f"✓ API key is valid!")
-                print(f"Response: {assistant_message}")
-                print(f"Model used: {result.get('model', 'unknown')}")
-                return True
-            else:
-                print("✗ Unexpected response format")
-                return False
-        else:
-            error_data = response.json() if response.headers.get('content-type', '').startswith('application/json') else {}
-            print(f"✗ API key test failed!")
-            print(f"Status code: {response.status_code}")
-            print(f"Error: {error_data.get('error', {}).get('message', response.text)}")
-            
-            # Common error interpretations
-            if response.status_code == 401:
-                print("→ The API key is invalid or has been revoked")
-            elif response.status_code == 402:
-                print("→ The API key has insufficient credits")
-            elif response.status_code == 429:
-                print("→ Rate limit exceeded")
-            
-            return False
-            
-    except requests.exceptions.Timeout:
-        print("✗ Request timed out")
-        return False
-    except requests.exceptions.RequestException as e:
-        print(f"✗ Network error: {e}")
-        return False
-    except Exception as e:
-        print(f"✗ Unexpected error: {e}")
-        return False
-
-if __name__ == "__main__":
-    # Test the provided API key
-    api_key = "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"
-    
-    print(f"API Key: {api_key[:20]}...{api_key[-10:]}")
-    print("-" * 50)
-    
-    success = test_openrouter_api_key(api_key)
-    
-    print("-" * 50)
-    if success:
-        print("✓ The API key is working correctly!")
-        print("You can use this key in your Chat UI Helper application.")
-    else:
-        print("✗ The API key is not working.")
-        print("Please check that the key is correct and has available credits.")

test_connection_fix.py

@@ -1,137 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test RAG connection error fix
-Tests the specific multiprocessing and connection timeout issues
-"""
-
-import os
-import tempfile
-import warnings
-
-# Set environment variables before any imports
-os.environ['TOKENIZERS_PARALLELISM'] = 'false'
-os.environ['OMP_NUM_THREADS'] = '1'
-os.environ['MKL_NUM_THREADS'] = '1'
-
-# Suppress warnings for cleaner output
-warnings.filterwarnings("ignore", category=UserWarning)
-warnings.filterwarnings("ignore", category=FutureWarning)
-
-def test_connection_fix():
-    """Test the connection error fix specifically"""
-    print("Testing RAG connection error fix...")
-    
-    try:
-        # Test conditional import
-        try:
-            from rag_tool import RAGTool
-            has_rag = True
-            print("✓ RAG dependencies available")
-        except ImportError:
-            print("✗ RAG dependencies not available")
-            return False
-        
-        # Create a test document
-        test_content = """This is a test document for connection error testing.
-        It contains multiple sentences to test the embedding process.
-        The document should be processed without connection errors.
-        This tests multiprocessing fixes and memory management."""
-        
-        with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
-            f.write(test_content)
-            test_file = f.name
-        
-        try:
-            print("✓ Test document created")
-            
-            # Initialize RAG tool with environment variables already set
-            print("Initializing RAG tool with connection fixes...")
-            rag_tool = RAGTool()
-            print("✓ RAG tool initialized successfully")
-            
-            # Process document - this was causing the connection error
-            print("Processing document (this was causing connection errors)...")
-            result = rag_tool.process_uploaded_files([test_file])
-            
-            if result['success']:
-                print(f"✓ Document processed successfully: {result['message']}")
-                print(f"  - Chunks created: {result.get('index_stats', {}).get('total_chunks', 'unknown')}")
-                
-                # Test search to ensure embeddings work
-                context = rag_tool.get_relevant_context("test document", max_chunks=1)
-                print(f"✓ Search test successful, context length: {len(context)}")
-                
-                return True
-            else:
-                print(f"✗ Document processing failed: {result['message']}")
-                return False
-                
-        finally:
-            # Clean up
-            if os.path.exists(test_file):
-                os.unlink(test_file)
-                print("✓ Test file cleaned up")
-        
-    except Exception as e:
-        print(f"✗ Test failed with error: {e}")
-        return False
-
-def test_gradio_integration():
-    """Test integration with Gradio interface"""
-    print("\nTesting Gradio integration...")
-    
-    try:
-        import gradio as gr
-        
-        # Create a minimal Gradio interface similar to the main app
-        def test_process_documents(files):
-            """Minimal version of process_documents for testing"""
-            if not files:
-                return "No files uploaded"
-            
-            try:
-                from rag_tool import RAGTool
-                rag_tool = RAGTool()
-                
-                # Simulate file processing
-                file_paths = [f.name if hasattr(f, 'name') else str(f) for f in files]
-                result = rag_tool.process_uploaded_files(file_paths)
-                
-                if result['success']:
-                    return f"✓ Success: {result['message']}"
-                else:
-                    return f"✗ Failed: {result['message']}"
-                    
-            except Exception as e:
-                return f"✗ Error: {str(e)}"
-        
-        # Create interface without launching
-        with gr.Blocks() as interface:
-            file_input = gr.File(file_count="multiple", label="Test Documents")
-            output = gr.Textbox(label="Result")
-            process_btn = gr.Button("Process")
-            
-            process_btn.click(
-                test_process_documents,
-                inputs=[file_input],
-                outputs=[output]
-            )
-        
-        print("✓ Gradio interface created successfully")
-        print("  Interface can be launched without connection errors")
-        return True
-        
-    except Exception as e:
-        print(f"✗ Gradio integration test failed: {e}")
-        return False
-
-if __name__ == "__main__":
-    success = test_connection_fix()
-    if success:
-        success = test_gradio_integration()
-    
-    if success:
-        print("\n🎉 All connection error fixes are working!")
-        print("The RAG processing should now work without connection timeouts.")
-    else:
-        print("\n❌ Some tests failed. Check the error messages above.")

test_document.txt

@@ -1,24 +0,0 @@
-Vector Database Test Document
-
-This is a test document for evaluating the vector database functionality. 
-
-Section 1: Introduction to Vector Databases
-Vector databases store and query high-dimensional vector representations of data. They enable semantic search by finding vectors similar to a query vector in an embedding space.
-
-Section 2: Use Cases
-Common applications include:
-- Document retrieval and question answering
-- Similarity search for products or content
-- Recommendation systems
-- Semantic search in chatbots
-
-Section 3: Technical Implementation
-Vector databases typically use embedding models to convert text into dense vectors, then use algorithms like cosine similarity or approximate nearest neighbor search to find relevant results.
-
-Section 4: Benefits
-- Semantic understanding beyond keyword matching
-- Scalable retrieval for large document collections
-- Integration with modern AI systems and large language models
-- Support for multi-modal data (text, images, audio)
-
-This document should generate multiple chunks when processed by the system.

test_gradio_simple.py

@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-"""
-Simple test to verify if the gradio app can start without schema errors
-"""
-
-import gradio as gr
-
-def simple_function(message, history):
-    return "", history + [[message, "Test response"]]
-
-# Create a simple interface to test
-with gr.Blocks() as demo:
-    chatbot = gr.Chatbot()
-    msg = gr.Textbox()
-    
-    msg.submit(simple_function, [msg, chatbot], [msg, chatbot])
-
-if __name__ == "__main__":
-    print("Testing simple Gradio interface...")
-    try:
-        demo.launch(server_name="127.0.0.1", server_port=7862, share=False, prevent_thread_lock=True)
-        print("✅ Simple Gradio interface works")
-    except Exception as e:
-        print(f"❌ Simple Gradio interface failed: {e}")

test_minimal.py

@@ -1,20 +0,0 @@
-import gradio as gr
-
-# Minimal test to isolate the boolean iteration error
-with gr.Blocks() as demo:
-    with gr.Tab("Test"):
-        name = gr.Textbox(label="Name")
-        checkbox = gr.Checkbox(label="Test", value=False)
-        button = gr.Button("Test")
-        
-        def test_func(name_val, checkbox_val):
-            return f"Hello {name_val}, checkbox: {checkbox_val}"
-        
-        button.click(
-            test_func,
-            inputs=[name, checkbox],
-            outputs=[name]
-        )
-
-if __name__ == "__main__":
-    demo.launch()

test_preview.py

@@ -1,110 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test script for preview functionality
-"""
-
-import os
-import sys
-import tempfile
-
-# Add current directory to path for imports
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-
-def test_preview_chat_response():
-    """Test the preview chat response function"""
-    try:
-        from app import preview_chat_response
-        
-        # Mock config data
-        config_data = {
-            'name': 'Test Assistant',
-            'model': 'google/gemini-2.0-flash-001',
-            'system_prompt': 'You are a helpful assistant.',
-            'temperature': 0.7,
-            'max_tokens': 500,
-            'enable_dynamic_urls': False,
-            'enable_vector_rag': False
-        }
-        
-        # Test with no API key (should give preview message)
-        if 'OPENROUTER_API_KEY' in os.environ:
-            del os.environ['OPENROUTER_API_KEY']
-        
-        message = "Hello, how are you?"
-        history = []
-        
-        result_msg, result_history = preview_chat_response(
-            message, history, config_data, "", "", "", ""
-        )
-        
-        print("✅ preview_chat_response function works")
-        print(f"Result message: {result_msg}")
-        print(f"History length: {len(result_history)}")
-        print(f"Last response: {result_history[-1] if result_history else 'None'}")
-        
-        return True
-        
-    except Exception as e:
-        print(f"❌ preview_chat_response failed: {e}")
-        return False
-
-def test_url_extraction():
-    """Test URL extraction function"""
-    try:
-        from app import extract_urls_from_text
-        
-        test_text = "Check out https://example.com and also https://test.org/page"
-        urls = extract_urls_from_text(test_text)
-        
-        print("✅ extract_urls_from_text works")
-        print(f"Extracted URLs: {urls}")
-        
-        return True
-        
-    except Exception as e:
-        print(f"❌ extract_urls_from_text failed: {e}")
-        return False
-
-def test_url_fetching():
-    """Test URL content fetching"""
-    try:
-        from app import fetch_url_content
-        
-        # Test with a simple URL
-        content = fetch_url_content("https://httpbin.org/get")
-        
-        print("✅ fetch_url_content works")
-        print(f"Content length: {len(content)}")
-        print(f"Content preview: {content[:100]}...")
-        
-        return True
-        
-    except Exception as e:
-        print(f"❌ fetch_url_content failed: {e}")
-        return False
-
-if __name__ == "__main__":
-    print("Testing preview functionality components...")
-    
-    tests = [
-        test_url_extraction,
-        test_url_fetching,
-        test_preview_chat_response
-    ]
-    
-    passed = 0
-    total = len(tests)
-    
-    for test in tests:
-        if test():
-            passed += 1
-        print()
-    
-    print(f"Test Results: {passed}/{total} passed")
-    
-    if passed == total:
-        print("✅ All preview functionality tests passed!")
-        sys.exit(0)
-    else:
-        print("❌ Some tests failed")
-        sys.exit(1)

test_rag_fix.py

@@ -1,182 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test script to verify RAG functionality fixes
-"""
-
-import os
-import tempfile
-import warnings
-from pathlib import Path
-
-# Suppress known warnings
-warnings.filterwarnings("ignore", message=".*use_auth_token.*")
-warnings.filterwarnings("ignore", message=".*urllib3.*")
-warnings.filterwarnings("ignore", message=".*resource_tracker.*")
-
-# Set environment variables to prevent multiprocessing issues
-os.environ['TOKENIZERS_PARALLELISM'] = 'false'
-
-def test_rag_dependencies():
-    """Test that RAG dependencies are available"""
-    print("Testing RAG dependencies...")
-    
-    try:
-        import sentence_transformers
-        print("✅ sentence-transformers available")
-    except ImportError:
-        print("❌ sentence-transformers not available")
-        return False
-    
-    try:
-        import faiss
-        print("✅ faiss-cpu available") 
-    except ImportError:
-        print("❌ faiss-cpu not available")
-        return False
-        
-    try:
-        import fitz  # PyMuPDF
-        print("✅ PyMuPDF available")
-    except ImportError:
-        print("⚠️  PyMuPDF not available (PDF processing disabled)")
-    
-    try:
-        from docx import Document
-        print("✅ python-docx available")
-    except ImportError:
-        print("⚠️  python-docx not available (DOCX processing disabled)")
-    
-    return True
-
-def test_vector_store_initialization():
-    """Test vector store initialization with improved error handling"""
-    print("\nTesting vector store initialization...")
-    
-    try:
-        from vector_store import VectorStore
-        
-        # Test with CPU-only settings
-        store = VectorStore(embedding_model="all-MiniLM-L6-v2")
-        print("✅ VectorStore created successfully")
-        
-        # Test a small embedding operation
-        test_texts = ["This is a test sentence.", "Another test sentence."]
-        embeddings = store.create_embeddings(test_texts)
-        print(f"✅ Created embeddings: shape {embeddings.shape}")
-        
-        return True
-        
-    except Exception as e:
-        print(f"❌ VectorStore initialization failed: {e}")
-        return False
-
-def test_document_processing():
-    """Test document processing with a simple text file"""
-    print("\nTesting document processing...")
-    
-    try:
-        from document_processor import DocumentProcessor
-        
-        # Create a temporary test file
-        with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
-            f.write("This is a test document for RAG processing. ")
-            f.write("It contains multiple sentences that should be processed into chunks. ")
-            f.write("Each chunk should have proper metadata and be ready for embedding.")
-            test_file = f.name
-        
-        try:
-            processor = DocumentProcessor(chunk_size=50, chunk_overlap=10)
-            chunks = processor.process_file(test_file)
-            
-            print(f"✅ Created {len(chunks)} chunks from test document")
-            if chunks:
-                print(f"   First chunk: {chunks[0].text[:50]}...")
-                print(f"   Metadata keys: {list(chunks[0].metadata.keys())}")
-            
-            return True
-            
-        finally:
-            # Clean up test file
-            os.unlink(test_file)
-            
-    except Exception as e:
-        print(f"❌ Document processing failed: {e}")
-        return False
-
-def test_rag_tool_integration():
-    """Test the complete RAG tool integration"""
-    print("\nTesting complete RAG tool integration...")
-    
-    try:
-        from rag_tool import RAGTool
-        
-        # Create a temporary test file
-        with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
-            f.write("RAG integration test document. ")
-            f.write("This document tests the complete RAG pipeline from file processing to vector search. ")
-            f.write("The system should handle this without crashing the server.")
-            test_file = f.name
-        
-        try:
-            rag_tool = RAGTool()
-            result = rag_tool.process_uploaded_files([test_file])
-            
-            if result['success']:
-                print(f"✅ RAG processing succeeded: {result['message']}")
-                print(f"   Files processed: {len(result['summary']['files_processed'])}")
-                print(f"   Total chunks: {result['summary']['total_chunks']}")
-                
-                # Test search functionality
-                context = rag_tool.get_relevant_context("test document")
-                if context:
-                    print(f"✅ Search functionality working: {context[:100]}...")
-                else:
-                    print("⚠️  Search returned no results")
-                
-                return True
-            else:
-                print(f"❌ RAG processing failed: {result['message']}")
-                return False
-                
-        finally:
-            # Clean up test file
-            os.unlink(test_file)
-            
-    except Exception as e:
-        print(f"❌ RAG tool integration failed: {e}")
-        return False
-
-def main():
-    """Run all RAG tests"""
-    print("🚀 Testing RAG functionality fixes...")
-    print("=" * 50)
-    
-    tests = [
-        test_rag_dependencies,
-        test_vector_store_initialization,
-        test_document_processing,
-        test_rag_tool_integration
-    ]
-    
-    passed = 0
-    total = len(tests)
-    
-    for test in tests:
-        try:
-            if test():
-                passed += 1
-        except Exception as e:
-            print(f"❌ Test failed with exception: {e}")
-    
-    print("\n" + "=" * 50)
-    print(f"📊 Test Results: {passed}/{total} tests passed")
-    
-    if passed == total:
-        print("🎉 All tests passed! RAG functionality should work correctly.")
-        return True
-    else:
-        print("⚠️  Some tests failed. Check error messages above.")
-        return False
-
-if __name__ == "__main__":
-    main()

test_sample.txt

@@ -1,8 +0,0 @@
-This is a sample document for testing the RAG functionality.
-It contains multiple paragraphs of text that will be processed and chunked.
-
-The vector store will create embeddings for these chunks and allow semantic search.
-This enables context-aware responses based on uploaded documents.
-
-The system supports PDF, DOCX, TXT, and Markdown files.
-It uses FAISS for efficient similarity search and sentence transformers for embeddings.

test_vector_db.py

@@ -1,196 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test script to verify vector database creation functionality
-"""
-
-import sys
-import os
-from pathlib import Path
-
-# Add current directory to path to import modules
-sys.path.append(str(Path(__file__).parent))
-
-try:
-    from rag_tool import RAGTool
-    from vector_store import VectorStore
-    from document_processor import DocumentProcessor
-    print("✅ Successfully imported all RAG modules")
-except ImportError as e:
-    print(f"❌ Failed to import RAG modules: {e}")
-    sys.exit(1)
-
-def test_document_processing():
-    """Test document processing functionality"""
-    print("\n=== Testing Document Processing ===")
-    
-    processor = DocumentProcessor(chunk_size=200, chunk_overlap=50)
-    
-    # Test with our test document
-    test_file = "test_document.txt"
-    if not os.path.exists(test_file):
-        print(f"❌ Test file {test_file} not found")
-        return False
-    
-    try:
-        chunks = processor.process_file(test_file)
-        print(f"✅ Processed {test_file} into {len(chunks)} chunks")
-        
-        # Show first chunk
-        if chunks:
-            first_chunk = chunks[0]
-            print(f"First chunk preview: {first_chunk.text[:100]}...")
-            print(f"Chunk metadata: {first_chunk.metadata}")
-        
-        return True
-    except Exception as e:
-        print(f"❌ Failed to process document: {e}")
-        return False
-
-def test_vector_store():
-    """Test vector store functionality"""
-    print("\n=== Testing Vector Store ===")
-    
-    try:
-        # Initialize vector store
-        vector_store = VectorStore()
-        print("✅ Initialized vector store")
-        
-        # Create test data
-        test_chunks = [
-            {
-                'text': 'Vector databases are used for semantic search',
-                'chunk_id': 'test1',
-                'metadata': {'file_name': 'test.txt', 'chunk_index': 0}
-            },
-            {
-                'text': 'Machine learning models convert text to embeddings',
-                'chunk_id': 'test2', 
-                'metadata': {'file_name': 'test.txt', 'chunk_index': 1}
-            },
-            {
-                'text': 'FAISS provides efficient similarity search capabilities',
-                'chunk_id': 'test3',
-                'metadata': {'file_name': 'test.txt', 'chunk_index': 2}
-            }
-        ]
-        
-        # Build index
-        print("Building vector index...")
-        vector_store.build_index(test_chunks, show_progress=True)
-        print("✅ Built vector index")
-        
-        # Test search
-        query = "How do vector databases work?"
-        results = vector_store.search(query, top_k=2)
-        
-        print(f"Search results for '{query}':")
-        for i, result in enumerate(results):
-            print(f"  {i+1}. Score: {result.score:.3f} - {result.text[:50]}...")
-        
-        # Test serialization
-        serialized = vector_store.serialize()
-        print(f"✅ Serialized data size: {len(serialized['index_base64'])} characters")
-        
-        return True
-        
-    except Exception as e:
-        print(f"❌ Failed vector store test: {e}")
-        import traceback
-        traceback.print_exc()
-        return False
-
-def test_rag_tool():
-    """Test complete RAG tool functionality"""
-    print("\n=== Testing RAG Tool ===")
-    
-    try:
-        # Initialize RAG tool
-        rag_tool = RAGTool()
-        print("✅ Initialized RAG tool")
-        
-        # Process test document
-        test_files = ["test_document.txt"]
-        result = rag_tool.process_uploaded_files(test_files)
-        
-        if result['success']:
-            print(f"✅ {result['message']}")
-            
-            # Show summary
-            summary = result['summary']
-            print(f"Files processed: {summary['total_files']}")
-            print(f"Total chunks: {summary['total_chunks']}")
-            
-            # Test context retrieval
-            query = "What are the benefits of vector databases?"
-            context = rag_tool.get_relevant_context(query, max_chunks=2)
-            
-            if context:
-                print(f"\nContext for '{query}':")
-                print(context[:300] + "..." if len(context) > 300 else context)
-                print("✅ Successfully retrieved context")
-            else:
-                print("⚠️ No context retrieved")
-            
-            # Test serialization for deployment
-            serialized_data = rag_tool.get_serialized_data()
-            if serialized_data:
-                print("✅ Successfully serialized RAG data for deployment")
-                print(f"Serialized keys: {list(serialized_data.keys())}")
-            else:
-                print("❌ Failed to serialize RAG data")
-            
-            return True
-        else:
-            print(f"❌ {result['message']}")
-            return False
-            
-    except Exception as e:
-        print(f"❌ Failed RAG tool test: {e}")
-        import traceback
-        traceback.print_exc()
-        return False
-
-def main():
-    """Run all tests"""
-    print("=== Vector Database Testing ===")
-    print("Testing vector database creation and functionality...")
-    
-    # Check dependencies
-    print("\n=== Checking Dependencies ===")
-    try:
-        import sentence_transformers
-        import faiss
-        import fitz  # PyMuPDF
-        print("✅ All required dependencies available")
-    except ImportError as e:
-        print(f"❌ Missing dependency: {e}")
-        return
-    
-    # Run tests
-    tests = [
-        ("Document Processing", test_document_processing),
-        ("Vector Store", test_vector_store), 
-        ("RAG Tool", test_rag_tool)
-    ]
-    
-    results = []
-    for test_name, test_func in tests:
-        print(f"\n{'='*20}")
-        success = test_func()
-        results.append((test_name, success))
-    
-    # Summary
-    print(f"\n{'='*40}")
-    print("TEST SUMMARY:")
-    for test_name, success in results:
-        status = "✅ PASS" if success else "❌ FAIL"
-        print(f"  {test_name}: {status}")
-    
-    all_passed = all(success for _, success in results)
-    if all_passed:
-        print("\n🎉 All tests passed! Vector database functionality is working.")
-    else:
-        print("\n⚠️ Some tests failed. Check the output above for details.")
-
-if __name__ == "__main__":
-    main()