Optimize v2-search with enhanced URL processing and search validation

Features added:
- Enhanced URL extraction with validation and cleanup
- Improved URL content fetching with smart truncation
- Enhanced content cleaning and main content extraction
- Backward compatibility maintained
- Fixed broken research template references

CLAUDE.md

@@ -1,217 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-Chat UI Helper is a Gradio-based tool for generating and configuring chat interfaces for HuggingFace Spaces. It creates deployable packages with custom assistants, web scraping capabilities, and optional vector RAG functionality.
-
-## Core Architecture
-
-### Main Application (`app.py`)
-- **Primary Interface**: Two-tab Gradio application - "Spaces Configuration" for generating chat interfaces and "Chat Support" for getting help
-- **Template System**: `SPACE_TEMPLATE` generates complete HuggingFace Space apps with embedded configuration
-- **Web Scraping**: Simple HTTP requests with BeautifulSoup for URL content fetching and context grounding
-- **Vector RAG**: Optional document processing pipeline for course materials and knowledge bases
-
-### Document Processing Pipeline
-- **RAGTool** (`rag_tool.py`): Main orchestrator for document upload and processing
-- **DocumentProcessor** (`document_processor.py`): Handles PDF, DOCX, TXT, MD file parsing and chunking
-- **VectorStore** (`vector_store.py`): FAISS-based similarity search and embedding management
-
-### Package Generation
-The tool generates complete HuggingFace Spaces with:
-- `app.py`: Chat interface with OpenRouter API integration
-- `requirements.txt`: Gradio 5.x and dependencies 
-- `README.md`: Deployment instructions with security setup
-- `config.json`: Configuration backup
-- Optional embedded RAG data for document-aware responses
-
-## Development Commands
-
-### Running the Application
-```bash
-python app.py
-```
-
-### Testing Vector Database Functionality
-```bash
-python test_vector_db.py
-```
-
-### Testing Individual Components
-```bash
-# Test document processing only
-python -c "from test_vector_db import test_document_processing; test_document_processing()"
-
-# Test vector store only
-python -c "from test_vector_db import test_vector_store; test_vector_store()"
-
-# Test full RAG pipeline
-python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
-```
-
-### Dependencies Management
-```bash
-pip install -r requirements.txt
-```
-
-### Key Dependencies
-- **Gradio 5.35.0+**: Main UI framework
-- **requests**: HTTP requests for web content fetching
-- **sentence-transformers**: Embeddings for RAG (optional)
-- **faiss-cpu**: Vector similarity search (optional)
-- **PyMuPDF**: PDF text extraction (optional)
-- **python-docx**: DOCX document processing (optional)
-- **beautifulsoup4**: HTML parsing for web scraping
-- **python-dotenv**: Environment variable management
-
-## Configuration Patterns
-
-### Template Variables
-Generated spaces use these template substitutions:
-- `{name}`, `{description}`: Basic space metadata
-- `{system_prompt}`: Combined assistant configuration
-- `{model}`: OpenRouter model selection
-- `{grounding_urls}`: Static URL list for context
-- `{enable_dynamic_urls}`: Runtime URL fetching capability
-- `{enable_vector_rag}`: Document search integration
-- `{rag_data_json}`: Serialized embeddings and chunks
-
-### Access Control
-- Environment variable `SPACE_ACCESS_CODE` for student access control
-- Global state management for session-based access in generated spaces
-- Security-first approach storing credentials as HuggingFace Spaces secrets
-
-### RAG Integration
-- Modular design with optional imports (`HAS_RAG` flag in app.py:23)
-- FAISS index serialization for deployment portability
-- 10MB file size limits with validation
-- Semantic chunking (800 chars, 100 overlap) for optimal retrieval
-- Graceful degradation when vector dependencies unavailable
-
-## Architecture Notes
-
-### State Management
-- Extensive use of `gr.State()` for maintaining session data
-- Global variables for access control in generated templates
-- URL content caching to prevent redundant web requests
-
-### Template Generation Pattern
-All generated HuggingFace Spaces follow consistent structure:
-1. Configuration section with environment variable loading
-2. Web scraping functions (simple HTTP requests with BeautifulSoup)  
-3. RAG context retrieval (if enabled)
-4. OpenRouter API integration with conversation history
-5. Gradio ChatInterface with access control
-
-### Error Handling
-- Graceful degradation when optional dependencies unavailable
-- Comprehensive validation for file uploads and URL processing
-- User-friendly error messages with specific guidance
-
-### Security Considerations
-- Never embed API keys or access codes in generated templates
-- Environment variable pattern for all sensitive configuration
-- Input validation for uploaded files and URL processing
-- Content length limits for web scraping operations
-
-### Dependency Management Pattern
-The codebase uses conditional imports with feature flags:
-```python
-try:
-    from rag_tool import RAGTool
-    HAS_RAG = True
-except ImportError:
-    HAS_RAG = False
-    RAGTool = None
-```
-This pattern allows the main application to function even when optional vector database dependencies are unavailable.
-
-## Important Implementation Details
-
-### Gradio 5.x Compatibility
-- Uses `type="messages"` for chat history format
-- `gr.ChatInterface` for modern chat UI components
-- Proper message format handling for OpenRouter API
-
-### Dynamic URL Fetching
-When enabled, generated spaces can extract URLs from user messages and fetch content dynamically using regex pattern matching and simple HTTP requests with BeautifulSoup.
-
-### Vector RAG Workflow
-1. Documents uploaded through Gradio File component
-2. Processed via DocumentProcessor (PDF/DOCX/TXT/MD support)
-3. Chunked and embedded using sentence-transformers
-4. FAISS index created and serialized to base64
-5. Embedded in generated template for deployment portability
-6. Runtime similarity search for context-aware responses
-
-### Web Scraping Implementation
-The application uses simple HTTP requests with BeautifulSoup for web content extraction:
-- **Simple HTTP requests**: Uses `requests` library with timeout and user-agent headers
-- **Content extraction**: BeautifulSoup for HTML parsing and text extraction
-- **Content cleaning**: Removes scripts, styles, navigation elements and normalizes whitespace
-- **Content limits**: Truncates content to ~4000 characters for context management
-
-## Testing and Quality Assurance
-
-### Comprehensive Test Procedure
-The project includes a comprehensive test procedure documented in `TEST_PROCEDURE.md` that covers:
-
-#### Test Categories
-1. **Core Application Tests**: Startup validation and Gradio interface testing
-2. **Vector RAG Component Tests**: Document processing, vector store, and RAG pipeline validation
-3. **Space Generation Tests**: Template creation and file generation verification
-4. **Web Scraping Tests**: Mock vs production mode validation and URL processing
-5. **Security and Configuration Tests**: Environment variable handling and input validation
-6. **Chat Support Tests**: OpenRouter integration and Gradio 5.x compatibility
-
-#### Automated Testing Commands
-```bash
-# Quick test suite for essential validation
-./quick_test.sh
-
-# Full comprehensive test suite
-./full_test.sh
-
-# Individual component testing
-python test_vector_db.py
-python -c "from test_vector_db import test_document_processing; test_document_processing()"
-python -c "from test_vector_db import test_vector_store; test_vector_store()"
-python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
-```
-
-#### Pre-Test Setup
-```bash
-# Environment verification
-python --version  # Should be 3.8+
-pip install -r requirements.txt
-
-# Test data preparation
-echo "This is a test document for RAG functionality testing." > test_document.txt
-mkdir -p test_outputs
-```
-
-#### Regression Testing
-After each commit, verify:
-- All existing functionality still works
-- New features don't break existing features
-- Generated spaces deploy successfully to HuggingFace
-- Documentation is updated appropriately
-- Dependencies are correctly specified
-- Security patterns are maintained
-
-#### Performance Benchmarking
-```bash
-# Startup time measurement
-time python -c "import app; print('App loaded')"
-
-# Space generation time
-time python -c "import app; app.generate_zip('Benchmark', 'Test', 'Role', 'Audience', 'Tasks', '', [], '', '', 'gpt-3.5-turbo', 0.7, 4000, [], False, False, None)"
-
-# RAG processing time
-time python -c "from test_vector_db import test_rag_tool; test_rag_tool()"
-```
-
-#### Continuous Integration
-The test procedure is designed to integrate with GitHub Actions for automated testing on commits and pull requests. See `TEST_PROCEDURE.md` for complete setup instructions and CI configuration.

devjournal.md

@@ -0,0 +1,5 @@
+# Dev Journal - ChatUI Helper
+
+system prompts:
+
+- You are blah. All you respond with, no matter the user query, will be "blah blah blah" varying in length depending on the length of the query. Respond only with blah blah. Nothing else. No other words.

gradio_docs.py

@@ -0,0 +1,109 @@
+"""
+Gradio Documentation MCP Server Integration
+
+DEVELOPMENT ONLY - DO NOT IMPORT IN MAIN APPLICATION
+
+This module provides access to live Gradio documentation via MCP server
+for local development assistance. It should NOT be imported in app.py or 
+support_docs.py as it's not needed for the deployed application.
+
+Usage:
+    python -c "from gradio_docs import gradio_docs; print(gradio_docs.search_docs('ChatInterface'))"
+"""
+
+import requests
+import json
+from typing import Optional, Dict, Any
+
+class GradioDocsClient:
+    """Client for accessing Gradio documentation via MCP server"""
+    
+    def __init__(self, base_url: str = "https://gradio-docs-mcp.hf.space"):
+        self.base_url = base_url.rstrip('/')
+        self.session = requests.Session()
+        self.session.headers.update({
+            'User-Agent': 'ChatUI-Helper/1.0'
+        })
+    
+    def search_docs(self, query: str, limit: int = 5) -> Optional[Dict[str, Any]]:
+        """
+        Search Gradio documentation for relevant information
+        
+        Args:
+            query: Search query string
+            limit: Maximum number of results to return
+            
+        Returns:
+            Dictionary with search results or None if error
+        """
+        try:
+            # Try to use the API endpoint for search
+            search_url = f"{self.base_url}/gradio_api"
+            
+            response = self.session.post(
+                search_url,
+                json={"query": query, "limit": limit},
+                timeout=10
+            )
+            
+            if response.status_code == 200:
+                return response.json()
+            else:
+                print(f"Gradio docs search failed: {response.status_code}")
+                return None
+                
+        except requests.RequestException as e:
+            print(f"Error connecting to Gradio docs server: {e}")
+            return None
+    
+    def get_component_info(self, component_name: str) -> Optional[str]:
+        """
+        Get information about a specific Gradio component
+        
+        Args:
+            component_name: Name of the Gradio component (e.g., 'ChatInterface', 'Textbox')
+            
+        Returns:
+            Documentation string or None if not found
+        """
+        result = self.search_docs(f"gr.{component_name}")
+        if result and "results" in result:
+            # Extract relevant documentation from results
+            docs = []
+            for item in result["results"][:3]:  # Top 3 results
+                if "content" in item:
+                    docs.append(item["content"])
+            return "\n\n".join(docs) if docs else None
+        return None
+    
+    def get_latest_changes(self) -> Optional[str]:
+        """
+        Get information about latest Gradio changes and updates
+        
+        Returns:
+            Latest changes information or None if not available
+        """
+        result = self.search_docs("changelog updates latest version")
+        if result and "results" in result:
+            changes = []
+            for item in result["results"][:2]:  # Top 2 results
+                if "content" in item:
+                    changes.append(item["content"])
+            return "\n\n".join(changes) if changes else None
+        return None
+    
+    def is_available(self) -> bool:
+        """
+        Check if the Gradio docs server is available
+        
+        Returns:
+            True if server is accessible, False otherwise
+        """
+        try:
+            response = self.session.get(self.base_url, timeout=5)
+            return response.status_code == 200
+        except requests.RequestException:
+            return False
+
+# Global instance for easy access
+gradio_docs = GradioDocsClient()

hf_comparisons.aiconfig.json

@@ -0,0 +1,115 @@
+{
+  "name": "Hugging Face LLM Comparisons",
+  "schema_version": "latest",
+  "metadata": {
+    "parameters": {
+      "CoLA_ex_prompt": "Is the sentence grammatical or ungrammatical?\n\n\"This building is than that one.\"",
+      "SST_2_ex_prompt": "Is the movie review positive, negative, or neutral?\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\"",
+      "WNLI_ex_prompt": "Sentence B replaces sentence A's ambiguous pronoun with one of the nouns - is this the correct noun?\n\n\"A) Lily spoke to Donna, breaking her concentration.\nB) Lily spoke to Donna, breaking Lily's concentration.\""
+    },
+    "models": {},
+    "default_model": null,
+    "model_parsers": null
+  },
+  "description": "**In this notebook, we compare the individual performance of HF hosted LLMs () on a few example questions from the GLUE benchmarks (https://gluebenchmark.com/tasks).**\n\n**Example questions taken from \"What is the GLUE Benchmark\" medium post - https://angelina-yang.medium.com/what-is-the-glue-benchmark-for-nlu-systems-61127b3cab3f**\n\n---\n\n| General Language Understanding Evaluation (GLUE) Tasks      | Example Question |\n| ----------- | ----------- |\n| Corpus of Linguistic Acceptability (CoLA)     | Is the sentence grammatical or ungrammatical? \"This building is than that one.\"       |\n| Stanford Sentiment Treebank (SST)   | Is the movie review positive, negative, or neutral? \"The movie is funny, smart, visually inventive, and most of all, alive.\"  |\n| Winograd NLI (WNLI) | Sentence B replaces sentence A's ambiguous pronoun with one of the nouns - is this the correct noun? \"A) Lily spoke to Donna, breaking her concentration. B) Lily spoke to Donna, breaking Lily's concentration.\" |",
+  "prompts": [
+    {
+      "name": "mistral_7b_instruct_v0.1",
+      "input": "Is the movie review positive, negative, or neutral?\n\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\"",
+      "metadata": {
+        "model": {
+          "name": "Text Generation",
+          "settings": {
+            "model": "mistralai/Mistral-7B-Instruct-v0.1"
+          }
+        },
+        "tags": null,
+        "parameters": {}
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "execution_count": 0,
+          "data": "\n\nThe movie review is positive.</s>",
+          "mime_type": null,
+          "metadata": {}
+        }
+      ]
+    },
+    {
+      "name": "google_flan_t5_sm",
+      "input": "Is the movie review positive, negative, or neutral?\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\"",
+      "metadata": {
+        "model": {
+          "name": "Conversational",
+          "settings": {
+            "model": "google/flan-t5-small",
+            "max_new_tokens": 250,
+            "stream": false
+          }
+        },
+        "tags": null,
+        "parameters": {}
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "execution_count": 0,
+          "data": "positive",
+          "mime_type": null,
+          "metadata": {
+            "raw_response": {
+              "generated_text": "positive",
+              "conversation": {
+                "generated_responses": [
+                  "positive"
+                ],
+                "past_user_inputs": [
+                  "Is the movie review positive, negative, or neutral?\n\n\"The movie is funny, smart, visually inventive, and most of all, alive.\""
+                ]
+              },
+              "warnings": [
+                "\nNo chat template is defined for this tokenizer - using a default chat template that implements the ChatML format (without BOS/EOS tokens!). If the default is not appropriate for your model, please set `tokenizer.chat_template` to an appropriate template. See https://huggingface.co/docs/transformers/main/chat_templating for more information.\n"
+              ]
+            }
+          }
+        }
+      ]
+    },
+    {
+      "name": "tinyllama-1_1B",
+      "input": "<|system|>\nYou are to answer the following question by the user</s>\n<|user|>\n{{SST_2_ex_prompt}}</s>\n<|assistant|>",
+      "metadata": {
+        "model": {
+          "name": "Conversational",
+          "settings": {
+            "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
+          }
+        },
+        "tags": null,
+        "parameters": {}
+      },
+      "outputs": [
+        {
+          "output_type": "execute_result",
+          "execution_count": 0,
+          "data": "The movie review is positive.",
+          "mime_type": null,
+          "metadata": {
+            "raw_response": {
+              "generated_text": "The movie review is positive.",
+              "conversation": {
+                "generated_responses": [
+                  "The movie review is positive."
+                ],
+                "past_user_inputs": [
+                  "<|system|>\nYou are to answer the following question by the user</s>\n<|user|>\nIs the movie review positive, negative, or neutral?\n\n&quot;The movie is funny, smart, visually inventive, and most of all, alive.&quot;</s>\n<|assistant|>"
+                ]
+              }
+            }
+          }
+        }
+      ]
+    }
+  ]
+}

requirements.txt

@@ -1,4 +1,4 @@
-gradio==4.19.2
+gradio>=4.44.1
 requests>=2.32.3
 beautifulsoup4>=4.12.3
 python-dotenv>=1.0.0

support_docs.py

@@ -5,6 +5,7 @@ Support documentation module with accordion-style help sections
 import gradio as gr
 from datetime import datetime
 
+
 def create_support_docs():
     """Create the support documentation interface with accordion menus"""
     
@@ -224,9 +225,34 @@ def create_support_docs():
             **2. Add Your OpenRouter API Key**
             Since you already have an OpenRouter API key, add it as a secret:
             - Go to Settings → Variables and secrets → New secret
-            - Use the interface shown below to add your key:
+            - Name: `OPENROUTER_API_KEY` (or your chosen variable name)
+            - Value: Your OpenRouter API key (starts with `sk-or-`)
+            - Click "Add" to save the secret
+            
+            The secret configuration interface looks like this:
+            """)
             
-            ![HuggingFace Secret Configuration](./secret.png)
+            # Add the secret configuration image (flush left)
+            with gr.Row():
+                with gr.Column(scale=1):
+                    gr.Image(
+                        value="https://drive.google.com/uc?export=view&id=1z67BZrYlJkpvHJ0Dp6vvIWAkN2iwUxCv",
+                        label="HuggingFace Secret Configuration Interface",
+                        show_label=False,
+                        interactive=False,
+                        width=400,
+                        height=300,
+                        container=False
+                    )
+                with gr.Column(scale=2):
+                    pass  # Empty column to push image left
+            
+            gr.Markdown("""
+            *If the image above doesn't load, the interface shows:*
+            - **Name**: `OPENROUTER_API_KEY` (your API key variable name)
+            - **Description**: Optional description
+            - **Value**: Your OpenRouter API key (starts with `sk-or-`)
+            - **Save** button to store the secret
             
             **3. Optional: Add Access Code**
             If you configured student access protection:

test_api_key.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+"""Test OpenRouter API key functionality"""
+
+import requests
+import json
+
+def test_openrouter_api_key(api_key):
+    """Test if an OpenRouter API key is valid by making a simple completion request"""
+    
+    url = "https://openrouter.ai/api/v1/chat/completions"
+    
+    headers = {
+        "Authorization": f"Bearer {api_key}",
+        "Content-Type": "application/json",
+        "HTTP-Referer": "https://github.com/test-api-key",  # Required by OpenRouter
+        "X-Title": "API Key Test"  # Optional but recommended
+    }
+    
+    # Simple test message
+    data = {
+        "model": "openrouter/auto",  # Auto-select cheapest available model
+        "messages": [
+            {"role": "user", "content": "Say 'API key is working!' in exactly 4 words."}
+        ],
+        "max_tokens": 10,
+        "temperature": 0.1
+    }
+    
+    try:
+        print("Testing OpenRouter API key...")
+        response = requests.post(url, headers=headers, json=data, timeout=30)
+        
+        if response.status_code == 200:
+            result = response.json()
+            if "choices" in result and len(result["choices"]) > 0:
+                assistant_message = result["choices"][0]["message"]["content"]
+                print(f"✓ API key is valid!")
+                print(f"Response: {assistant_message}")
+                print(f"Model used: {result.get('model', 'unknown')}")
+                return True
+            else:
+                print("✗ Unexpected response format")
+                return False
+        else:
+            error_data = response.json() if response.headers.get('content-type', '').startswith('application/json') else {}
+            print(f"✗ API key test failed!")
+            print(f"Status code: {response.status_code}")
+            print(f"Error: {error_data.get('error', {}).get('message', response.text)}")
+            
+            # Common error interpretations
+            if response.status_code == 401:
+                print("→ The API key is invalid or has been revoked")
+            elif response.status_code == 402:
+                print("→ The API key has insufficient credits")
+            elif response.status_code == 429:
+                print("→ Rate limit exceeded")
+            
+            return False
+            
+    except requests.exceptions.Timeout:
+        print("✗ Request timed out")
+        return False
+    except requests.exceptions.RequestException as e:
+        print(f"✗ Network error: {e}")
+        return False
+    except Exception as e:
+        print(f"✗ Unexpected error: {e}")
+        return False
+
+if __name__ == "__main__":
+    # Test the provided API key
+    api_key = "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"
+    
+    print(f"API Key: {api_key[:20]}...{api_key[-10:]}")
+    print("-" * 50)
+    
+    success = test_openrouter_api_key(api_key)
+    
+    print("-" * 50)
+    if success:
+        print("✓ The API key is working correctly!")
+        print("You can use this key in your Chat UI Helper application.")
+    else:
+        print("✗ The API key is not working.")
+        print("Please check that the key is correct and has available credits.")

test_gradio_simple.py

@@ -0,0 +1,24 @@
+#!/usr/bin/env python3
+"""
+Simple test to verify if the gradio app can start without schema errors
+"""
+
+import gradio as gr
+
+def simple_function(message, history):
+    return "", history + [[message, "Test response"]]
+
+# Create a simple interface to test
+with gr.Blocks() as demo:
+    chatbot = gr.Chatbot()
+    msg = gr.Textbox()
+    
+    msg.submit(simple_function, [msg, chatbot], [msg, chatbot])
+
+if __name__ == "__main__":
+    print("Testing simple Gradio interface...")
+    try:
+        demo.launch(server_name="127.0.0.1", server_port=7862, share=False, prevent_thread_lock=True)
+        print("✅ Simple Gradio interface works")
+    except Exception as e:
+        print(f"❌ Simple Gradio interface failed: {e}")

test_preview.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""
+Test script for preview functionality
+"""
+
+import os
+import sys
+import tempfile
+
+# Add current directory to path for imports
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+def test_preview_chat_response():
+    """Test the preview chat response function"""
+    try:
+        from app import preview_chat_response
+        
+        # Mock config data
+        config_data = {
+            'name': 'Test Assistant',
+            'model': 'google/gemini-2.0-flash-001',
+            'system_prompt': 'You are a helpful assistant.',
+            'temperature': 0.7,
+            'max_tokens': 500,
+            'enable_dynamic_urls': False,
+            'enable_vector_rag': False
+        }
+        
+        # Test with no API key (should give preview message)
+        if 'OPENROUTER_API_KEY' in os.environ:
+            del os.environ['OPENROUTER_API_KEY']
+        
+        message = "Hello, how are you?"
+        history = []
+        
+        result_msg, result_history = preview_chat_response(
+            message, history, config_data, "", "", "", ""
+        )
+        
+        print("✅ preview_chat_response function works")
+        print(f"Result message: {result_msg}")
+        print(f"History length: {len(result_history)}")
+        print(f"Last response: {result_history[-1] if result_history else 'None'}")
+        
+        return True
+        
+    except Exception as e:
+        print(f"❌ preview_chat_response failed: {e}")
+        return False
+
+def test_url_extraction():
+    """Test URL extraction function"""
+    try:
+        from app import extract_urls_from_text
+        
+        test_text = "Check out https://example.com and also https://test.org/page"
+        urls = extract_urls_from_text(test_text)
+        
+        print("✅ extract_urls_from_text works")
+        print(f"Extracted URLs: {urls}")
+        
+        return True
+        
+    except Exception as e:
+        print(f"❌ extract_urls_from_text failed: {e}")
+        return False
+
+def test_url_fetching():
+    """Test URL content fetching"""
+    try:
+        from app import fetch_url_content
+        
+        # Test with a simple URL
+        content = fetch_url_content("https://httpbin.org/get")
+        
+        print("✅ fetch_url_content works")
+        print(f"Content length: {len(content)}")
+        print(f"Content preview: {content[:100]}...")
+        
+        return True
+        
+    except Exception as e:
+        print(f"❌ fetch_url_content failed: {e}")
+        return False
+
+if __name__ == "__main__":
+    print("Testing preview functionality components...")
+    
+    tests = [
+        test_url_extraction,
+        test_url_fetching,
+        test_preview_chat_response
+    ]
+    
+    passed = 0
+    total = len(tests)
+    
+    for test in tests:
+        if test():
+            passed += 1
+        print()
+    
+    print(f"Test Results: {passed}/{total} passed")
+    
+    if passed == total:
+        print("✅ All preview functionality tests passed!")
+        sys.exit(0)
+    else:
+        print("❌ Some tests failed")
+        sys.exit(1)