fix cline

.clinerules/activeContext.md

@@ -1,31 +0,0 @@
-# Current Work Focus
-
-Refining image preprocessing pipelines to better balance cleaning and preservation of fine details (especially for handwritten inputs)
-
-Improving document type detection accuracy to feed better prompts to the OCR system
-
-Enhancing structured output schemas to cover additional types like travel logs and scientific diagrams
-
-Recent Changes
-
-Implemented more document-type-specific preprocessing pipelines
-
-Switched default OCR engine to Mistral for both printed and handwritten material
-
-Modularized utility functions across utils.py, ocr_utils.py, and newly proposed submodules
-
-Active Decisions and Considerations
-
-Whether to expose preprocessing options to end users (e.g., deskew threshold)
-
-Whether to allow fallback to local Tesseract OCR for offline cases
-
-Determining best practices for handling multi-page PDFs with mixed layouts
-
-Important Patterns and Learnings
-
-Document type detection greatly improves OCR quality when tuned per-class
-
-Over-aggressive preprocessing can erase faint handwriting; thresholds must be conservative for historical artifacts
-
-Keeping preprocessing modular enables rapid experimentation and tuning.

.clinerules/apiDocumentation.md

@@ -1,29 +0,0 @@
-apiDocumentation.md
-API Interaction Documentation
-Mistral OCR API
-
-    Endpoint: /v1/ocr
-
-    Payload:
-
-        image (binary)
-
-        prompt (optional contextual instructions)
-
-    Response:
-
-        structured_data: Hierarchical text + metadata output
-
-        raw_text: Plain extracted text
-
-    Error Handling:
-
-        Timeout retries (up to 3 attempts)
-
-        Local fallback to Tesseract if Mistral service unavailable
-
-Tesseract Fallback
-
-    Only invoked if Mistral API fails after retries.
-
-    No structured output; raw text only.

.clinerules/complexFeature.md

@@ -1,29 +0,0 @@
-# Complex Feature Documentation
-
-Document Type Detection
-
-    Utilizes lightweight statistical heuristics combined with visual features.
-
-    Preprocessing-driven (thresholding, aspect ratios, contour analysis).
-
-    Outputs labels such as "handwritten letter", "scientific report", "recipe".
-
-Preprocessing Pipelines
-
-    Customizable per document type.
-
-    Adaptive thresholding for delicate handwriting.
-
-    Morphological operations for removing bleed-through or artifacts.
-
-Multilingual Handling
-
-    Language detection on OCR snippets using language_detection.py.
-
-    Allows contextual OCR prompting based on dominant language.
-
-Structured Output Generation
-
-    Parsing OCR results into structured categories: titles, subtitles, body, marginalia, dates.
-
-    Supports output in raw text, JSON, and annotated Markdown.

.clinerules/hocr-basics-api.md

@@ -1,106 +0,0 @@
-# HOCR Basics: API Integrations (Streamlit and Mistral OCR)
-
-This rule defines the essential development standards for integrating the Mistral OCR API and using Streamlit components in the `milwright/historical-ocr` application.
-
-## 📌 Rule 1: Mistral OCR API Usage
-
-* **Endpoint:**
-  `POST https://api.mistral.ai/v1/ocr`
-
-* **Headers:**
-
-  ```http
-  Authorization: Bearer YOUR_API_KEY  
-  Content-Type: application/json
-  ```
-
-* **Required JSON Body Fields:**
-
-  ```json
-  {
-    "file_url": "https://example.com/your.pdf"
-  }
-  ```
-
-* **Expected Response Fields:**
-
-  * `text`: Raw OCR output
-  * `metadata`: Document structure, language, layout information
-
-> **Note:** Always validate presence of required fields and handle error codes gracefully.
-
----
-
-### 🖼️ Rule 2: Streamlit Usage Standards
-
-* Use these core components:
-
-  * `st.file_uploader()`
-  * `st.selectbox()`
-  * `st.image()`
-  * `st.markdown()`
-  * `st.download_button()`
-
-* Always set:
-  `use_container_width=True` for responsive display where supported
-
-* Avoid global state; prefer `st.session_state` for interactivity and stateful inputs
-
-## Mistral OCR Examples
-
-``` json
-{
-  "id": "string",
-  "object": "model",
-  "created": 0,{
-  "model": "string",
-  "id": "string",
-  "document": {
-    "document_url": "string",
-    "document_name": "string",
-    "type": "document_url"
-  },
-  "pages": [
-    0
-  ],
-  "include_image_base64": true,
-  "image_limit": 0,
-  "image_min_size": 0
-}
-```
-
-``` json
-{
-  "pages": [
-    {
-      "index": 0,
-      "markdown": "string",
-      "images": [
-        {
-          "id": "string",
-          "top_left_x": 0,
-          "top_left_y": 0,
-          "bottom_right_x": 0,
-          "bottom_right_y": 0,
-          "image_base64": "string"
-        }
-      ],
-      "dimensions": {
-        "dpi": 0,
-        "height": 0,
-        "width": 0
-      }
-    }
-  ],
-  "model": "string",
-  "usage_info": {
-    "pages_processed": 0,
-    "doc_size_bytes": 0
-  }
-}
-```
-
-### Links and Resources to Understand
-
-* [URL to Mistral OCR APi doc](https://docs.mistral.ai/api/#tag/batch/operation/jobs_api_routes_batch_cancel_batch_job)
-* [URL to Streamlit API documentation](https://docs.streamlit.io/develop/api-reference)

.clinerules/integrationSpecs.md

@@ -1,27 +0,0 @@
-# Integration Specifications
-
-External Services
-
-    Mistral OCR API: Primary service for document transcription and structured extraction.
-
-    Tesseract OCR (Local Fallback): Optional backup when API unavailable.
-
-Internal Module Communication
-
-    app.py triggers ocr_processing.py orchestration based on user input.
-
-    ocr_processing.py dynamically calls preprocessing and OCR modules based on document type.
-
-    Preprocessed images passed through structured_ocr.py for API interaction and postprocessing.
-
-Session State
-
-    Streamlit session stores:
-
-        Uploaded file metadata
-
-        Preprocessing parameters
-
-        Detected document type
-
-        OCR structured output

.clinerules/principle-of-simplicity.md

@@ -1,55 +0,0 @@
-# Project Rule: Maintain Clean Separation Between Data and Presentation
-The Principle of Content Purity
-Core rule: Data that needs to be processed or stored should never contain presentation markup that's only meant for display.
-
-What This Means in Practice
-Avoid HTML in data structures
-
-✅ DO: Keep raw text as pure text content
-❌ DON'T: Embed HTML, CSS, or other presentation-specific markup in data fields
-Design clear boundaries between data and presentation layers
-
-✅ DO: Add presentation elements at the final rendering stage only
-❌ DON'T: Add and strip presentation elements repeatedly throughout the processing pipeline
-Fix problems at their source, not their symptoms
-
-✅ DO: Prevent markup injection at the origin rather than adding complex stripping logic later
-❌ DON'T: Create complex sanitization functions to clean data that shouldn't be contaminated in the first place
-The OCR Text Formatter Example
-Before (problematic):
-
-def format_ocr_text(text, for_display=True):
-    # Text processing...
-    
-    if for_display:
-        html = f"""
-        <div class="ocr-text-container">
-            {formatted_text}
-        </div>
-        """
-        return html
-    else:
-        return formatted_text
-After (better):
-
-def format_ocr_text(text, for_display=False):
-    # Text processing...
-    
-    if for_display:
-        html = f"""
-            {formatted_text}
-        """
-        return html
-    else:
-        return formatted_text
-What changed:
-
-Default parameter changed to avoid accidental HTML addition
-HTML wrapper div completely removed to eliminate the source of pollution
-The simplest solution (removing the container) was better than any complex stripping logic
-Benefits
-Cleaner data: Raw content remains genuinely raw and easier to work with
-More predictable processing: No need to account for unexpected HTML in processing pipelines
-Easier debugging: Problems are visible at their source rather than as mysterious artifacts later
-Reduced complexity: Eliminates the need for complex HTML stripping and sanitization logic
-Remember: Simplicity is not just an ideal—it's a practical strategy that prevents entire classes of bugs.

.clinerules/productContext.md

@@ -1,25 +0,0 @@
-# Why the Project Exists
-
-Historians, archivists, and researchers often struggle to extract reliable text from scanned archival materials. Many OCR tools fail when dealing with handwritten letters, historical scientific documents, and poorly digitized photographs.
-
-Problems Being Solved
-
-Low OCR accuracy on handwritten or degraded historical documents
-
-Lack of structured metadata extraction for archival research
-
-Inability to easily apply context-specific AI prompting for nuanced historical material
-
-How the Product Should Work
-
-Users upload images or PDFs
-
-Preprocessing automatically improves OCR readiness
-
-Document type detection informs customized AI prompting
-
-Mistral OCR processes the document to output structured data (titles, authors, dates, body text, marginalia, etc.)
-
-Users can download raw text, structured JSON, or annotated markdown
-
-Example: "The OCR system must intelligently handle multilingual documents, support marginal notes and irregular layouts, and allow historians to guide the extraction process."

.clinerules/progress.md

@@ -1,27 +0,0 @@
-# Current Status of Features
-
-    ✅ Upload and preprocess historical documents
-
-    ⚙️ Document type detection (estimated 80% accuracy)
-
-    ✅ OCR extraction with structured outputs (titles, body, marginalia)
-
-    ✅ Multiple output formats (Raw text, JSON, Markdown)
-
-Known Issues and Limitations
-
-    Inconsistent marginalia capture in low-quality scans
-
-    Difficulties with heavily degraded non-Latin handwritten scripts
-
-    Layout detection errors on highly irregular, mixed-content PDFs
-
-Evolution of Project Decisions
-
-    Migrated from Tesseract-only OCR to Mistral-first hybrid approach
-
-    Modularized preprocessing steps to allow flexible experimentation
-
-    Added support for marginalia and footnotes where feasible
-
-    Enhanced session state management to preserve intermediate results.

.clinerules/systemPatterns.md

@@ -1,31 +0,0 @@
-# System Architecture
-
-    Frontend: Streamlit app (app.py) for user interface and interactions.
-
-    Core Processing: ocr_processing.py orchestrates preprocessing, document type detection, and OCR operations.
-
-    Image Preprocessing: preprocessing.py, image_segmentation.py handle deskewing, thresholding, and cleaning.
-
-    OCR and Structuring: structured_ocr.py and ocr_utils.py manage API communication and formatting structured outputs.
-
-    Utilities and Detection: language_detection.py, utils.py, and constants.py provide language detection, helpers, and prompt templates.
-
-Key Technical Decisions
-
-    Streamlit cache management for upload processing efficiency.
-
-    Modular design of preprocessing paths based on document type.
-
-    Mistral AI as the primary OCR processor, with Tesseract fallback for redundancy.
-
-Design Patterns in Use
-
-    Delegation: Frontend delegates all processing to backend orchestrators.
-
-    Modularity: Preprocessing and OCR tasks divided into clean, testable modules.
-
-    State-driven Processing: Output dynamically reflects session state and user input.
-
-Component Relationships
-
-    app.py ⇨ ocr_processing.py ⇨ preprocessing.py, structured_ocr.py, language_detection.py, etc.

.clinerules/techContext.md

@@ -1,40 +0,0 @@
-techContext.md
-Technologies and Frameworks Used
-
-    Frontend Framework: Streamlit 1.44.1
-
-    OCR Engine: Mistralai Python SDK (≥ 0.1.0)
-
-    Image Processing: OpenCV, Pillow
-
-    PDF Parsing: pdf2image
-
-    Fallback OCR: Pytesseract
-
-    Utilities: NumPy, Requests, pycountry
-
-Development Setup
-
-    Python 3.11+ virtual environment
-
-    Requirements managed through requirements.txt
-
-    .env file setup for API keys and environment configs
-
-    Type checking with mypy, linting with ruff
-
-Technical Constraints
-
-    API rate limits and payload size restrictions from Mistral
-
-    Streamlit's session state limitations for very large files
-
-    Processing timeouts for oversized or complex PDFs
-
-Dependencies and Tool Configurations
-
-    Mistralai pinned version (≥ 0.1.0)
-
-    OpenCV configured for headless environments
-
-    Pillow used for post-processing and visualization checks

.gitignore

@@ -18,6 +18,13 @@ output/preview/
 logs/
 *.backup
 *.json
+*.jpg
+*.png
+*.txt
+*.csv
+*.log
+*.zip
+*.tar
 
 # Test files
 test_*.py
@@ -33,3 +40,5 @@ input/*.pdf
 # Temporary documents
 Tmplf6xnkgr*
 .env
+output/pipeline_test/americae-retectio/americae-retectio_comparison.jpg
+docs/environment_variables.md

docs/config_refactoring.md

@@ -0,0 +1,47 @@
+# Configuration Refactoring
+
+## Overview
+This document outlines the changes made to centralize configuration parameters and reduce technical debt in the OCR processing system.
+
+## Key Changes
+
+### Centralized Configuration
+All previously hard-coded parameters have been moved to `config.py` and organized by functional category:
+
+- **PDF_SETTINGS**: Parameters for PDF processing
+- **SEGMENTATION_SETTINGS**: Image segmentation configuration
+- **CACHE_SETTINGS**: Cache TTL and capacity settings
+- **TEXT_REPAIR_SETTINGS**: Duplication detection and repair thresholds
+
+### Environment Variable Support
+All configuration parameters can now be overridden via environment variables:
+
+```bash
+# Example: Override PDF DPI
+export PDF_DEFAULT_DPI=200
+
+# Example: Increase cache size
+export CACHE_MAX_ENTRIES=50
+```
+
+### Import Strategy
+To prevent circular dependencies, configuration is imported at function level where needed:
+
+```python
+def process_image():
+    from config import SEGMENTATION_SETTINGS
+    # Function implementation using settings
+```
+
+## Benefits
+
+- **Maintainability**: Settings are centralized and documented
+- **Flexibility**: Configuration can be adjusted without code changes
+- **Consistency**: Standardized approach to configuration across modules
+- **Traceability**: Clear overview of all configurable parameters
+
+## Future Improvements
+
+- Add configuration schema validation
+- Support for configuration profiles (dev/test/prod)
+- Add detailed documentation for each parameter

improvements.md

@@ -0,0 +1,587 @@
+# Historical OCR Application Improvements
+
+Based on a thorough code review of the Historical OCR application, I've identified several areas for improvement to reduce technical debt and enhance the application's functionality, maintainability, and performance.
+
+## 1. Code Organization and Structure
+
+### 1.1 Modularize Large Functions
+Several functions in the codebase are excessively long and handle multiple responsibilities:
+
+- **Issue**: `process_file()` in ocr_processing.py is over 400 lines and handles file validation, preprocessing, OCR processing, and result formatting.
+- **Solution**: Break down into smaller, focused functions:
+  ```python
+  def process_file(uploaded_file, options):
+      # Validate and prepare file
+      file_info = validate_and_prepare_file(uploaded_file)
+      
+      # Apply preprocessing based on document type
+      preprocessed_file = preprocess_document(file_info, options)
+      
+      # Perform OCR processing
+      ocr_result = perform_ocr(preprocessed_file, options)
+      
+      # Format and enhance results
+      return format_and_enhance_results(ocr_result, file_info)
+  ```
+
+### 1.2 Consistent Error Handling
+Error handling approaches vary across modules:
+
+- **Issue**: Some functions use try/except blocks with detailed logging, while others return error dictionaries or raise exceptions.
+- **Solution**: Implement a consistent error handling strategy:
+  ```python
+  class OCRError(Exception):
+      def __init__(self, message, error_code=None, details=None):
+          self.message = message
+          self.error_code = error_code
+          self.details = details
+          super().__init__(self.message)
+          
+  def handle_error(func):
+      @functools.wraps(func)
+      def wrapper(*args, **kwargs):
+          try:
+              return func(*args, **kwargs)
+          except OCRError as e:
+              logger.error(f"OCR Error: {e.message} (Code: {e.error_code})")
+              return {"error": e.message, "error_code": e.error_code, "details": e.details}
+          except Exception as e:
+              logger.error(f"Unexpected error: {str(e)}")
+              return {"error": "An unexpected error occurred", "details": str(e)}
+      return wrapper
+  ```
+
+## 2. API Integration and Performance
+
+### 2.1 API Client Optimization
+The Mistral API client initialization and usage can be improved:
+
+- **Issue**: The client is initialized for each request and error handling is duplicated.
+- **Solution**: Create a singleton API client with centralized error handling:
+  ```python
+  class MistralClient:
+      _instance = None
+      
+      @classmethod
+      def get_instance(cls, api_key=None):
+          if cls._instance is None:
+              cls._instance = cls(api_key)
+          return cls._instance
+          
+      def __init__(self, api_key=None):
+          self.api_key = api_key or os.environ.get("MISTRAL_API_KEY", "")
+          self.client = Mistral(api_key=self.api_key)
+          
+      def process_ocr(self, document, **kwargs):
+          try:
+              return self.client.ocr.process(document=document, **kwargs)
+          except Exception as e:
+              # Centralized error handling
+              return self._handle_api_error(e)
+  ```
+
+### 2.2 Caching Strategy
+The current caching approach can be improved:
+
+- **Issue**: Cache keys don't always account for all relevant parameters, and TTL is fixed at 24 hours.
+- **Solution**: Implement a more sophisticated caching strategy:
+  ```python
+  def generate_cache_key(file_content, options):
+      # Create a comprehensive hash of all relevant parameters
+      options_str = json.dumps(options, sort_keys=True)
+      content_hash = hashlib.md5(file_content).hexdigest()
+      return f"{content_hash}_{hashlib.md5(options_str.encode()).hexdigest()}"
+      
+  # Adaptive TTL based on document type
+  def get_cache_ttl(document_type):
+      ttl_map = {
+          "handwritten": 48 * 3600,  # 48 hours for handwritten docs
+          "newspaper": 24 * 3600,    # 24 hours for newspapers
+          "standard": 12 * 3600      # 12 hours for standard docs
+      }
+      return ttl_map.get(document_type, 24 * 3600)
+  ```
+
+## 3. State Management
+
+### 3.1 Streamlit Session State
+The application uses a complex state management approach:
+
+- **Issue**: Many session state variables with unclear relationships and reset logic.
+- **Solution**: Implement a more structured state management approach:
+  ```python
+  class DocumentState:
+      def __init__(self):
+          self.document = None
+          self.original_bytes = None
+          self.name = None
+          self.mime_type = None
+          self.is_sample = False
+          self.processed = False
+          self.temp_files = []
+          
+      def reset(self):
+          # Clean up temp files
+          for temp_file in self.temp_files:
+              if os.path.exists(temp_file):
+                  os.unlink(temp_file)
+          
+          # Reset state
+          self.__init__()
+          
+  # Initialize in session state
+  if 'document_state' not in st.session_state:
+      st.session_state.document_state = DocumentState()
+  ```
+
+### 3.2 Result History Management
+The current approach to managing result history can be improved:
+
+- **Issue**: Results are stored directly in session state with limited management.
+- **Solution**: Create a dedicated class for result history:
+  ```python
+  class ResultHistory:
+      def __init__(self, max_results=20):
+          self.results = []
+          self.max_results = max_results
+          
+      def add_result(self, result):
+          # Add timestamp and ensure result is serializable
+          result = self._prepare_result(result)
+          self.results.insert(0, result)
+          
+          # Trim to max size
+          if len(self.results) > self.max_results:
+              self.results = self.results[:self.max_results]
+              
+      def _prepare_result(self, result):
+          # Add timestamp and ensure result is serializable
+          result = result.copy()
+          result['timestamp'] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+          
+          # Ensure result is serializable
+          return json.loads(json.dumps(result, default=str))
+  ```
+
+## 4. Image Processing Pipeline
+
+### 4.1 Preprocessing Configuration
+The preprocessing configuration can be improved:
+
+- **Issue**: Preprocessing options are scattered across different parts of the code.
+- **Solution**: Create a centralized preprocessing configuration:
+  ```python
+  PREPROCESSING_CONFIGS = {
+      "standard": {
+          "grayscale": True,
+          "denoise": True,
+          "contrast": 5,
+          "deskew": True
+      },
+      "handwritten": {
+          "grayscale": True,
+          "denoise": True,
+          "contrast": 10,
+          "deskew": True,
+          "adaptive_threshold": {
+              "block_size": 21,
+              "constant": 5
+          }
+      },
+      "newspaper": {
+          "grayscale": True,
+          "denoise": True,
+          "contrast": 5,
+          "deskew": True,
+          "column_detection": True
+      }
+  }
+  ```
+
+### 4.2 Image Segmentation
+The image segmentation approach can be improved:
+
+- **Issue**: Segmentation is optional and not well-integrated with the preprocessing pipeline.
+- **Solution**: Make segmentation a standard part of the preprocessing pipeline for certain document types:
+  ```python
+  def preprocess_document(file_info, options):
+      # Apply basic preprocessing
+      preprocessed_file = apply_basic_preprocessing(file_info, options)
+      
+      # Apply segmentation for specific document types
+      if options["document_type"] in ["newspaper", "book", "multi_column"]:
+          return apply_segmentation(preprocessed_file, options)
+      
+      return preprocessed_file
+  ```
+
+## 5. User Experience Enhancements
+
+### 5.1 Progressive Loading
+Improve the user experience during processing:
+
+- **Issue**: The UI can appear frozen during long-running operations.
+- **Solution**: Implement progressive loading and feedback:
+  ```python
+  def process_with_feedback(file, options, progress_callback):
+      # Update progress at each step
+      progress_callback(10, "Validating document...")
+      file_info = validate_and_prepare_file(file)
+      
+      progress_callback(30, "Preprocessing document...")
+      preprocessed_file = preprocess_document(file_info, options)
+      
+      progress_callback(50, "Performing OCR...")
+      ocr_result = perform_ocr(preprocessed_file, options)
+      
+      progress_callback(80, "Enhancing results...")
+      final_result = format_and_enhance_results(ocr_result, file_info)
+      
+      progress_callback(100, "Complete!")
+      return final_result
+  ```
+
+### 5.2 Result Visualization
+Enhance the visualization of OCR results:
+
+- **Issue**: Results are displayed in a basic format with limited visualization.
+- **Solution**: Implement enhanced visualization options:
+  ```python
+  def display_enhanced_results(result):
+      # Create tabs for different views
+      tabs = st.tabs(["Text", "Annotated", "Side-by-Side", "JSON"])
+      
+      with tabs[0]:
+          # Display formatted text
+          st.markdown(format_ocr_text(result["ocr_contents"]["raw_text"]))
+          
+      with tabs[1]:
+          # Display annotated image with bounding boxes
+          display_annotated_image(result)
+          
+      with tabs[2]:
+          # Display side-by-side comparison
+          col1, col2 = st.columns(2)
+          with col1:
+              st.image(result["original_image"])
+          with col2:
+              st.markdown(format_ocr_text(result["ocr_contents"]["raw_text"]))
+              
+      with tabs[3]:
+          # Display raw JSON
+          st.json(result)
+  ```
+
+## 6. Testing and Reliability
+
+### 6.1 Automated Testing
+Implement comprehensive testing:
+
+- **Issue**: Limited or no automated testing.
+- **Solution**: Implement unit and integration tests:
+  ```python
+  # Unit test for preprocessing
+  def test_preprocess_image():
+      # Test with various document types
+      for doc_type in ["standard", "handwritten", "newspaper"]:
+          # Load test image
+          with open(f"test_data/{doc_type}_sample.jpg", "rb") as f:
+              image_bytes = f.read()
+              
+          # Apply preprocessing
+          options = {"document_type": doc_type, "grayscale": True, "denoise": True}
+          result = preprocess_image(image_bytes, options)
+          
+          # Assert result is not None and different from original
+          assert result is not None
+          assert result != image_bytes
+  ```
+
+### 6.2 Error Recovery
+Implement better error recovery mechanisms:
+
+- **Issue**: Errors in one part of the pipeline can cause the entire process to fail.
+- **Solution**: Implement graceful degradation:
+  ```python
+  def process_with_fallbacks(file, options):
+      try:
+          # Try full processing pipeline
+          return full_processing_pipeline(file, options)
+      except OCRError as e:
+          logger.warning(f"Full pipeline failed: {e.message}. Trying simplified pipeline.")
+          try:
+              # Try simplified pipeline
+              return simplified_processing_pipeline(file, options)
+          except Exception as e2:
+              logger.error(f"Simplified pipeline failed: {str(e2)}. Falling back to basic OCR.")
+              # Fall back to basic OCR
+              return basic_ocr_only(file)
+  ```
+
+## 7. Documentation and Maintainability
+
+### 7.1 Code Documentation
+Improve code documentation:
+
+- **Issue**: Inconsistent documentation across modules.
+- **Solution**: Implement consistent docstring format and add module-level documentation:
+  ```python
+  """
+  OCR Processing Module
+  
+  This module handles the core OCR processing functionality, including:
+  - File validation and preparation
+  - Image preprocessing
+  - OCR processing with Mistral AI
+  - Result formatting and enhancement
+  
+  The main entry point is the `process_file` function.
+  """
+  
+  def process_file(file, options):
+      """
+      Process a file with OCR.
+      
+      Args:
+          file: The file to process (UploadedFile or bytes)
+          options: Dictionary of processing options
+              - document_type: Type of document (standard, handwritten, etc.)
+              - preprocessing: Dictionary of preprocessing options
+              - use_vision: Whether to use vision model
+              
+      Returns:
+          Dictionary containing OCR results and metadata
+          
+      Raises:
+          OCRError: If OCR processing fails
+      """
+      # Implementation
+  ```
+
+### 7.2 Configuration Management
+Improve configuration management:
+
+- **Issue**: Configuration is scattered across multiple files.
+- **Solution**: Implement a centralized configuration system:
+  ```python
+  """
+  Configuration Module
+  
+  This module provides a centralized configuration system for the application.
+  """
+  
+  import os
+  import yaml
+  from pathlib import Path
+  
+  class Config:
+      _instance = None
+      
+      @classmethod
+      def get_instance(cls):
+          if cls._instance is None:
+              cls._instance = cls()
+          return cls._instance
+          
+      def __init__(self):
+          self.config = {}
+          self.load_config()
+          
+      def load_config(self):
+          # Load from config file
+          config_path = Path(__file__).parent / "config.yaml"
+          if config_path.exists():
+              with open(config_path, "r") as f:
+                  self.config = yaml.safe_load(f)
+                  
+          # Override with environment variables
+          for key, value in os.environ.items():
+              if key.startswith("OCR_"):
+                  config_key = key[4:].lower()
+                  self.config[config_key] = value
+                  
+      def get(self, key, default=None):
+          return self.config.get(key, default)
+  ```
+
+## 8. Security Enhancements
+
+### 8.1 API Key Management
+Improve API key management:
+
+- **Issue**: API keys are stored in environment variables with limited validation.
+- **Solution**: Implement secure API key management:
+  ```python
+  def get_api_key():
+      # Try to get from secure storage first
+      api_key = get_from_secure_storage("mistral_api_key")
+      
+      # Fall back to environment variable
+      if not api_key:
+          api_key = os.environ.get("MISTRAL_API_KEY", "")
+          
+      # Validate key format
+      if api_key and not re.match(r'^[A-Za-z0-9_-]{30,}$', api_key):
+          logger.warning("API key format appears invalid")
+          
+      return api_key
+  ```
+
+### 8.2 Input Validation
+Improve input validation:
+
+- **Issue**: Limited validation of user inputs.
+- **Solution**: Implement comprehensive input validation:
+  ```python
+  def validate_file(file):
+      # Check file size
+      if len(file.getvalue()) > MAX_FILE_SIZE:
+          raise OCRError("File too large", "FILE_TOO_LARGE")
+          
+      # Check file type
+      file_type = get_file_type(file)
+      if file_type not in ALLOWED_FILE_TYPES:
+          raise OCRError(f"Unsupported file type: {file_type}", "UNSUPPORTED_FILE_TYPE")
+          
+      # Check for malicious content
+      if is_potentially_malicious(file):
+          raise OCRError("File appears to be malicious", "SECURITY_RISK")
+          
+      return file_type
+  ```
+
+## 9. Performance Optimizations
+
+### 9.1 Parallel Processing
+Implement parallel processing for multi-page documents:
+
+- **Issue**: Pages are processed sequentially, which can be slow for large documents.
+- **Solution**: Implement parallel processing:
+  ```python
+  def process_pdf_pages(pdf_path, options):
+      # Extract pages
+      pages = extract_pdf_pages(pdf_path)
+      
+      # Process pages in parallel
+      with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
+          future_to_page = {executor.submit(process_page, page, options): i 
+                           for i, page in enumerate(pages)}
+          
+          results = []
+          for future in concurrent.futures.as_completed(future_to_page):
+              page_idx = future_to_page[future]
+              try:
+                  result = future.result()
+                  results.append((page_idx, result))
+              except Exception as e:
+                  logger.error(f"Error processing page {page_idx}: {str(e)}")
+                  
+      # Sort results by page index
+      results.sort(key=lambda x: x[0])
+      
+      # Combine results
+      return combine_page_results([r[1] for r in results])
+  ```
+
+### 9.2 Resource Management
+Improve resource management:
+
+- **Issue**: Temporary files are not always cleaned up properly.
+- **Solution**: Implement better resource management:
+  ```python
+  class TempFileManager:
+      def __init__(self):
+          self.temp_files = []
+          
+      def create_temp_file(self, content, suffix=".tmp"):
+          with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
+              tmp.write(content)
+              self.temp_files.append(tmp.name)
+              return tmp.name
+              
+      def cleanup(self):
+          for temp_file in self.temp_files:
+              try:
+                  if os.path.exists(temp_file):
+                      os.unlink(temp_file)
+              except Exception as e:
+                  logger.warning(f"Failed to remove temp file {temp_file}: {str(e)}")
+          self.temp_files = []
+          
+      def __enter__(self):
+          return self
+          
+      def __exit__(self, exc_type, exc_val, exc_tb):
+          self.cleanup()
+  ```
+
+## 10. Extensibility
+
+### 10.1 Plugin System
+Implement a plugin system for extensibility:
+
+- **Issue**: Adding new document types or processing methods requires code changes.
+- **Solution**: Implement a plugin system:
+  ```python
+  class OCRPlugin:
+      def __init__(self, name, description):
+          self.name = name
+          self.description = description
+          
+      def can_handle(self, file_info):
+          """Return True if this plugin can handle the file"""
+          raise NotImplementedError
+          
+      def process(self, file_info, options):
+          """Process the file and return results"""
+          raise NotImplementedError
+          
+  # Example plugin
+  class HandwrittenDocumentPlugin(OCRPlugin):
+      def __init__(self):
+          super().__init__("handwritten", "Handwritten document processor")
+          
+      def can_handle(self, file_info):
+          # Check if this is a handwritten document
+          return file_info.get("document_type") == "handwritten"
+          
+      def process(self, file_info, options):
+          # Specialized processing for handwritten documents
+          # ...
+  ```
+
+### 10.2 API Abstraction
+Create an abstraction layer for the OCR API:
+
+- **Issue**: The application is tightly coupled to the Mistral AI API.
+- **Solution**: Implement an abstraction layer:
+  ```python
+  class OCRProvider:
+      def process_image(self, image_path, options):
+          """Process an image and return OCR results"""
+          raise NotImplementedError
+          
+      def process_pdf(self, pdf_path, options):
+          """Process a PDF and return OCR results"""
+          raise NotImplementedError
+          
+  class MistralOCRProvider(OCRProvider):
+      def __init__(self, api_key=None):
+          self.client = MistralClient.get_instance(api_key)
+          
+      def process_image(self, image_path, options):
+          # Implementation using Mistral API
+          
+      def process_pdf(self, pdf_path, options):
+          # Implementation using Mistral API
+          
+  # Factory function to get the appropriate provider
+  def get_ocr_provider(provider_name="mistral"):
+      if provider_name == "mistral":
+          return MistralOCRProvider()
+      # Add more providers as needed
+      raise ValueError(f"Unknown OCR provider: {provider_name}")
+  ```
+
+## Implementation Priority
+

memory-bank/activeContext.md

@@ -0,0 +1,31 @@
+# Active Context
+
+## Current Work Focus
+
+*   Initializing the core Memory Bank files (`projectbrief.md`, `productContext.md`, `systemPatterns.md`, `techContext.md`, `activeContext.md`, `progress.md`) based on the existing project structure, defined `.clinerules`, and the global `memory-bank.md` specification.
+
+## Recent Changes
+
+*   This is the initial population of the Memory Bank. No prior changes within this system.
+
+## Next Steps
+
+*   Complete the creation of the initial Memory Bank files (specifically `progress.md`).
+*   Await further instructions from the user regarding the next development task for the `historical-ocr` project.
+
+## Active Decisions, Patterns, and Preferences
+
+*   **Memory Bank Structure:** Adopting the 6-core-file structure defined in the global `memory-bank.md`.
+*   **Content Derivation:** Populating initial Memory Bank content by analyzing the existing file structure and `.clinerules`.
+*   **Iterative Improvement Workflow:** Implementing a post-task review step after completing tasks in Act Mode. This involves:
+    1.  Completing the assigned task.
+    2.  Presenting the result using `attempt_completion`.
+    3.  Explicitly asking the user: "Are there any takeaways from this interaction that should be added to the project's `.clinerules`?"
+    4.  If takeaways are provided, update the relevant `.clinerules` file(s).
+    5.  Recursively update the Memory Bank (especially `activeContext.md` and `progress.md`) to reflect the new rule or pattern.
+
+## Learnings and Project Insights
+
+*   The project emphasizes clear documentation and rule definition from the start (`.clinerules`).
+*   Maintaining synchronization between `.clinerules` and the Memory Bank is crucial for consistent development.
+*   The iterative feedback loop (post-task review) is a key requirement for evolving project standards.

memory-bank/productContext.md

@@ -0,0 +1,31 @@
+# Product Context
+
+## Why This Project Exists
+
+Historical documents often contain invaluable information but are locked away in formats (scans, photos, complex PDFs) that are difficult to search, analyze, or integrate into digital research workflows. Standard OCR tools may struggle with the unique challenges presented by archival materials, such as varying print quality, handwriting, complex layouts, and archaic language.
+
+## Problems It Solves
+
+This application aims to bridge the gap between physical historical archives and modern digital research by:
+
+*   Providing high-accuracy text extraction from challenging historical documents.
+*   Structuring the extracted text and metadata in a usable format.
+*   Making advanced OCR capabilities accessible to researchers without requiring deep technical expertise.
+*   Optimizing documents specifically for OCR to improve accuracy.
+
+## How It Should Work
+
+The core workflow involves:
+
+1.  **Upload:** Users upload historical documents (images or PDFs) via a web interface.
+2.  **Preprocessing:** The application automatically applies image enhancement and optimization techniques tailored to historical materials.
+3.  **OCR:** Processed documents are sent to the Mistral AI OCR API for text extraction. Document type detection may inform specific OCR prompting.
+4.  **Structuring:** The raw OCR output is processed to extract structured information (e.g., paragraphs, headings, metadata) based on document type and potentially user instructions.
+5.  **Output:** Users can view the extracted text and download structured transcripts and analysis.
+
+## User Experience Goals
+
+*   **Intuitive Interface:** A clean, straightforward Streamlit web application that is easy for researchers to use.
+*   **Clear Feedback:** Provide users with status updates during processing and clear presentation of results.
+*   **Flexibility:** Allow users some control over the process (e.g., contextual instructions) where appropriate.
+*   **Reliability:** Ensure consistent and accurate results.

memory-bank/progress.md

@@ -0,0 +1,34 @@
+# Project Progress
+
+## Current Status
+
+*   **Overall:** Initial project setup phase. Core application structure exists, and foundational documentation (Memory Bank, `.clinerules`) is being established.
+*   **Memory Bank:** The six core Memory Bank files have just been created with initial content derived from the project structure and rules.
+
+## What Works
+
+*   The basic file and directory structure for a modular Python/Streamlit application is in place.
+*   Project rules (`.clinerules`) defining the brief, API usage, simplicity principle, and technical debt priorities exist.
+*   The Memory Bank system is now initialized.
+
+## What's Left to Build / Next Steps
+
+*   Implement the actual functionality described in the project brief (file upload, preprocessing, OCR integration, structuring, UI interactions).
+*   Address the technical debt items listed below.
+*   Refine and expand Memory Bank documentation as development progresses.
+*   Specific next development tasks are pending user direction.
+
+## Known Issues / Technical Debt
+
+The following technical debt items have been identified in `.clinerules/technical-debt.md` and should be addressed during development:
+
+1.  **Modularize large functions:** Break down functions exceeding 100 lines into smaller, focused units.
+2.  **Consistent Error Handling:** Implement a uniform error handling strategy across all modules.
+3.  **Preprocessing Pipeline Improvement:** Enhance the preprocessing steps to better handle diverse historical document types.
+4.  **Image Segmentation Enhancement:** Improve the current approach for identifying text regions.
+5.  **Documentation:** Create comprehensive documentation (docstrings, comments) for public functions and APIs.
+
+## Evolution of Project Decisions
+
+*   **[YYYY-MM-DD]:** Initialized Memory Bank structure based on global rules.
+*   **[YYYY-MM-DD]:** Adopted a post-task review workflow to iteratively update `.clinerules` and the Memory Bank.

.clinerules/projectBrief.md → memory-bank/project-brief.md

@@ -1,21 +1,19 @@
-# Foundation
+# Project Brief
 
 Historical OCR is an advanced optical character recognition (OCR) application designed to support historical research. It leverages Mistral AI's OCR models alongside image preprocessing pipelines optimized for archival material.
 
-High-Level Overview
+## High-Level Overview
 
 Building a Streamlit-based web application to process historical documents (images or PDFs), optimize them for OCR using advanced preprocessing techniques, and extract structured text and metadata through Mistral's large language models.
 
-Core Requirements and Goals
+## Core Requirements and Goals
 
-Upload and preprocess historical documents
+* Upload and preprocess historical documents
 
-Automatically detect document types (e.g., handwritten letters, scientific papers)
+* Apply tailored OCR prompting and structured output based on document type
 
-Apply tailored OCR prompting and structured output based on document type
+* Support user-defined contextual instructions to refine output
 
-Support user-defined contextual instructions to refine output
+* Provide downloadable structured transcripts and analysis
 
-Provide downloadable structured transcripts and analysis
-
-Example: "Building a Streamlit web app for OCR transcription and structured extraction from historical documents using Mistral AI."
+* Example: "Building a Streamlit web app for OCR transcription and structured extraction from historical documents using Mistral AI."

memory-bank/systemPatterns.md

@@ -0,0 +1,66 @@
+# System Patterns
+
+## Architecture Overview
+
+The application follows a modular Python structure, orchestrated by a main Streamlit application script (`app.py`). Key architectural components include:
+
+*   **Entry Point:** `app.py` likely initializes the Streamlit application and coordinates calls to other modules.
+*   **UI Layer:** Managed by Streamlit. Core UI elements are defined in `ui/layout.py` and potentially reusable components in `ui/ui_components.py` (or the root `ui_components.py`). Custom styling is applied via `ui/custom.css`.
+*   **Processing Modules:** Functionality is separated into distinct Python modules:
+    *   `preprocessing.py`: Handles image optimization and preparation for OCR.
+    *   `image_segmentation.py`: Deals with identifying regions of interest within documents.
+    *   `ocr_processing.py`: Manages the interaction with the OCR engine (Mistral API).
+    *   `structured_ocr.py`: Focuses on interpreting raw OCR output and structuring it.
+    *   `language_detection.py`: Detects the language of the document content.
+    *   `letterhead_handler.py`: Specific logic for dealing with letterheads.
+    *   `pdf_ocr.py`: Handles OCR specific to PDF inputs (likely coordinating other modules).
+    *   `process_file.py`: A potential high-level orchestrator for the entire file processing pipeline.
+*   **Configuration:** `config.py` likely holds application settings, potentially including API keys or processing parameters. `constants.py` holds fixed values used across the application.
+*   **Utilities:** Common functions are grouped in the `utils/` directory, further categorized (e.g., `image_utils.py`, `text_utils.py`, `file_utils.py`).
+*   **Error Handling:** A dedicated `error_handler.py` suggests a centralized approach to managing exceptions.
+
+## Key Technical Decisions & Patterns
+
+*   **Modularity:** Code is organized into feature-specific modules, promoting separation of concerns.
+*   **External API Integration:** Relies on the Mistral AI OCR API for core text extraction (`.clinerules/hocr-basics-api.md`). API interaction logic is likely within `ocr_processing.py` or related utilities.
+*   **Streamlit Framework:** Leverages Streamlit for the web interface, using standard components (`.clinerules/hocr-basics-api.md`). State management likely uses `st.session_state`.
+*   **Content Purity:** Adheres to the principle of separating data from presentation markup (`.clinerules/principle-of-simplicity.md`). Presentation logic should reside primarily in the UI layer.
+*   **Configuration Management:** Centralized configuration likely managed through `config.py`.
+
+## Component Relationships (Conceptual)
+
+```mermaid
+graph TD
+    A[User via Streamlit UI] --> B(app.py);
+    B --> C{process_file.py?};
+    C --> D[preprocessing.py];
+    C --> E[image_segmentation.py];
+    C --> F[language_detection.py];
+    C --> G[pdf_ocr.py / ocr_processing.py];
+    G -- Mistral API --> H((External Mistral OCR));
+    H -- OCR Result --> G;
+    G --> I[structured_ocr.py];
+    I --> J[Output Generation];
+    J --> A;
+
+    subgraph Modules
+        D; E; F; G; I; J;
+    end
+
+    subgraph Configuration & Utils
+        K[config.py];
+        L[constants.py];
+        M[utils/];
+        N[error_handler.py];
+    end
+
+    B --> K; B --> L; B --> M; B --> N;
+    Modules --> K; Modules --> L; Modules --> M; Modules --> N;
+```
+
+## Critical Implementation Paths
+
+*   The end-to-end flow from file upload (`st.file_uploader`) through preprocessing, OCR API call, structuring, and displaying results (`st.markdown`, `st.download_button`).
+*   Handling different input file types (Images vs. PDFs).
+*   Integration and error handling for the Mistral API calls.
+*   Implementation of specific preprocessing steps relevant to historical documents.

memory-bank/techContext.md

@@ -0,0 +1,35 @@
+# Technical Context
+
+## Technologies Used
+
+*   **Primary Language:** Python 3.x
+*   **Web Framework:** Streamlit
+*   **Core API:** Mistral AI OCR API (via HTTPS requests)
+*   **Potential Libraries:**
+    *   `requests`: For making HTTP calls to the Mistral API.
+    *   `streamlit`: For the web UI framework.
+    *   `Pillow` (PIL Fork): For basic image loading and manipulation.
+    *   `OpenCV` (`cv2`): Likely used for more advanced image preprocessing tasks (e.g., thresholding, noise reduction, deskewing).
+    *   `python-dotenv`: Potentially used for managing environment variables like API keys (especially if `config.py` loads from a `.env` file).
+    *   `PyMuPDF` or similar: If PDF processing involves direct text/image extraction from PDF structures beyond just sending to OCR.
+
+*(Note: Specific libraries beyond Streamlit and requests need confirmation, e.g., by inspecting `requirements.txt` or import statements in the code).*
+
+## Development Setup
+
+*   **Environment:** Standard Python environment (virtual environment recommended, e.g., `venv` or `conda`).
+*   **Dependencies:** Install required packages (likely via `pip install -r requirements.txt` if a requirements file exists).
+*   **API Keys:** Requires a Mistral AI API key, which needs to be configured securely (likely via environment variables loaded in `config.py`).
+*   **Running the App:** Typically run using `streamlit run app.py` from the project root directory.
+
+## Technical Constraints
+
+*   **API Limits:** Subject to Mistral AI API usage limits, rate limits, and potential costs. Error handling for API responses (e.g., 429 Too Many Requests, 401 Unauthorized, 5xx Server Errors) is crucial.
+*   **Processing Time:** OCR and complex image preprocessing can be time-consuming, especially for large documents or high-resolution images. Streamlit's execution model needs to be considered for long-running tasks (e.g., using background processes or providing user feedback).
+*   **Resource Usage:** Image processing can be memory and CPU intensive.
+
+## Tool Usage Patterns
+
+*   **Streamlit Components:** Primarily use core components as specified in `.clinerules/hocr-basics-api.md` (`st.file_uploader`, `st.selectbox`, `st.image`, `st.markdown`, `st.download_button`).
+*   **State Management:** Use `st.session_state` for managing user interactions and state across reruns.
+*   **API Interaction:** Follow standard practices for REST API calls (headers, JSON body, error checking) as defined in `.clinerules/hocr-basics-api.md`.

utils/README.md

@@ -0,0 +1,75 @@
+# OCR Utilities
+
+This directory contains utility modules for the Historical OCR project.
+
+## PDF OCR Processing
+
+The `pdf_ocr.py` module provides specialized functionality for processing PDF documents with OCR.
+
+### Features
+
+- **Robust PDF-to-Image Conversion**: Converts PDF documents to images using optimized settings before OCR processing
+- **Multi-Page Support**: Intelligently handles multi-page documents, allowing processing of specific pages or page ranges
+- **Memory-Efficient Processing**: Processes PDFs in batches to prevent memory issues with large documents
+- **Fallback Mechanism**: Falls back to structured_ocr's internal processing if direct conversion fails
+- **Cleanup Management**: Automatically cleans up temporary files after processing
+
+### Key Components
+
+- **PDFOCR**: Main class for processing PDF files with OCR
+- **PDFConversionResult**: Helper class that holds PDF conversion results and manages cleanup
+
+### Basic Usage
+
+```python
+from utils.pdf_ocr import PDFOCR
+
+# Initialize the processor
+processor = PDFOCR()
+
+# Process a PDF file (all pages, with vision model)
+result = processor.process_pdf('document.pdf')
+
+# Process a PDF file (specific pages, with vision model)
+result = processor.process_pdf('document.pdf', custom_pages=[1, 3, 5])
+
+# Process a PDF file (first N pages, without vision model)
+result = processor.process_pdf('document.pdf', max_pages=3, use_vision=False)
+
+# Process a PDF file with custom prompt
+result = processor.process_pdf(
+    'document.pdf', 
+    custom_prompt="This is a historical newspaper with multiple columns."
+)
+
+# Save results to JSON
+output_path = processor.save_json_output('document.pdf', 'results.json')
+```
+
+### Command Line Usage
+
+The module can also be used directly from the command line:
+
+```bash
+python utils/pdf_ocr.py document.pdf --output results.json
+python utils/pdf_ocr.py document.pdf --max-pages 3
+python utils/pdf_ocr.py document.pdf --pages 1,3,5
+python utils/pdf_ocr.py document.pdf --prompt "This is a historical newspaper with multiple columns."
+python utils/pdf_ocr.py document.pdf --no-vision
+```
+
+### How It Works
+
+1. The module first attempts to convert the PDF to images using `pdf2image`
+2. It processes the first page with the vision model (if requested) for detailed analysis
+3. Additional pages are processed with the text model for efficiency
+4. All text is combined into a single result with appropriate metadata
+5. If direct conversion fails, it falls back to using `structured_ocr.py` for PDF processing
+
+### Parameters
+
+- **pdf_path**: Path to the PDF file to process
+- **use_vision**: Whether to use vision model for improved analysis (default: True)
+- **max_pages**: Maximum number of pages to process (default: all pages)
+- **custom_pages**: Specific page numbers to process, 1-based indexing (e.g., [1, 3, 5])
+- **custom_prompt**: Custom instructions for OCR processing