add memory

.clinerules/activeContext.md

@@ -0,0 +1,18 @@
+# Active Context
+
+## Current Development Focus
+- Improving preprocessing pipeline for better OCR results
+- Enhancing image segmentation for complex documents
+- Building structured OCR capabilities
+- Testing and validation with different document types
+
+## Recent Changes
+- Modularized code structure
+- Added helpers in utils directory
+- Fixed metadata field ordering and tag classification issues
+- Updated gitignore to exclude test files and output directories
+
+## Next Steps
+- Continue refining the OCR processing pipeline
+- Improve handling of handwritten documents
+- Enhance the UI components for better user experience

.clinerules/memory-bank.md

@@ -1,113 +1,19 @@
-# Cline's Memory Bank
+# HOCR Project Memory Bank
 
-I am Cline, an expert software engineer with a unique characteristic: my memory resets completely between sessions. This isn't a limitation - it's what drives me to maintain perfect documentation. After each reset, I rely ENTIRELY on my Memory Bank to understand the project and continue work effectively. I MUST read ALL memory bank files at the start of EVERY task - this is not optional.
+This memory bank is for the HOCR (OCR processing) project.
 
-## Memory Bank Structure
+## Project Context
 
-The Memory Bank consists of core files and optional context files, all in Markdown format. Files build upon each other in a clear hierarchy:
+This project appears to be focused on OCR (Optical Character Recognition) processing, with capabilities for image segmentation, preprocessing, and various text extraction techniques.
 
-flowchart TD
-    PB[projectbrief.md] --> PC[productContext.md]
-    PB --> SP[systemPatterns.md]
-    PB --> TC[techContext.md] 
-    PC --> AC[activeContext.md]
-    SP --> AC
-    TC --> AC 
-    AC --> P[progress.md]
+## System Information
 
-### Core Files (Required)
-1. `projectbrief.md`
-   - Foundation document that shapes all other files
-   - Created at project start if it doesn't exist
-   - Defines core requirements and goals
-   - Source of truth for project scope
+- Project directory: /Users/zacharymuhlbauer/Desktop/tools/hocr
+- Main Python files include app.py, preprocessing.py, ocr_processing.py, and various utility modules
+- Output directories for test results and processing stages
 
-2. `productContext.md`
-   - Why this project exists
-   - Problems it solves
-   - How it should work
-   - User experience goals
+## Notes
 
-3. `activeContext.md`
-   - Current work focus
-   - Recent changes
-   - Next steps
-   - Active decisions and considerations
-   - Important patterns and preferences
-   - Learnings and project insights
-
-4. `systemPatterns.md`
-   - System architecture
-   - Key technical decisions
-   - Design patterns in use
-   - Component relationships
-   - Critical implementation paths
-
-5. `techContext.md`
-   - Technologies used
-   - Development setup
-   - Technical constraints
-   - Dependencies
-   - Tool usage patterns
-
-6. `progress.md`
-   - What works
-   - What's left to build
-   - Current status
-   - Known issues
-   - Evolution of project decisions
-
-### Additional Context
-Create additional files/folders within memory-bank/ when they help organize:
-- Complex feature documentation
-- Integration specifications
-- API documentation
-- Testing strategies
-- Deployment procedures
-
-## Core Workflows
-
-### Plan Mode
-flowchart TD
-    Start[Start] --> ReadFiles[Read Memory Bank]
-    ReadFiles --> CheckFiles{Files Complete?}
-    
-    CheckFiles -->|No| Plan[Create Plan]
-    Plan --> Document[Document in Chat]
-    
-    CheckFiles -->|Yes| Verify[Verify Context]
-    Verify --> Strategy[Develop Strategy]
-    Strategy --> Present[Present Approach]
-
-### Act Mode
-flowchart TD
-    Start[Start] --> Context[Check Memory Bank]
-    Context --> Update[Update Documentation]
-    Update --> Execute[Execute Task]
-    Execute --> Document[Document Changes]
-
-## Documentation Updates
-
-Memory Bank updates occur when:
-1. Discovering new project patterns
-2. After implementing significant changes
-3. When user requests with **update memory bank** (MUST review ALL files)
-4. When context needs clarification
-
-flowchart TD
-    Start[Update Process]
-    
-    subgraph Process
-        P1[Review ALL Files]
-        P2[Document Current State]
-        P3[Clarify Next Steps]
-        P4[Document Insights & Patterns]
-        
-        P1 --> P2 --> P3 --> P4
-    end
-    
-    Start --> Process
-
-Note: When triggered by **update memory bank**, I MUST review every memory bank file, even if some don't require updates. Focus particularly on activeContext.md and progress.md as they track current state.
-
-REMEMBER: After every memory reset, I begin completely fresh. The Memory Bank is my only link to previous work. It must be maintained with precision and clarity, as my effectiveness depends entirely on its accuracy.
+- The project handles various document types including handwritten documents, printed text, and mixed content
+- Contains preprocessing steps for image enhancement before OCR
+- Has testing directories for different document types and processing approaches

.clinerules/productContext.md

@@ -0,0 +1,11 @@
+# Product Context
+
+This is an OCR processing tool for various document types including handwritten, printed, and mixed content documents. The system handles preprocessing, image segmentation, OCR processing, and text extraction with various enhancements.
+
+## Features
+- Document preprocessing (deskewing, thresholding, etc.)
+- Image segmentation to identify text regions
+- OCR processing with different strategies for different document types
+- Language detection
+- Letterhead handling
+- Structured data extraction

.clinerules/progress.md

@@ -0,0 +1,20 @@
+# Progress
+
+## Completed
+- Basic OCR processing pipeline
+- Image preprocessing capabilities
+- Text segmentation algorithm
+- Initial UI components
+- Testing framework for various document types
+
+## In Progress
+- Improving preprocessing for handwritten documents
+- Enhancing segmentation accuracy
+- Building structured output formatting
+- Refining language detection
+
+## Planned
+- Additional output formats
+- Performance optimization
+- More comprehensive testing
+- Enhanced UI features

.clinerules/project-brief.md

@@ -0,0 +1,13 @@
+# Project Brief
+
+## Overview
+This project focuses on Optical Character Recognition (OCR) processing for various document types. It handles different document formats and qualities, applying specialized preprocessing and recognition techniques.
+
+## Goals
+- Improve OCR accuracy for challenging document types
+- Support multiple input formats (images, PDFs)
+- Provide structured output of recognized text
+- Enable interactive usage with UI components
+
+## Current Status
+The project has multiple components in place including preprocessing, segmentation, OCR processing, and utility functions. Testing infrastructure is available for different document types and processing approaches.

.clinerules/systemPatterns.md

@@ -0,0 +1,19 @@
+# System Patterns
+
+## Code Organization
+- Main processing components in root directory
+- Utility functions in utils/ directory with specific submodules
+- UI components in ui/ directory
+- Test cases and samples in testing/ directory
+- Input/output directories for document processing
+
+## Naming Conventions
+- Snake case for file names and functions
+- Module names reflect their purpose (e.g., ocr_processing.py, image_segmentation.py)
+- Consistent test output naming with descriptive prefixes
+
+## Processing Pipeline
+1. Preprocessing step (enhancement, cleaning)
+2. Segmentation (identifying text regions)
+3. OCR processing with context-specific strategies
+4. Post-processing and output formatting

.clinerules/techContext.md

@@ -0,0 +1,16 @@
+# Technical Context
+
+This project is a Python-based OCR solution with the following components:
+
+## Tech Stack
+- Python
+- Image processing libraries (OpenCV, PIL)
+- OCR engines
+- UI components for interactive usage
+- File processing utilities
+
+## Architecture
+- Modular design with separate components for preprocessing, OCR, and output formatting
+- Utility modules organized in utils/ directory
+- Testing framework for various document types
+- Configuration system for processing parameters

memory-bank/activeContext.md

@@ -1,31 +1,36 @@
-# Active Context
+# Active Context: HOCR Processing Tool - Initial Setup
 
-## Current Work Focus
+## 1. Current Focus
 
-*   Initializing the core Memory Bank files (`projectbrief.md`, `productContext.md`, `systemPatterns.md`, `techContext.md`, `activeContext.md`, `progress.md`) based on the existing project structure, defined `.clinerules`, and the global `memory-bank.md` specification.
+*   **Initial Project Setup:** Establishing the core Memory Bank documentation structure for the HOCR project as per Cline's requirements.
+*   Creating the foundational markdown files (`projectbrief.md`, `productContext.md`, `activeContext.md`, `systemPatterns.md`, `techContext.md`, `progress.md`).
 
-## Recent Changes
+## 2. Recent Changes
 
-*   This is the initial population of the Memory Bank. No prior changes within this system.
+*   Created `memory-bank` directory within the `hocr` project folder.
+*   Created `projectbrief.md` with initial project goals and scope.
+*   Created `productContext.md` outlining the problem space, vision, and user goals.
+*   Created this `activeContext.md` file.
 
-## Next Steps
+## 3. Next Steps
 
-*   Complete the creation of the initial Memory Bank files (specifically `progress.md`).
-*   Await further instructions from the user regarding the next development task for the `historical-ocr` project.
+*   Create `systemPatterns.md` to document the high-level architecture and design patterns (based on initial analysis of existing code).
+*   Create `techContext.md` to detail the technologies, libraries, and setup requirements.
+*   Create `progress.md` to establish the baseline for tracking project status.
+*   Once the initial Memory Bank is set up, await further instructions or tasks from the user regarding the HOCR project itself.
 
-## Active Decisions, Patterns, and Preferences
+## 4. Active Decisions & Considerations
 
-*   **Memory Bank Structure:** Adopting the 6-core-file structure defined in the global `memory-bank.md`.
-*   **Content Derivation:** Populating initial Memory Bank content by analyzing the existing file structure and `.clinerules`.
-*   **Iterative Improvement Workflow:** Implementing a post-task review step after completing tasks in Act Mode. This involves:
-    1.  Completing the assigned task.
-    2.  Presenting the result using `attempt_completion`.
-    3.  Explicitly asking the user: "Are there any takeaways from this interaction that should be added to the project's `.clinerules`?"
-    4.  If takeaways are provided, update the relevant `.clinerules` file(s).
-    5.  Recursively update the Memory Bank (especially `activeContext.md` and `progress.md`) to reflect the new rule or pattern.
+*   Following Cline's standard Memory Bank structure.
+*   Populating initial files with baseline information derived from the project's file structure and general OCR principles. These will need refinement as the project is explored further.
 
-## Learnings and Project Insights
+## 5. Important Patterns & Preferences
 
-*   The project emphasizes clear documentation and rule definition from the start (`.clinerules`).
-*   Maintaining synchronization between `.clinerules` and the Memory Bank is crucial for consistent development.
-*   The iterative feedback loop (post-task review) is a key requirement for evolving project standards.
+*   *(To be filled in as patterns are identified in the codebase or specified by the user)*
+
+## 6. Learnings & Insights
+
+*   The HOCR project appears to be a substantial Python application with distinct modules for different OCR stages (preprocessing, segmentation, OCR processing, etc.) and includes a UI component.
+*   Initial setup requires creating the standard Memory Bank files.
+
+*(This file tracks the immediate state and short-term plans. It should be updated frequently.)*

memory-bank/productContext.md

@@ -1,31 +1,44 @@
-# Product Context
+# Product Context: HOCR Processing Tool
 
-## Why This Project Exists
+## 1. Problem Space
 
-Historical documents often contain invaluable information but are locked away in formats (scans, photos, complex PDFs) that are difficult to search, analyze, or integrate into digital research workflows. Standard OCR tools may struggle with the unique challenges presented by archival materials, such as varying print quality, handwriting, complex layouts, and archaic language.
+*   Extracting text from images and scanned documents (PDFs) is a common but often challenging task.
+*   Documents vary widely in quality, layout, language, and format.
+*   Manual transcription is time-consuming, error-prone, and not scalable.
+*   Existing OCR tools may lack flexibility, specific preprocessing capabilities, or produce output that isn't easily usable for downstream tasks.
 
-## Problems It Solves
+## 2. Solution Vision
 
-This application aims to bridge the gap between physical historical archives and modern digital research by:
+*   Provide a reliable, configurable, and extensible OCR processing pipeline.
+*   Empower users to convert document images and PDFs into usable, structured text data.
+*   Offer fine-grained control over the OCR process, from preprocessing to output formatting.
+*   Serve as a foundational tool that can be integrated into larger document processing workflows.
 
-*   Providing high-accuracy text extraction from challenging historical documents.
-*   Structuring the extracted text and metadata in a usable format.
-*   Making advanced OCR capabilities accessible to researchers without requiring deep technical expertise.
-*   Optimizing documents specifically for OCR to improve accuracy.
+## 3. Target Audience & Needs
 
-## How It Should Work
+*   **Researchers/Academics:** Need to extract text from historical documents, scanned books, or research papers for analysis. Require high accuracy and potentially language-specific handling.
+*   **Archivists/Librarians:** Need to digitize large collections of documents, making them searchable and accessible. Require batch processing capabilities and robust handling of varied document types.
+*   **Developers:** Need an OCR component to integrate into applications (e.g., document management systems, data extraction tools). Require a clear API or command-line interface and structured output (like JSON or hOCR).
+*   **General Users:** May need to occasionally extract text from a scanned form, receipt, or image. Require a simple interface (if applicable) and reasonable default settings.
 
-The core workflow involves:
+## 4. User Experience Goals
 
-1.  **Upload:** Users upload historical documents (images or PDFs) via a web interface.
-2.  **Preprocessing:** The application automatically applies image enhancement and optimization techniques tailored to historical materials.
-3.  **OCR:** Processed documents are sent to the Mistral AI OCR API for text extraction. Document type detection may inform specific OCR prompting.
-4.  **Structuring:** The raw OCR output is processed to extract structured information (e.g., paragraphs, headings, metadata) based on document type and potentially user instructions.
-5.  **Output:** Users can view the extracted text and download structured transcripts and analysis.
+*   **Accuracy:** The primary goal is to maximize the accuracy of the extracted text.
+*   **Configurability:** Users should be able to tailor the processing steps and parameters to their specific document types and needs.
+*   **Transparency:** The tool should provide feedback on the processing steps and allow for debugging (e.g., viewing intermediate images, logs).
+*   **Performance:** Processing should be reasonably efficient, especially for batch operations.
+*   **Ease of Use:** While powerful, the tool should be approachable, whether through a command-line interface or a potential GUI. Configuration should be clear and well-documented.
 
-## User Experience Goals
+## 5. How it Should Work (High-Level Flow)
 
-*   **Intuitive Interface:** A clean, straightforward Streamlit web application that is easy for researchers to use.
-*   **Clear Feedback:** Provide users with status updates during processing and clear presentation of results.
-*   **Flexibility:** Allow users some control over the process (e.g., contextual instructions) where appropriate.
-*   **Reliability:** Ensure consistent and accurate results.
+1.  User provides input file(s) (image or PDF).
+2.  User specifies configuration options (or uses defaults).
+3.  The tool executes the configured pipeline:
+    *   Preprocessing (optional steps like deskew, binarize).
+    *   Segmentation (detecting text regions).
+    *   OCR (applying Tesseract or another engine).
+    *   Post-processing (e.g., text correction, structuring output).
+4.  The tool outputs the extracted text in the desired format (plain text, hOCR, JSON).
+5.  Logs and potentially intermediate results are generated for review.
+
+*(This context provides the 'why' and 'how' from a user perspective. Technical details are in systemPatterns.md and techContext.md.)*

memory-bank/progress.md

@@ -1,34 +1,30 @@
-# Project Progress
+# Project Progress: HOCR Processing Tool
 
-## Current Status
+## 1. What Works
 
-*   **Overall:** Initial project setup phase. Core application structure exists, and foundational documentation (Memory Bank, `.clinerules`) is being established.
-*   **Memory Bank:** The six core Memory Bank files have just been created with initial content derived from the project structure and rules.
+*   **Initial Memory Bank Setup:** The core documentation structure (projectbrief, productContext, activeContext, systemPatterns, techContext, progress) has been established in the `memory-bank/` directory.
 
-## What Works
+## 2. What's Left to Build / Verify
 
-*   The basic file and directory structure for a modular Python/Streamlit application is in place.
-*   Project rules (`.clinerules`) defining the brief, API usage, simplicity principle, and technical debt priorities exist.
-*   The Memory Bank system is now initialized.
+*   **Verify Core Functionality:** Need to run the application and test its basic OCR capabilities on sample images and PDFs.
+*   **Confirm Technical Assumptions:** Validate the libraries and dependencies outlined in `techContext.md` by checking `requirements.txt` and relevant code sections.
+*   **Understand Configuration:** Investigate `config.py` to determine how users configure the pipeline.
+*   **Test UI Layer:** If `app.py` provides a UI (Streamlit/Flask), test its usability and connection to the backend pipeline.
+*   **Review Existing Code:** Deeper dive into the modules (`preprocessing.py`, `ocr_processing.py`, etc.) to understand implementation details.
+*   **Assess Test Coverage:** Examine the tests in `testing/` to understand what is currently covered.
+*   **Address Specific User Goals:** Once the baseline is understood, tackle any specific feature requests, bug fixes, or improvements requested by the user.
 
-## What's Left to Build / Next Steps
+## 3. Current Status
 
-*   Implement the actual functionality described in the project brief (file upload, preprocessing, OCR integration, structuring, UI interactions).
-*   Address the technical debt items listed below.
-*   Refine and expand Memory Bank documentation as development progresses.
-*   Specific next development tasks are pending user direction.
+*   **Baseline Established (Memory Bank):** As of 2025-05-05, the initial Memory Bank structure is in place.
+*   **Code Functionality:** The operational status of the HOCR tool itself is yet to be verified.
 
-## Known Issues / Technical Debt
+## 4. Known Issues / Bugs
 
-The following technical debt items have been identified in `.clinerules/technical-debt.md` and should be addressed during development:
+*   *(None identified yet. To be populated as testing and development proceed.)*
 
-1.  **Modularize large functions:** Break down functions exceeding 100 lines into smaller, focused units.
-2.  **Consistent Error Handling:** Implement a uniform error handling strategy across all modules.
-3.  **Preprocessing Pipeline Improvement:** Enhance the preprocessing steps to better handle diverse historical document types.
-4.  **Image Segmentation Enhancement:** Improve the current approach for identifying text regions.
-5.  **Documentation:** Create comprehensive documentation (docstrings, comments) for public functions and APIs.
+## 5. Evolution of Project Decisions (Decision Log)
 
-## Evolution of Project Decisions
+*   **2025-05-05:** Decided to create the standard Cline Memory Bank structure for the `hocr` project upon user request to check configuration. Found no existing `memory-bank` directory and proceeded with creation of core files.
 
-*   **[YYYY-MM-DD]:** Initialized Memory Bank structure based on global rules.
-*   **[YYYY-MM-DD]:** Adopted a post-task review workflow to iteratively update `.clinerules` and the Memory Bank.
+*(This document tracks the overall progress and state of the project.)*

memory-bank/project-brief.md

@@ -1,19 +0,0 @@
-# Project Brief
-
-Historical OCR is an advanced optical character recognition (OCR) application designed to support historical research. It leverages Mistral AI's OCR models alongside image preprocessing pipelines optimized for archival material.
-
-## High-Level Overview
-
-Building a Streamlit-based web application to process historical documents (images or PDFs), optimize them for OCR using advanced preprocessing techniques, and extract structured text and metadata through Mistral's large language models.
-
-## Core Requirements and Goals
-
-* Upload and preprocess historical documents
-
-* Apply tailored OCR prompting and structured output based on document type
-
-* Support user-defined contextual instructions to refine output
-
-* Provide downloadable structured transcripts and analysis
-
-* Example: "Building a Streamlit web app for OCR transcription and structured extraction from historical documents using Mistral AI."

memory-bank/projectbrief.md

@@ -1,21 +1,39 @@
-# Project Brief
+# Project Brief: HOCR Processing Tool
 
-Historical OCR is an advanced optical character recognition (OCR) application designed to support historical research. It leverages Mistral AI's OCR models alongside image preprocessing pipelines optimized for archival material.
+## 1. Project Goal
 
-High-Level Overview
+*   **Primary Objective:** To develop and maintain a robust tool for performing Optical Character Recognition (OCR) on various document types (images, PDFs), extracting structured text, and handling potential complexities like diverse layouts, languages, and image quality issues.
+*   **Target Users:** Researchers, archivists, developers, or anyone needing to extract text from scanned documents or images.
 
-Building a Streamlit-based web application to process historical documents (images or PDFs), optimize them for OCR using advanced preprocessing techniques, and extract structured text and metadata through Mistral's large language models.
+## 2. Core Requirements
 
-Core Requirements and Goals
+*   **Input:** Accept image files (PNG, JPG, TIFF, etc.) and PDF documents.
+*   **Processing Pipeline:**
+    *   Image Preprocessing (deskewing, noise reduction, binarization, etc.)
+    *   Layout Analysis / Image Segmentation (identifying text blocks, columns, images)
+    *   OCR Engine Integration (e.g., Tesseract)
+    *   Language Detection
+    *   Structured Output Generation (e.g., hOCR format, JSON, plain text)
+    *   Error Handling and Logging
+*   **Configuration:** Allow users to configure processing parameters (e.g., language, preprocessing steps, output format).
+*   **Extensibility:** Design for potential future enhancements (e.g., handwriting recognition, specific template handling).
 
-Upload and preprocess historical documents
+## 3. Scope
 
-Automatically detect document types (e.g., handwritten letters, scientific papers)
+*   **In Scope:** Core OCR pipeline, configuration management, basic UI (if applicable), testing framework, documentation.
+*   **Out of Scope (Initially):** Advanced AI-driven layout analysis beyond standard segmentation, real-time processing for video streams, integration with specific external databases or workflows unless specified.
 
-Apply tailored OCR prompting and structured output based on document type
+## 4. Key Technologies (Initial Assessment - *To be refined in techContext.md*)
 
-Support user-defined contextual instructions to refine output
+*   **Language:** Python
+*   **Core Libraries:** OpenCV, Tesseract (pytesseract), Pillow, pdf2image, potentially others for specific tasks.
+*   **Framework (if UI exists):** Flask/Streamlit (based on existing files like `app.py`, `ui/`)
 
-Provide downloadable structured transcripts and analysis
+## 5. Success Metrics
 
-Example: "Building a Streamlit web app for OCR transcription and structured extraction from historical documents using Mistral AI."
+*   Accuracy of text extraction (measured against ground truth data).
+*   Robustness across different document types and qualities.
+*   Ease of configuration and use.
+*   Maintainability and extensibility of the codebase.
+
+*(This is a foundational brief. Details will be expanded in other Memory Bank documents.)*

memory-bank/systemPatterns.md

@@ -1,66 +1,79 @@
-# System Patterns
+# System Patterns: HOCR Processing Tool
 
-## Architecture Overview
+## 1. High-Level Architecture
 
-The application follows a modular Python structure, orchestrated by a main Streamlit application script (`app.py`). Key architectural components include:
+*   **Modular Pipeline:** The system appears structured as a pipeline with distinct modules for different stages of OCR processing. Key modules suggested by filenames include:
+    *   `preprocessing.py`: Handles initial image adjustments.
+    *   `image_segmentation.py`: Identifies regions of interest (text blocks).
+    *   `ocr_processing.py`: Manages the core OCR engine interaction.
+    *   `language_detection.py`: Determines the language of the text.
+    *   `pdf_ocr.py`: Specific handling for PDF inputs.
+    *   `structured_ocr.py`: Likely involved in formatting the output.
+*   **Configuration Driven:** `config.py` suggests a centralized configuration management approach, allowing pipeline behavior to be customized.
+*   **Entry Point / Orchestration:** `app.py` likely serves as the main entry point or orchestrator, possibly for a web UI or API, coordinating the pipeline execution based on user input and configuration. `process_file.py` might be an alternative entry point or a core processing function called by `app.py`.
+*   **UI Layer:** The `ui/` directory (`ui/layout.py`, `ui/ui_components.py`) indicates a dedicated user interface layer, possibly built with Streamlit or Flask (as suggested in `projectbrief.md`).
+*   **Utility Functions:** The `utils/` directory (`utils/image_utils.py`, `utils/text_utils.py`, etc.) points to a pattern of encapsulating reusable helper functions.
+*   **Error Handling:** `error_handler.py` suggests a dedicated mechanism for managing and reporting errors during processing.
 
-*   **Entry Point:** `app.py` likely initializes the Streamlit application and coordinates calls to other modules.
-*   **UI Layer:** Managed by Streamlit. Core UI elements are defined in `ui/layout.py` and potentially reusable components in `ui/ui_components.py` (or the root `ui_components.py`). Custom styling is applied via `ui/custom.css`.
-*   **Processing Modules:** Functionality is separated into distinct Python modules:
-    *   `preprocessing.py`: Handles image optimization and preparation for OCR.
-    *   `image_segmentation.py`: Deals with identifying regions of interest within documents.
-    *   `ocr_processing.py`: Manages the interaction with the OCR engine (Mistral API).
-    *   `structured_ocr.py`: Focuses on interpreting raw OCR output and structuring it.
-    *   `language_detection.py`: Detects the language of the document content.
-    *   `letterhead_handler.py`: Specific logic for dealing with letterheads.
-    *   `pdf_ocr.py`: Handles OCR specific to PDF inputs (likely coordinating other modules).
-    *   `process_file.py`: A potential high-level orchestrator for the entire file processing pipeline.
-*   **Configuration:** `config.py` likely holds application settings, potentially including API keys or processing parameters. `constants.py` holds fixed values used across the application.
-*   **Utilities:** Common functions are grouped in the `utils/` directory, further categorized (e.g., `image_utils.py`, `text_utils.py`, `file_utils.py`).
-*   **Error Handling:** A dedicated `error_handler.py` suggests a centralized approach to managing exceptions.
+## 2. Key Design Patterns (Inferred)
 
-## Key Technical Decisions & Patterns
+*   **Pipeline Pattern:** The core processing flow seems to follow a pipeline pattern, where data (image/document) passes through sequential processing stages.
+*   **Configuration Management:** Centralized configuration (`config.py`) allows for decoupling settings from code.
+*   **Separation of Concerns:** Different functionalities (UI, core processing, utilities, configuration) appear to be separated into distinct modules/files.
+*   **Utility/Helper Modules:** Common, reusable functions are grouped into utility modules.
 
-*   **Modularity:** Code is organized into feature-specific modules, promoting separation of concerns.
-*   **External API Integration:** Relies on the Mistral AI OCR API for core text extraction (`.clinerules/hocr-basics-api.md`). API interaction logic is likely within `ocr_processing.py` or related utilities.
-*   **Streamlit Framework:** Leverages Streamlit for the web interface, using standard components (`.clinerules/hocr-basics-api.md`). State management likely uses `st.session_state`.
-*   **Content Purity:** Adheres to the principle of separating data from presentation markup (`.clinerules/principle-of-simplicity.md`). Presentation logic should reside primarily in the UI layer.
-*   **Configuration Management:** Centralized configuration likely managed through `config.py`.
-
-## Component Relationships (Conceptual)
+## 3. Component Relationships (Initial Diagram - Mermaid)
 
 ```mermaid
 graph TD
-    A[User via Streamlit UI] --> B(app.py);
-    B --> C{process_file.py?};
-    C --> D[preprocessing.py];
-    C --> E[image_segmentation.py];
-    C --> F[language_detection.py];
-    C --> G[pdf_ocr.py / ocr_processing.py];
-    G -- Mistral API --> H((External Mistral OCR));
-    H -- OCR Result --> G;
-    G --> I[structured_ocr.py];
-    I --> J[Output Generation];
-    J --> A;
-
-    subgraph Modules
-        D; E; F; G; I; J;
+    subgraph User Interface / Entry Point
+        A[app.py / UI Layer] --> B(process_file.py);
+    end
+
+    subgraph Configuration
+        C[config.py];
+    end
+
+    subgraph Core OCR Pipeline
+        B --> D(preprocessing.py);
+        D --> E(image_segmentation.py);
+        E --> F(ocr_processing.py);
+        F --> G(language_detection.py);
+        G --> H(structured_ocr.py);
     end
 
-    subgraph Configuration & Utils
-        K[config.py];
-        L[constants.py];
-        M[utils/];
-        N[error_handler.py];
+    subgraph Input Handling
+        I[pdf_ocr.py] --> B;
+        J[Image Input] --> B;
     end
 
-    B --> K; B --> L; B --> M; B --> N;
-    Modules --> K; Modules --> L; Modules --> M; Modules --> N;
+    subgraph Utilities
+        K[utils/];
+        L[error_handler.py];
+    end
+
+    A --> C;
+    B --> C;
+    D --> K;
+    E --> K;
+    F --> K;
+    G --> K;
+    H --> K;
+    I --> K;
+    B --> L;
+
+    style User Interface / Entry Point fill:#f9f,stroke:#333,stroke-width:2px
+    style Configuration fill:#ccf,stroke:#333,stroke-width:2px
+    style Core OCR Pipeline fill:#cfc,stroke:#333,stroke-width:2px
+    style Input Handling fill:#ffc,stroke:#333,stroke-width:2px
+    style Utilities fill:#eee,stroke:#333,stroke-width:2px
+
 ```
 
-## Critical Implementation Paths
+## 4. Critical Implementation Paths
+
+*   **Image Input -> Preprocessing -> Segmentation -> OCR -> Structured Output:** The main flow for image files.
+*   **PDF Input -> PDF Extraction -> Image Conversion (per page) -> [Main Flow] -> Aggregated Output:** The likely path for PDF documents.
+*   **Configuration Loading -> Pipeline Execution:** How settings influence the process.
 
-*   The end-to-end flow from file upload (`st.file_uploader`) through preprocessing, OCR API call, structuring, and displaying results (`st.markdown`, `st.download_button`).
-*   Handling different input file types (Images vs. PDFs).
-*   Integration and error handling for the Mistral API calls.
-*   Implementation of specific preprocessing steps relevant to historical documents.
+*(This document outlines the observed structure. It will be refined as the codebase is analyzed in more detail.)*

memory-bank/techContext.md

@@ -1,35 +1,37 @@
-# Technical Context
+# Technical Context: HOCR Processing Tool
 
-## Technologies Used
+## 1. Core Language
 
-*   **Primary Language:** Python 3.x
-*   **Web Framework:** Streamlit
-*   **Core API:** Mistral AI OCR API (via HTTPS requests)
-*   **Potential Libraries:**
-    *   `requests`: For making HTTP calls to the Mistral API.
-    *   `streamlit`: For the web UI framework.
-    *   `Pillow` (PIL Fork): For basic image loading and manipulation.
-    *   `OpenCV` (`cv2`): Likely used for more advanced image preprocessing tasks (e.g., thresholding, noise reduction, deskewing).
-    *   `python-dotenv`: Potentially used for managing environment variables like API keys (especially if `config.py` loads from a `.env` file).
-    *   `PyMuPDF` or similar: If PDF processing involves direct text/image extraction from PDF structures beyond just sending to OCR.
+*   **Python:** The project is primarily written in Python, as indicated by the `.py` files. The specific version should be confirmed (e.g., via `requirements.txt` or environment setup).
 
-*(Note: Specific libraries beyond Streamlit and requests need confirmation, e.g., by inspecting `requirements.txt` or import statements in the code).*
+## 2. Key Libraries & Frameworks
 
-## Development Setup
+*   **OCR Engine:** Likely **Tesseract OCR**, potentially accessed via the `pytesseract` wrapper (common practice). This needs confirmation by inspecting `ocr_processing.py` or dependencies.
+*   **Image Processing:** **OpenCV (`cv2`)** and/or **Pillow (PIL)** are highly probable for tasks in `preprocessing.py` and `image_segmentation.py`.
+*   **PDF Handling:** **`pdf2image`** (which often relies on Poppler) is a common choice for converting PDF pages to images, relevant for `pdf_ocr.py`. Other PDF libraries like PyMuPDF or PyPDF2 might also be used.
+*   **Web Framework/UI:** Based on `app.py` and the `ui/` directory, **Flask** or **Streamlit** are potential candidates for the user interface or API layer.
+*   **Configuration:** Standard Python mechanisms (e.g., `.ini` files with `configparser`, `.json` files, or custom Python modules like `config.py`).
+*   **Dependency Management:** Likely uses `pip` with a `requirements.txt` file (observed in the file listing). Virtual environments (like `venv` or `conda`) are standard practice.
 
-*   **Environment:** Standard Python environment (virtual environment recommended, e.g., `venv` or `conda`).
-*   **Dependencies:** Install required packages (likely via `pip install -r requirements.txt` if a requirements file exists).
-*   **API Keys:** Requires a Mistral AI API key, which needs to be configured securely (likely via environment variables loaded in `config.py`).
-*   **Running the App:** Typically run using `streamlit run app.py` from the project root directory.
+## 3. External Dependencies & Setup
 
-## Technical Constraints
+*   **Tesseract OCR Engine:** Requires separate installation on the host system. The path to the Tesseract executable might need configuration.
+*   **Poppler:** Often required by `pdf2image` for PDF processing; needs separate installation.
+*   **Python Environment:** A specific Python version and installed packages via `requirements.txt`.
+*   **Environment Variables:** Potential use of environment variables for configuration (e.g., API keys, paths), possibly managed via a `.env` file (observed in the file listing).
 
-*   **API Limits:** Subject to Mistral AI API usage limits, rate limits, and potential costs. Error handling for API responses (e.g., 429 Too Many Requests, 401 Unauthorized, 5xx Server Errors) is crucial.
-*   **Processing Time:** OCR and complex image preprocessing can be time-consuming, especially for large documents or high-resolution images. Streamlit's execution model needs to be considered for long-running tasks (e.g., using background processes or providing user feedback).
-*   **Resource Usage:** Image processing can be memory and CPU intensive.
+## 4. Development Environment
 
-## Tool Usage Patterns
+*   **Standard Python Setup:** Requires a Python interpreter, `pip`, and likely `virtualenv`.
+*   **Code Editor/IDE:** VS Code is being used (based on environment details). Settings might be stored in `.vscode/`.
+*   **Version Control:** Git is likely used (indicated by `.gitignore`, `.gitattributes`). The `.git_disabled` directory suggests Git might have been temporarily disabled or renamed.
+*   **Testing:** The `testing/` directory and `pytest_cache` suggest **pytest** is used for running tests.
 
-*   **Streamlit Components:** Primarily use core components as specified in `.clinerules/hocr-basics-api.md` (`st.file_uploader`, `st.selectbox`, `st.image`, `st.markdown`, `st.download_button`).
-*   **State Management:** Use `st.session_state` for managing user interactions and state across reruns.
-*   **API Interaction:** Follow standard practices for REST API calls (headers, JSON body, error checking) as defined in `.clinerules/hocr-basics-api.md`.
+## 5. Technical Constraints & Considerations
+
+*   **Performance:** OCR, especially on large documents or batches, can be computationally intensive. Image processing steps add overhead.
+*   **Tesseract Limitations:** Tesseract's accuracy depends heavily on image quality, preprocessing, and language model availability.
+*   **Dependency Hell:** Managing Python dependencies and external binaries (Tesseract, Poppler) across different operating systems can be challenging.
+*   **Layout Complexity:** Handling complex layouts (multi-column, tables, embedded images) requires sophisticated segmentation logic.
+
+*(This document provides an initial technical overview based on file structure and common practices. It requires verification by examining code and configuration files like requirements.txt.)*