Enhance README and Streamlit app for Medical Document Parser & Redactor

- Added detailed project overview and structure in README.md.
- Implemented file upload, processing, and redaction features in streamlit_app.py.
- Integrated logging and temporary file management.
- Enhanced UI with custom styling and synchronized scrolling for document comparison.
- Added functionality for viewing original and redacted documents side by side, including download options for markdown and JSON structures.

.gitignore

@@ -0,0 +1,63 @@
+# Temporary files
+temp_files/
+temp_*.pdf
+temp_*.json
+temp_*.txt
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+# Logs
+*.log
+
+# Streamlit
+.streamlit/
+
+# Azure credentials (if any)
+*.pem
+*.key 

.python-version

@@ -0,0 +1 @@
+3.13

README.md

@@ -17,3 +17,337 @@ Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :hear
 
 If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
 forums](https://discuss.streamlit.io).
+
+# Medical Document Parser & Redactor
+
+A sophisticated medical document processing application that uses **Docling** (structure-aware parser) to parse PDF medical documents and automatically redact medication information using AI-powered analysis.
+
+## 🎯 Overview
+
+This application provides a Streamlit-based interface for uploading medical PDF documents, parsing them with Docling to extract structured content, and using Azure OpenAI to intelligently identify and redact formal medication lists while preserving clinical context.
+
+## 🏗️ Project Structure
+
+```
+docling/
+├── src/                          # Main source code
+│   ├── processing/               # Core processing logic
+│   │   ├── __init__.py
+│   │   ├── document_processor.py # Main document processing pipeline
+│   │   ├── llm_extractor.py      # Azure OpenAI integration for medication detection
+│   │   └── sections.py           # Section extraction and redaction logic
+│   ├── utils/                    # Utility functions
+│   │   ├── __init__.py
+│   │   └── logging_utils.py      # Logging configuration and handlers
+│   └── streamlit_app.py          # Main Streamlit application interface
+├── temp_files/                   # Temporary file storage (auto-created)
+├── .env                          # Environment variables (Azure OpenAI credentials)
+├── requirements.txt              # Python dependencies
+├── pyproject.toml               # Project configuration
+├── Dockerfile                   # Container configuration
+└── README.md                    # This file
+```
+
+## 📁 File Responsibilities
+
+### Core Processing Files
+
+#### `src/processing/document_processor.py`
+**Purpose**: Main document processing pipeline that orchestrates the entire workflow.
+
+**Key Classes**:
+- `DocumentResult`: Data class holding processed results
+- `DocumentProcessor`: Main processing class
+
+**Key Functions**:
+- `process(file_path)`: Main processing method
+- `_export_redacted_markdown()`: Generates redacted markdown
+- `_reconstruct_markdown_from_filtered_texts()`: Reconstructs markdown from filtered content
+
+**Responsibilities**:
+- Document conversion using Docling
+- Section redaction coordination
+- Markdown generation and reconstruction
+- File persistence and logging
+
+#### `src/processing/llm_extractor.py`
+**Purpose**: Azure OpenAI integration for intelligent medication detection.
+
+**Key Classes**:
+- `AzureO1MedicationExtractor`: LLM-based medication extractor
+
+**Key Functions**:
+- `extract_medication_sections(doc_json)`: Main extraction method
+- `__init__()`: Azure OpenAI client initialization
+
+**Responsibilities**:
+- Azure OpenAI API communication
+- Medication section identification
+- Structured JSON response generation
+- Error handling and logging
+
+#### `src/processing/sections.py`
+**Purpose**: Section extraction and redaction logic.
+
+**Key Classes**:
+- `ReasoningSectionExtractor`: AI-powered section extractor
+- `SectionDefinition`: Section definition data class
+- `SectionExtractor`: Traditional regex-based extractor
+
+**Key Functions**:
+- `remove_sections_from_json()`: JSON-based section removal
+- `remove_sections()`: Text-based section removal (fallback)
+
+**Responsibilities**:
+- Section identification and removal
+- JSON structure manipulation
+- Text processing and redaction
+- Reasoning logging and transparency
+
+### Interface Files
+
+#### `src/streamlit_app.py`
+**Purpose**: Main Streamlit web application interface.
+
+**Key Functions**:
+- `save_uploaded_file()`: File upload handling
+- `cleanup_temp_files()`: Temporary file management
+- `create_diff_content()`: Diff view generation
+
+**Responsibilities**:
+- User interface and interaction
+- File upload and management
+- Visualization and diff display
+- Session state management
+- Download functionality
+
+### Utility Files
+
+#### `src/utils/logging_utils.py`
+**Purpose**: Logging configuration and management.
+
+**Key Functions**:
+- `get_log_handler()`: Creates in-memory log handlers
+- Log buffer management for UI display
+
+**Responsibilities**:
+- Logging setup and configuration
+- In-memory log capture
+- Log display in UI
+
+## 🔧 Detailed Function Documentation
+
+### Document Processing Pipeline
+
+#### `DocumentProcessor.process(file_path: str) -> DocumentResult`
+**Purpose**: Main entry point for document processing.
+
+**Parameters**:
+- `file_path`: Path to the PDF file to process
+
+**Returns**:
+- `DocumentResult`: Object containing all processing results
+
+**Process Flow**:
+1. Converts PDF using Docling
+2. Exports structured markdown and JSON
+3. Applies section redaction if extractor is provided
+4. Persists results to temporary files
+5. Returns comprehensive result object
+
+**Example Usage**:
+```python
+processor = DocumentProcessor(section_extractor=extractor)
+result = processor.process("document.pdf")
+print(f"Original: {len(result.structured_markdown)} chars")
+print(f"Redacted: {len(result.redacted_markdown)} chars")
+```
+
+#### `AzureO1MedicationExtractor.extract_medication_sections(doc_json: Dict) -> Dict`
+**Purpose**: Uses Azure OpenAI to identify medication sections for redaction.
+
+**Parameters**:
+- `doc_json`: Docling-generated JSON structure
+
+**Returns**:
+- Dictionary with indices to remove and reasoning
+
+**Process Flow**:
+1. Analyzes document structure
+2. Sends structured prompt to Azure OpenAI
+3. Parses JSON response
+4. Validates and limits results
+5. Returns structured analysis
+
+**Example Usage**:
+```python
+extractor = AzureO1MedicationExtractor(endpoint, api_key, version, deployment)
+result = extractor.extract_medication_sections(doc_json)
+print(f"Removing {len(result['indices_to_remove'])} elements")
+```
+
+#### `ReasoningSectionExtractor.remove_sections_from_json(doc_json: Dict) -> Dict`
+**Purpose**: Removes identified sections from JSON structure.
+
+**Parameters**:
+- `doc_json`: Original document JSON structure
+
+**Returns**:
+- Redacted JSON structure
+
+**Process Flow**:
+1. Calls LLM extractor for analysis
+2. Logs detailed reasoning
+3. Removes identified text elements
+4. Updates document structure
+5. Returns redacted JSON
+
+### UI and Visualization Functions
+
+#### `create_diff_content(original_text: str, redacted_text: str, view_type: str) -> str`
+**Purpose**: Generates HTML content for side-by-side diff view.
+
+**Parameters**:
+- `original_text`: Original document content
+- `redacted_text`: Redacted document content
+- `view_type`: 'original' or 'redacted'
+
+**Returns**:
+- HTML string for diff display
+
+**Features**:
+- Text normalization (headers, quotes)
+- Synchronized scrolling
+- Color-coded highlighting
+- Git-style diff visualization
+
+#### `save_uploaded_file(uploaded_file, filename) -> str`
+**Purpose**: Safely saves uploaded files to temporary directory.
+
+**Parameters**:
+- `uploaded_file`: Streamlit uploaded file object
+- `filename`: Target filename
+
+**Returns**:
+- Path to saved temporary file
+
+**Features**:
+- File pointer reset handling
+- Temporary directory management
+- Error handling and logging
+
+## 🔄 Sequence Diagram
+
+```
+User Uploads PDF
+       │
+       ▼
+┌─────────────────┐
+│ Streamlit App   │
+│ - File Upload   │
+│ - Validation    │
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ save_uploaded_  │
+│ file()          │
+│ - Reset pointer │
+│ - Save to temp  │
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ DocumentProcessor│
+│ .process()      │
+│ - Docling conv  │
+│ - Export JSON   │
+│ - Export MD     │
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ ReasoningSection│
+│ Extractor       │
+│ .remove_sections│
+│ _from_json()    │
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ AzureO1Medication│
+│ Extractor       │
+│ .extract_medicat│
+│ ion_sections()  │
+│ - API call      │
+│ - JSON parsing  │
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ DocumentProcessor│
+│ _export_redacted│
+│ _markdown()     │
+│ - Filter texts  │
+│ - Reconstruct MD│
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ Streamlit App   │
+│ - Store results │
+│ - Update UI     │
+│ - Show diff     │
+└─────────────────┘
+       │
+       ▼
+┌─────────────────┐
+│ create_diff_    │
+│ content()       │
+│ - Normalize text│
+│ - Generate HTML │
+│ - Sync scrolling│
+└─────────────────┘
+```
+
+## 🚀 Setup and Installation
+
+### Prerequisites
+- Python 3.11+
+- Azure OpenAI account with API access
+- Docling library
+
+### Environment Variables
+Create a `.env` file with:
+```env
+AZURE_OPENAI_ENDPOINT=your_endpoint
+AZURE_OPENAI_KEY=your_api_key
+AZURE_OPENAI_VERSION=2024-12-01-preview
+AZURE_OPENAI_DEPLOYMENT=your_deployment_name
+```
+
+### Installation
+```bash
+# Clone repository
+git clone <repository-url>
+cd docling
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Run application
+streamlit run src/streamlit_app.py
+```
+
+## 🔧 Configuration
+
+### Azure OpenAI Settings
+- **Model**: O1-mini (recommended for medication extraction)
+- **Max Tokens**: 100,000 (for large documents)
+- **Temperature**: 0 (for consistent results)
+
+### Processing Settings
+- **Max Elements to Remove**: 10 (safety limit)
+- **Temp Directory**: `temp_files/` (auto-created)
+- **Log Level**: INFO (configurable)
+

main.py

@@ -0,0 +1,6 @@
+def main():
+    print("Hello from docling!")
+
+
+if __name__ == "__main__":
+    main()

pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "docling-app"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.13"
+dependencies = [
+    "streamlit>=1.46.0",
+    "docling>=2.38.0",
+    "pyyaml>=6.0",
+    "python-dotenv>=1.1.1",
+    "openai>=1.91.0",
+]

src/processing/document_processor.py

@@ -0,0 +1,140 @@
+import os
+import time
+import logging
+import json
+from dataclasses import dataclass
+from typing import Optional
+from docling.document_converter import DocumentConverter
+from processing.sections import SectionExtractor
+
+# Initialize a Docling converter globally (can be reused for multiple docs)
+_docling_converter = DocumentConverter()
+
+logger = logging.getLogger(__name__)  # Logger for this module
+
+@dataclass
+class DocumentResult:
+    """Holds processed results for a document."""
+    file_path: str
+    structured_markdown: str
+    structured_json: dict
+    redacted_markdown: str
+    redacted_json: dict
+
+class DocumentProcessor:
+    """Handles parsing of documents with Docling and redacting specified sections."""
+    def __init__(self, section_extractor: Optional[SectionExtractor] = None):
+        """
+        Initialize with an optional SectionExtractor for removing specific sections.
+        If None, no redaction will be performed (original structure only).
+        The Docling DocumentConverter is taken as a dependency (global or injected).
+        """
+        self.section_extractor = section_extractor
+        # Allow dependency injection of converter if needed (use global by default)
+        self.converter = _docling_converter
+    
+    def process(self, file_path: str) -> DocumentResult:
+        """Parse the document and optionally remove specified sections. Returns a DocumentResult."""
+        logger.info(f"Starting processing for file: {file_path}")
+        start_time = time.time()
+        # Convert the document using Docling
+        conv_result = self.converter.convert(file_path)
+        elapsed = time.time() - start_time
+        logger.info(f"Docling conversion completed in {elapsed:.2f} seconds")
+        
+        # Export results from Docling
+        structured_md = conv_result.document.export_to_markdown()
+        structured_text = conv_result.document.export_to_text()
+        doc_json = conv_result.document.export_to_dict()
+        logger.info(f"Extracted document content (text length {len(structured_text)} characters)")
+        
+        # Use SectionExtractor to remove target sections if provided
+        if self.section_extractor:
+            # Use the new JSON-based approach for better section removal
+            redacted_json = self.section_extractor.remove_sections_from_json(doc_json)
+            
+            # Convert the redacted JSON back to markdown using Docling's export method
+            # Create a modified document structure for proper markdown export
+            redacted_md = self._export_redacted_markdown(conv_result.document, redacted_json)
+            logger.info("Applied section redaction to remove specified sections")
+        else:
+            redacted_md = structured_md  # No redaction, use original
+            redacted_json = doc_json  # No redaction, use original
+            logger.info("No section redaction applied (showing original structure)")
+        
+        # Persist outputs to files (JSON and redacted text) for auditing
+        base_name = os.path.splitext(os.path.basename(file_path))[0]
+        # Use temp directory for output files
+        temp_dir = "temp_files"
+        os.makedirs(temp_dir, exist_ok=True)
+        
+        json_path = os.path.join(temp_dir, f"{base_name}_structured.json")
+        redacted_path = os.path.join(temp_dir, f"{base_name}_redacted.txt")
+        redacted_json_path = os.path.join(temp_dir, f"{base_name}_redacted.json")
+        
+        try:
+            with open(json_path, "w", encoding="utf-8") as jf:
+                json.dump(doc_json, jf, ensure_ascii=False, indent=2)
+            with open(redacted_path, "w", encoding="utf-8") as tf:
+                tf.write(redacted_md)
+            with open(redacted_json_path, "w", encoding="utf-8") as jf:
+                json.dump(redacted_json, jf, ensure_ascii=False, indent=2)
+            logger.info(f"Saved structured JSON to {json_path}, redacted text to {redacted_path}, and redacted JSON to {redacted_json_path}")
+        except Exception as e:
+            logger.error(f"Error saving outputs to files: {e}")
+        
+        # Prepare result object
+        result = DocumentResult(
+            file_path=file_path,
+            structured_markdown=structured_md,
+            structured_json=doc_json,
+            redacted_markdown=redacted_md,
+            redacted_json=redacted_json
+        )
+        logger.info(f"Finished processing for file: {file_path}")
+        return result
+
+    def _export_redacted_markdown(self, document, redacted_json):
+        """Export redacted markdown using the redacted JSON structure."""
+        # Simply convert the redacted JSON back to markdown
+        return self._json_to_markdown(redacted_json)
+    
+    def _json_to_markdown(self, json_data: dict) -> str:
+        """Convert JSON document structure back to markdown format using Docling's structure."""
+        markdown_lines = []
+        
+        # Get all text elements from the JSON
+        texts = json_data.get("texts", [])
+        
+        for text_elem in texts:
+            text_content = text_elem.get("text", "")
+            label = text_elem.get("label", "")
+            level = text_elem.get("level", 0)
+            
+            if not text_content.strip():
+                continue
+                
+            # Format based on the label and level (following Docling's structure)
+            if label == "section_header":
+                # Add appropriate markdown headers
+                if level == 1:
+                    markdown_lines.append(f"# {text_content}")
+                elif level == 2:
+                    markdown_lines.append(f"## {text_content}")
+                elif level == 3:
+                    markdown_lines.append(f"### {text_content}")
+                else:
+                    markdown_lines.append(f"#### {text_content}")
+            elif label == "list_item":
+                # Handle list items - preserve the original marker
+                marker = text_elem.get("marker", "-")
+                markdown_lines.append(f"{marker} {text_content}")
+            elif label == "text":
+                # Regular text content - preserve as-is
+                markdown_lines.append(text_content)
+            else:
+                # Default to regular text
+                markdown_lines.append(text_content)
+        
+        # Join without extra spacing to match Docling's formatting
+        return "\n".join(markdown_lines)

src/processing/llm_extractor.py

@@ -0,0 +1,145 @@
+# src/processing/llm_extractor.py
+
+
+import json
+import logging
+from typing import Dict, Any
+
+from openai import AzureOpenAI
+
+logger = logging.getLogger(__name__)
+
+class AzureO1MedicationExtractor:
+    def __init__(
+        self,
+        endpoint: str,
+        api_key: str,
+        api_version: str,
+        deployment: str,
+        model_name: str = None,
+    ):
+        self.client = AzureOpenAI(
+            api_version=api_version,
+            azure_endpoint=endpoint,
+            api_key=api_key,
+        )
+        self.deployment = deployment
+        self.model_name = model_name or deployment
+
+    def extract_medication_sections(self, doc_json: Dict[str, Any]) -> Dict[str, Any]:
+        texts = doc_json.get("texts", [])
+        text_analysis = []
+        for i, text_elem in enumerate(texts):
+            text_analysis.append({
+                "index": i,
+                "text": text_elem.get("text", ""),
+                "label": text_elem.get("label", ""),
+                "level": text_elem.get("level", 0),
+                "parent": text_elem.get("parent", {}),
+            })
+
+        prompt = f"""
+You are a medical document analysis expert specializing in discharge letters. Your task is to identify ONLY the two formal medication lists that should be redacted, while preserving all medication mentions in clinical discussion.
+
+**CRITICAL: You should ONLY remove the two formal medication lists:**
+1. **Current medication list** (usually at the beginning of the document)
+2. **Discharge medication list** (usually at the end of the document, often under headers like "Als verdere behandeling stellen wij voor" or "Thuismedicatie")
+
+**Typical discharge letter structure:**
+- Patient information and admission details
+- Clinical discussion and treatment narrative (KEEP medication mentions here)
+- Current medication list (REMOVE this formal list)
+- Discharge instructions and follow-up
+- Discharge medication list (REMOVE this formal list)
+
+**DO NOT remove:**
+- Medication mentions in clinical discussion (e.g., "patient was treated with Eliquis")
+- Medication adjustments mentioned in the narrative
+- Dosage information in clinical context
+- Any medication information that appears in the main clinical text
+- Treatment decisions and clinical reasoning
+
+**ONLY remove:**
+- Complete medication lists with multiple drugs
+- Formal medication sections with headers
+- Standalone medication lists that are clearly separated from clinical text
+- Lists that appear to be formal medication documentation
+
+Document structure:
+{text_analysis}
+
+**Analysis Instructions:**
+1. Look for formal medication sections with clear headers (e.g., "Thuismedicatie", "Als verdere behandeling stellen wij voor")
+2. Identify complete medication lists that contain multiple drugs with dosages
+3. **IGNORE** any medication mentions that appear within clinical discussion or narrative text
+4. Focus on structural elements that represent formal medication documentation
+5. Be conservative - if in doubt, do NOT remove
+6. Consider the position in the document (beginning/end vs. middle)
+
+**Examples of what to REMOVE:**
+- Complete lists under "Thuismedicatie" header
+- Formal medication lists under "Als verdere behandeling stellen wij voor"
+- Standalone medication sections with multiple drugs
+- Lists that appear at the beginning or end of the document
+
+**Examples of what to KEEP:**
+- "Patient was treated with Eliquis 2x 2.5mg" (clinical discussion)
+- "Stop Clopidogrel bij opname" (clinical instruction)
+- "Jardiance 10mg & Burinex 5mg" (if mentioned in clinical context)
+- Any medication mentioned in the context of treatment discussion
+
+Return your analysis as JSON:
+{{
+    "indices_to_remove": [list of integer indices - ONLY formal medication lists],
+    "reasoning": {{
+        "formal_medication_lists": [list of identified formal medication list indices with explanations],
+        "clinical_medication_mentions": [list of clinical mentions that were correctly preserved],
+        "justification": "explanation of why only formal lists were selected for removal",
+        "confidence": "high/medium/low"
+    }}
+}}
+"""
+        logger.info(f"Prompt length: {len(prompt)}")
+        logger.info(f"Number of text elements: {len(text_analysis)}")
+        try:
+            response = self.client.chat.completions.create(
+            messages=[
+                {
+                    "role": "system",
+                    "content": "You are a helpful assistant.",
+                },
+                {
+                    "role": "user",
+                    "content": prompt,
+                }
+            ],
+            max_completion_tokens=100000,  # adjust as needed
+            model=self.deployment
+            )
+        except Exception as e:
+            logger.error(f"Exception during LLM call: {e}", exc_info=True)
+            return {"indices_to_remove": [], "reasoning": {"confidence": "low"}}
+
+        try:
+            logger.error(f"Raw LLM response: {response.choices[0].message.content!r}")
+            result = json.loads(response.choices[0].message.content)
+            
+            # Validate and limit the number of elements to remove
+            indices_to_remove = result.get("indices_to_remove", [])
+            
+            # Be conservative - limit to maximum 10 elements to prevent over-removal
+            if len(indices_to_remove) > 10:
+                logger.warning(f"LLM suggested removing {len(indices_to_remove)} elements, limiting to 10 most likely formal medication lists")
+                # Keep only the first 10 (assuming they're ordered by importance)
+                indices_to_remove = indices_to_remove[:10]
+                result["indices_to_remove"] = indices_to_remove
+                result["reasoning"]["justification"] += " [LIMITED: Only top 10 elements selected to prevent over-removal]"
+            
+            # Log the reasoning for transparency
+            reasoning = result.get("reasoning", {})
+            logger.info(f"LLM reasoning: {reasoning}")
+            
+            return result
+        except Exception as e:
+            logger.error(f"Failed to parse LLM response: {e}")
+            return {"indices_to_remove": [], "reasoning": {"confidence": "low"}}

src/processing/sections.py

@@ -0,0 +1,222 @@
+import re
+import logging
+from dataclasses import dataclass
+from typing import List, Optional, Dict, Any
+from .llm_extractor import AzureO1MedicationExtractor
+
+logger = logging.getLogger(__name__)
+
+class ReasoningSectionExtractor:
+    def __init__(self, endpoint, api_key, api_version, deployment):
+        self.llm_extractor = AzureO1MedicationExtractor(
+            endpoint=endpoint,
+            api_key=api_key,
+            api_version=api_version,
+            deployment=deployment,
+        )
+
+    def remove_sections_from_json(self, doc_json: Dict[str, Any]) -> Dict[str, Any]:
+        extraction_result = self.llm_extractor.extract_medication_sections(doc_json)
+        indices_to_remove = extraction_result["indices_to_remove"]
+        reasoning = extraction_result.get("reasoning", {})
+        
+        # Log detailed reasoning for transparency
+        logger.info(f"O1-mini reasoning: {reasoning}")
+        
+        # Provide specific feedback about what was removed
+        if indices_to_remove:
+            logger.info(f"Removing {len(indices_to_remove)} text elements: {indices_to_remove}")
+            
+            # Show what specific content is being removed
+            texts = doc_json.get("texts", [])
+            for idx in indices_to_remove:
+                if idx < len(texts):
+                    text_content = texts[idx].get("text", "")[:100]
+                    logger.info(f"  Removing text {idx}: '{text_content}...'")
+        else:
+            logger.info("No formal medication lists identified for removal")
+        
+        # Remove the identified text elements
+        import copy
+        redacted_json = copy.deepcopy(doc_json)
+        texts = redacted_json.get("texts", [])
+        redacted_texts = [t for i, t in enumerate(texts) if i not in indices_to_remove]
+        redacted_json["texts"] = redacted_texts
+        
+        # Log the result
+        removed_count = len(texts) - len(redacted_texts)
+        logger.info(f"Successfully removed {removed_count} text elements from document structure")
+        
+        return redacted_json
+    
+    def remove_sections(self, text: str) -> str:
+        """
+        Remove sections from markdown text. This is a fallback method for compatibility.
+        Since ReasoningSectionExtractor works with JSON structure, this method
+        returns the original text (no redaction) as the JSON-based approach is preferred.
+        """
+        logger.warning("ReasoningSectionExtractor.remove_sections() called - this method is not implemented for text-based redaction. Use remove_sections_from_json() instead.")
+        return text
+
+@dataclass
+class SectionDefinition:
+    """Defines a section to extract/remove by specifying its start (and optional end) regex."""
+    name: str
+    start_pattern: str  # Regex pattern to identify the section start (use multiline anchors as needed)
+    end_pattern: Optional[str] = None  # Regex for section end, or None if it goes until next section or EOF
+
+class SectionExtractor:
+    """Finds and removes specified sections from document content."""
+    def __init__(self, sections: List[SectionDefinition]):
+        # Compile regex patterns for performance
+        self.sections = [
+            SectionDefinition(sec.name, re.compile(sec.start_pattern), re.compile(sec.end_pattern) if sec.end_pattern else None)
+            for sec in sections
+        ]
+    
+    def remove_sections(self, text: str) -> str:
+        """
+        Remove all defined sections from the given text. Returns the redacted text.
+        The text is expected to be the full document content (in Markdown or plain text form).
+        """
+        logger.info("Removing sections from text...")
+        if not self.sections:
+            return text  # nothing to remove
+        
+        to_remove_ranges = []  # will hold (start_index, end_index) for removal
+        
+        # Find all section start positions
+        for sec in self.sections:
+            match = sec.start_pattern.search(text)
+            if match:
+                start_idx = match.start()
+                # Determine end of section
+                if sec.end_pattern:
+                    end_match = sec.end_pattern.search(text, start_idx)
+                    if end_match:
+                        # End pattern found; end index is start of end_match
+                        end_idx = end_match.start()
+                    else:
+                        end_idx = len(text)  # if no end pattern found, remove till end
+                else:
+                    end_idx = len(text)  # default end is end-of-text (will adjust later if there's another section)
+                to_remove_ranges.append((start_idx, end_idx, sec.name))
+                logger.info(f"Marked section '{sec.name}' for removal (positions {start_idx}-{end_idx})")
+            else:
+                logger.info(f"Section '{sec.name}' not found in text (pattern: {sec.start_pattern.pattern})")
+        
+        if not to_remove_ranges:
+            logger.info("No sections to remove.")
+            return text
+        
+        # Sort ranges by start index
+        to_remove_ranges.sort(key=lambda x: x[0])
+        # If sections overlap or touch, adjust ranges to avoid double-counting
+        redacted_text = ""
+        current_idx = 0
+        for start_idx, end_idx, sec_name in to_remove_ranges:
+            # Append text from current_idx up to start_idx (keeping content before section)
+            if current_idx < start_idx:
+                redacted_text += text[current_idx:start_idx]
+            else:
+                # Overlapping section (or consecutive) – already handled by previous removal
+                logger.warning(f"Section '{sec_name}' overlaps with a previous section removal region.")
+            current_idx = max(current_idx, end_idx)
+        # Append any remaining text after last removed section
+        if current_idx < len(text):
+            redacted_text += text[current_idx:]
+        
+        return redacted_text
+
+    def remove_sections_from_json(self, doc_json: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Remove specified sections from the structured JSON document.
+        This method works with the Docling JSON structure to identify and remove
+        sections based on their semantic content rather than just text patterns.
+        """
+        logger.info("Removing sections from structured JSON...")
+        if not self.sections:
+            return doc_json  # nothing to remove
+        
+        # Create a deep copy to avoid modifying the original
+        import copy
+        redacted_json = copy.deepcopy(doc_json)
+        
+        # Get all text elements from the document
+        texts = redacted_json.get("texts", [])
+        if not texts:
+            logger.warning("No texts found in document JSON")
+            return redacted_json
+        
+        # Find text elements that match our section patterns
+        text_indices_to_remove = set()
+        
+        for sec in self.sections:
+            logger.info(f"Looking for section '{sec.name}' with pattern: {sec.start_pattern.pattern}")
+            
+            # Find text elements that match the section start pattern
+            for i, text_elem in enumerate(texts):
+                text_content = text_elem.get("text", "")
+                if sec.start_pattern.search(text_content):
+                    logger.info(f"Found section '{sec.name}' in text element {i}: '{text_content[:50]}...'")
+                    text_indices_to_remove.add(i)
+                    
+                    # If we have an end pattern, also remove subsequent text elements until we find the end
+                    if sec.end_pattern:
+                        for j in range(i + 1, len(texts)):
+                            next_text_content = texts[j].get("text", "")
+                            if sec.end_pattern.search(next_text_content):
+                                logger.info(f"Found end of section '{sec.name}' in text element {j}")
+                                break
+                            text_indices_to_remove.add(j)
+                    else:
+                        # No end pattern - remove this text element only
+                        # For medication lists, we might want to remove the next few elements too
+                        # This is a heuristic that could be made more sophisticated
+                        if "medication" in sec.name.lower():
+                            # Remove up to 3 subsequent text elements for medication lists
+                            for j in range(i + 1, min(i + 4, len(texts))):
+                                text_indices_to_remove.add(j)
+        
+        # Remove the identified text elements
+        if text_indices_to_remove:
+            logger.info(f"Removing {len(text_indices_to_remove)} text elements: {sorted(text_indices_to_remove)}")
+            
+            # Remove from texts array
+            redacted_texts = [texts[i] for i in range(len(texts)) if i not in text_indices_to_remove]
+            redacted_json["texts"] = redacted_texts
+            
+            # Update body children to remove references to deleted texts
+            body = redacted_json.get("body", {})
+            if "children" in body:
+                # Filter out references to removed text elements
+                original_children = body["children"]
+                redacted_children = []
+                
+                for child_ref in original_children:
+                    if "$ref" in child_ref:
+                        ref_path = child_ref["$ref"]
+                        # Check if this reference points to a text element we're keeping
+                        if ref_path.startswith("#/texts/"):
+                            try:
+                                text_index = int(ref_path.split("/")[-1])
+                                if text_index not in text_indices_to_remove:
+                                    # Adjust the reference index since we removed some texts
+                                    new_index = text_index - sum(1 for x in text_indices_to_remove if x < text_index)
+                                    child_ref["$ref"] = f"#/texts/{new_index}"
+                                    redacted_children.append(child_ref)
+                            except (ValueError, IndexError):
+                                # Keep the reference if we can't parse it
+                                redacted_children.append(child_ref)
+                        else:
+                            # Keep non-text references
+                            redacted_children.append(child_ref)
+                    else:
+                        # Keep non-reference children
+                        redacted_children.append(child_ref)
+                
+                body["children"] = redacted_children
+        else:
+            logger.info("No sections found to remove")
+        
+        return redacted_json

src/streamlit_app.py

@@ -1,40 +1,549 @@
-import altair as alt
-import numpy as np
-import pandas as pd
 import streamlit as st
+import logging
+import os
+import tempfile
+import shutil
+from processing.document_processor import DocumentProcessor
+from processing.sections import ReasoningSectionExtractor
+from utils.logging_utils import get_log_handler
+from dotenv import load_dotenv
+import sys
+import html
+import difflib
+import re
 
-"""
-# Welcome to Streamlit!
-
-Edit `/streamlit_app.py` to customize this app to your heart's desire :heart:.
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
-
-In the meantime, below is an example of what you can do with just a few lines of code:
-"""
-
-num_points = st.slider("Number of points in spiral", 1, 10000, 1100)
-num_turns = st.slider("Number of turns in spiral", 1, 300, 31)
-
-indices = np.linspace(0, 1, num_points)
-theta = 2 * np.pi * num_turns * indices
-radius = indices
-
-x = radius * np.cos(theta)
-y = radius * np.sin(theta)
-
-df = pd.DataFrame({
-    "x": x,
-    "y": y,
-    "idx": indices,
-    "rand": np.random.randn(num_points),
-})
-
-st.altair_chart(alt.Chart(df, height=700, width=700)
-    .mark_point(filled=True)
-    .encode(
-        x=alt.X("x", axis=None),
-        y=alt.Y("y", axis=None),
-        color=alt.Color("idx", legend=None, scale=alt.Scale()),
-        size=alt.Size("rand", legend=None, scale=alt.Scale(range=[1, 150])),
-    ))
+# Load environment variables from .env
+load_dotenv()
+
+AZURE_OPENAI_ENDPOINT = os.getenv("AZURE_OPENAI_ENDPOINT")
+AZURE_OPENAI_KEY = os.getenv("AZURE_OPENAI_KEY")
+AZURE_OPENAI_VERSION = os.getenv("AZURE_OPENAI_VERSION")
+AZURE_OPENAI_DEPLOYMENT = os.getenv("AZURE_OPENAI_DEPLOYMENT")
+
+# Create temp directory if it doesn't exist
+TEMP_DIR = "temp_files"
+os.makedirs(TEMP_DIR, exist_ok=True)
+
+def cleanup_temp_files():
+    """Clean up temporary files in the temp directory."""
+    try:
+        if os.path.exists(TEMP_DIR):
+            for filename in os.listdir(TEMP_DIR):
+                file_path = os.path.join(TEMP_DIR, filename)
+                if os.path.isfile(file_path):
+                    os.remove(file_path)
+            logging.info(f"Cleaned up temporary files in {TEMP_DIR}")
+    except Exception as e:
+        logging.warning(f"Error cleaning up temp files: {e}")
+
+def save_uploaded_file(uploaded_file, filename):
+    """Save uploaded file to temp directory and return the path."""
+    temp_path = os.path.join(TEMP_DIR, f"temp_{filename}")
+    try:
+        uploaded_file.seek(0)  # Reset file pointer to beginning
+        file_bytes = uploaded_file.read()
+        with open(temp_path, "wb") as f:
+            f.write(file_bytes)
+        logging.info(f"Saved uploaded file to {temp_path}")
+        return temp_path
+    except Exception as e:
+        logging.error(f"Error saving uploaded file: {e}")
+        raise
+
+logging.basicConfig(
+    level=logging.INFO,  # or DEBUG for more verbosity
+    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
+    stream=sys.stdout,
+    force=True
+)
+
+# Configure page layout to use wide mode
+st.set_page_config(
+    page_title="Medical Document Parser & Redactor",
+    page_icon="📄",
+    layout="wide",
+    initial_sidebar_state="collapsed"
+)
+
+# Add custom CSS for better styling
+st.markdown("""
+<style>
+/* Custom styling for text areas */
+.stTextArea textarea {
+    font-family: 'Courier New', monospace !important;
+    font-size: 12px !important;
+    line-height: 1.4 !important;
+    border: 2px solid #e0e0e0 !important;
+    border-radius: 8px !important;
+}
+
+/* Hover effect for text areas */
+.stTextArea textarea:hover {
+    border-color: #1f77b4 !important;
+}
+
+/* Custom styling for download buttons */
+.stDownloadButton > button {
+    border-radius: 8px !important;
+    font-weight: 600 !important;
+}
+
+/* Custom styling for the comparison section */
+.comparison-container {
+    background-color: #f8f9fa;
+    padding: 20px;
+    border-radius: 10px;
+    border: 1px solid #e9ecef;
+}
+</style>
+""", unsafe_allow_html=True)
+
+# Configure root logger only once (avoid duplicate handlers on reruns)
+if len(logging.getLogger().handlers) == 0:
+    logging.getLogger().setLevel(logging.INFO)
+    # (We will attach custom handlers during processing as needed)
+
+# Title and description
+st.title("Medical Document Parser & Redactor")
+st.write("""
+Upload PDF medical documents to parse their content using **Docling** (structure-aware parser) 
+and automatically **redact specific sections** (e.g., initial and final medication lists). 
+Use the buttons below to view the original structure or process with redaction.
+""")
+
+# File uploader (accept multiple PDF files)
+uploaded_files = st.file_uploader("Upload PDF medical documents", type=["pdf"], accept_multiple_files=True)
+
+# Show temp directory status
+temp_file_count = len(os.listdir(TEMP_DIR)) if os.path.exists(TEMP_DIR) else 0
+if temp_file_count > 0:
+    st.caption(f"📁 {temp_file_count} temporary file(s) in temp_files/ directory")
+else:
+    st.caption("📁 No temporary files")
+
+# Initialize session state storage for results and logs
+if "processed_results" not in st.session_state:
+    st.session_state.processed_results = {}  # {filename: {"structured_json": ..., "redacted_md": ..., "redacted_json": ...}}
+if "logs" not in st.session_state:
+    st.session_state.logs = {}  # {filename: log_text}
+if "original_structures" not in st.session_state:
+    st.session_state.original_structures = {}  # {filename: structured_json}
+
+# Clean up temp files on app start (but keep the directory)
+if "temp_cleaned" not in st.session_state:
+    cleanup_temp_files()
+    st.session_state.temp_cleaned = True
+
+def create_diff_content(original_text: str, redacted_text: str, view_type: str) -> str:
+    """Create HTML content for diff view with highlighting."""
+    import difflib
+    import re
+    
+    # Normalize the text to reduce formatting differences
+    def normalize_text(text):
+        # Remove extra whitespace and normalize line endings
+        lines = text.split('\n')
+        normalized_lines = []
+        for line in lines:
+            # Strip whitespace but preserve content
+            stripped = line.strip()
+            if stripped:
+                # Normalize header formatting differences
+                # Convert ## to # for level 1 headers
+                if re.match(r'^##\s+', stripped):
+                    stripped = re.sub(r'^##\s+', '# ', stripped)
+                # Normalize quote formatting
+                if stripped.startswith('&gt; '):
+                    stripped = stripped.replace('&gt; ', '> ')
+                elif stripped.startswith('+ > '):
+                    stripped = stripped.replace('+ > ', '> ')
+                
+                normalized_lines.append(stripped)
+        return normalized_lines
+    
+    original_lines = normalize_text(original_text)
+    redacted_lines = normalize_text(redacted_text)
+    
+    # Use difflib to get a more sophisticated diff
+    differ = difflib.Differ()
+    diff = list(differ.compare(original_lines, redacted_lines))
+    
+    html_lines = []
+    
+    if view_type == 'original':
+        # Show original with removed content highlighted
+        for line in diff:
+            if line.startswith('  '):  # Unchanged line
+                escaped_line = html.escape(line[2:])
+                html_lines.append(f'<div style="padding: 2px 4px; margin: 1px 0; color: #333;">{escaped_line}</div>')
+            elif line.startswith('- '):  # Removed line
+                escaped_line = html.escape(line[2:])
+                html_lines.append(f'<div style="background-color: #ffebee; color: #c62828; padding: 2px 4px; margin: 1px 0; border-left: 3px solid #f44336; font-weight: bold;">- {escaped_line}</div>')
+            elif line.startswith('+ '):  # Added line (show as empty space in original view)
+                html_lines.append(f'<div style="background-color: #e8f5e8; color: #2e7d32; padding: 2px 4px; margin: 1px 0; border-left: 3px solid #4caf50; font-style: italic; opacity: 0.7;">+ (added in redacted version)</div>')
+            elif line.startswith('? '):  # Ignore difflib hints
+                continue
+    
+    elif view_type == 'redacted':
+        # Show redacted content with added content highlighted
+        for line in diff:
+            if line.startswith('  '):  # Unchanged line
+                escaped_line = html.escape(line[2:])
+                html_lines.append(f'<div style="padding: 2px 4px; margin: 1px 0; color: #333;">{escaped_line}</div>')
+            elif line.startswith('- '):  # Removed line (show as empty space in redacted view)
+                html_lines.append(f'<div style="background-color: #ffebee; color: #c62828; padding: 2px 4px; margin: 1px 0; border-left: 3px solid #f44336; font-style: italic; opacity: 0.7;">- (removed from original)</div>')
+            elif line.startswith('+ '):  # Added line
+                escaped_line = html.escape(line[2:])
+                html_lines.append(f'<div style="background-color: #e8f5e8; color: #2e7d32; padding: 2px 4px; margin: 1px 0; border-left: 3px solid #4caf50; font-weight: bold;">+ {escaped_line}</div>')
+            elif line.startswith('? '):  # Ignore difflib hints
+                continue
+    
+    return '\n'.join(html_lines)
+
+if uploaded_files:
+    # UI to select which file to work with (if multiple files uploaded)
+    file_names = [f.name for f in uploaded_files]
+    selected_file = st.selectbox("Select a file to work with", options=file_names)
+    
+    if selected_file:
+        # Find the selected uploaded file
+        uploaded_file = next(f for f in uploaded_files if f.name == selected_file)
+        
+        # Create buttons for different actions
+        col1, col2, col3 = st.columns(3)
+        
+        with col1:
+            if st.button("📄 Show Original", type="primary"):
+                # Process the document to get original structure (without redaction)
+                if selected_file not in st.session_state.original_structures:
+                    # Save uploaded file to a temporary location
+                    temp_path = save_uploaded_file(uploaded_file, selected_file)
+                    
+                    # Create a DocumentProcessor without section extraction (for original structure)
+                    processor = DocumentProcessor(section_extractor=None)
+                    
+                    # Process the document to get original structure
+                    result = processor.process(temp_path)
+                    st.session_state.original_structures[selected_file] = result.structured_json
+                    # Also store the original markdown for comparison
+                    st.session_state.original_structures[f"{selected_file}_markdown"] = result.structured_markdown
+                
+                # Display the original structure
+                st.session_state.show_original = True
+                st.session_state.show_processed = False
+        
+        with col2:
+            if st.button("🔒 Process with Redaction"):
+                # Process the document with redaction
+                if selected_file not in st.session_state.processed_results:
+                    # Save uploaded file to a temporary location
+                    temp_path = save_uploaded_file(uploaded_file, selected_file)
+                    
+                    # Create a DocumentProcessor with a SectionExtractor for our target sections
+                    section_extractor = ReasoningSectionExtractor(
+                        endpoint=AZURE_OPENAI_ENDPOINT,
+                        api_key=AZURE_OPENAI_KEY,
+                        api_version=AZURE_OPENAI_VERSION,
+                        deployment=AZURE_OPENAI_DEPLOYMENT,
+                    )
+                    processor = DocumentProcessor(section_extractor=section_extractor)
+                    
+                    # Attach an in-memory log handler to capture logs for this file
+                    log_handler, log_buffer = get_log_handler()
+                    root_logger = logging.getLogger()
+                    root_logger.addHandler(log_handler)
+                    try:
+                        # Process the document (Docling parse + section redaction)
+                        result = processor.process(temp_path)
+                    finally:
+                        # Remove handler and stop capturing logs
+                        root_logger.removeHandler(log_handler)
+                    
+                    # Save results in session state
+                    st.session_state.processed_results[selected_file] = {
+                        "structured_json": result.structured_json,
+                        "redacted_md": result.redacted_markdown,
+                        "redacted_json": result.redacted_json
+                    }
+                    # Combine log records into a single text
+                    log_text = "\n".join(log_buffer)
+                    st.session_state.logs[selected_file] = log_text
+                
+                st.session_state.show_original = False
+                st.session_state.show_processed = True
+        
+        with col3:
+            if st.button("🔄 Switch View"):
+                # Toggle between views
+                if st.session_state.get("show_original", False):
+                    st.session_state.show_original = False
+                    st.session_state.show_processed = True
+                else:
+                    st.session_state.show_original = True
+                    st.session_state.show_processed = False
+        
+        # Show current view status
+        if st.session_state.get("show_original", False):
+            st.info("📄 Currently viewing: **Original Document Structure**")
+        elif st.session_state.get("show_processed", False):
+            st.success("🔒 Currently viewing: **Processed Document with Redaction**")
+        else:
+            st.info("ℹ️ Select an action above to view document content")
+        
+        # Display results based on button clicked
+        if st.session_state.get("show_original", False):
+            st.markdown("---")
+            st.subheader(f"Original Document Structure - {selected_file}")
+            
+            # Get the original structure
+            original_json = st.session_state.original_structures[selected_file]
+            original_markdown = st.session_state.original_structures.get(f"{selected_file}_markdown", "")
+            
+            # Display PDF viewer and original markdown side by side
+            col1, col2 = st.columns([1, 1])
+            
+            with col1:
+                st.subheader("📄 Original PDF")
+                # Reset file pointer to beginning
+                uploaded_file.seek(0)
+                # Display PDF using base64 encoding for inline display
+                import base64
+                pdf_bytes = uploaded_file.getvalue()
+                b64_pdf = base64.b64encode(pdf_bytes).decode()
+                pdf_display = f'<iframe src="data:application/pdf;base64,{b64_pdf}" width="100%" height="600" type="application/pdf"></iframe>'
+                st.markdown(pdf_display, unsafe_allow_html=True)
+            
+            with col2:
+                st.subheader("📋 Original Document (Markdown)")
+                st.caption("Docling-generated markdown from the PDF")
+                # Use a text area for better readability and scrolling
+                st.text_area(
+                    label="Original markdown content",
+                    value=original_markdown,
+                    height=600,
+                    key="original_markdown_display",
+                    label_visibility="collapsed"
+                )
+            
+            # Add a download button for the original markdown
+            st.markdown("---")
+            col1, col2 = st.columns(2)
+            with col1:
+                st.download_button(
+                    label="📥 Download Original Markdown",
+                    data=original_markdown,
+                    file_name=f"{selected_file}_original.md",
+                    mime="text/markdown"
+                )
+            with col2:
+                st.subheader("📊 JSON Structure")
+                st.json(original_json)
+        
+        elif st.session_state.get("show_processed", False):
+            st.markdown("---")
+            st.subheader(f"Processed Document - {selected_file}")
+            
+            # Retrieve stored results
+            data = st.session_state.processed_results[selected_file]
+            structured_json = data["structured_json"]
+            redacted_md = data["redacted_md"]
+            redacted_json = data["redacted_json"]
+            
+            # Get the original markdown from the structured JSON
+            # We need to reconstruct the original markdown from the structured JSON
+            # For now, we'll use the structured_markdown from the DocumentResult
+            # But we need to store this in the session state
+            
+            # Create a DocumentProcessor to get the original markdown
+            if "original_markdown" not in st.session_state.processed_results[selected_file]:
+                # Save uploaded file to a temporary location
+                temp_path = save_uploaded_file(uploaded_file, selected_file)
+                
+                # Create a DocumentProcessor without section extraction to get original markdown
+                processor = DocumentProcessor(section_extractor=None)
+                result = processor.process(temp_path)
+                
+                # Store the original markdown
+                st.session_state.processed_results[selected_file]["original_markdown"] = result.structured_markdown
+            
+            original_md = st.session_state.processed_results[selected_file]["original_markdown"]
+            
+            # Show processing summary
+            original_texts = structured_json.get("texts", [])
+            redacted_texts = redacted_json.get("texts", [])
+            removed_count = len(original_texts) - len(redacted_texts)
+            
+            if removed_count > 0:
+                st.success(f"✅ Successfully removed {removed_count} text elements containing medication information")
+            else:
+                st.info("ℹ️ No medication sections were identified for removal")
+            
+            # Create tabs for different views
+            tab1, tab2, tab3 = st.tabs(["📄 Side-by-Side Comparison", "🔍 JSON Structure", "📊 Processing Details"])
+            
+            with tab1:
+                st.subheader("Original vs Redacted Content")
+                st.caption("Compare the original document content with the redacted version")
+                
+                # Create a diff-like interface with synchronized scrolling and highlighting
+                diff_html = f"""
+                <div style="display: flex; gap: 20px; height: 600px; font-family: 'Courier New', monospace; font-size: 12px;">
+                    <div style="flex: 1; border: 1px solid #ddd; border-radius: 5px; overflow: hidden;">
+                        <div style="background-color: #f8f9fa; padding: 10px; border-bottom: 1px solid #ddd; font-weight: bold; color: #1f77b4;">
+                            📋 Original Document
+                        </div>
+                        <div id="original-content" style="height: 540px; overflow-y: auto; padding: 10px; background-color: #fff;">
+                            {create_diff_content(original_md, redacted_md, 'original')}
+                        </div>
+                    </div>
+                    <div style="flex: 1; border: 1px solid #ddd; border-radius: 5px; overflow: hidden;">
+                        <div style="background-color: #f8f9fa; padding: 10px; border-bottom: 1px solid #ddd; font-weight: bold; color: #ff7f0e;">
+                            🔒 Redacted Document
+                        </div>
+                        <div id="redacted-content" style="height: 540px; overflow-y: auto; padding: 10px; background-color: #fff;">
+                            {create_diff_content(original_md, redacted_md, 'redacted')}
+                        </div>
+                    </div>
+                </div>
+                
+                <script>
+                // Synchronized scrolling
+                function syncScroll(sourceId, targetId) {{
+                    const source = document.getElementById(sourceId);
+                    const target = document.getElementById(targetId);
+                    if (source && target) {{
+                        target.scrollTop = source.scrollTop;
+                    }}
+                }}
+                
+                // Add scroll event listeners
+                document.addEventListener('DOMContentLoaded', function() {{
+                    const original = document.getElementById('original-content');
+                    const redacted = document.getElementById('redacted-content');
+                    
+                    if (original && redacted) {{
+                        original.addEventListener('scroll', function() {{
+                            syncScroll('original-content', 'redacted-content');
+                        }});
+                        
+                        redacted.addEventListener('scroll', function() {{
+                            syncScroll('redacted-content', 'original-content');
+                        }});
+                    }}
+                }});
+                </script>
+                """
+                
+                st.markdown(diff_html, unsafe_allow_html=True)
+                
+                # Add legend for the diff highlighting
+                st.markdown("---")
+                col1, col2, col3 = st.columns(3)
+                with col1:
+                    st.markdown("**🎨 Diff Legend:**")
+                    st.markdown("🔴 **Red background** = Removed content")
+                    st.markdown("🟢 **Green background** = Added content")
+                    st.markdown("⚪ **White background** = Unchanged content")
+                
+                with col2:
+                    st.markdown("**🔄 Scrolling:**")
+                    st.markdown("Scroll either panel to sync both views")
+                    st.markdown("Content is aligned for easy comparison")
+                
+                with col3:
+                    st.markdown("**💡 Tips:**")
+                    st.markdown("Look for red-highlighted sections")
+                    st.markdown("These show what was redacted")
+                    st.markdown("Use scroll to navigate long documents")
+                
+                # Add download buttons and other options
+                st.markdown("---")
+                st.subheader("📥 Download Options")
+                
+                col1, col2, col3 = st.columns(3)
+                with col1:
+                    st.download_button(
+                        label="📥 Download Original Markdown",
+                        data=original_md,
+                        file_name=f"{selected_file}_original.md",
+                        mime="text/markdown",
+                        use_container_width=True
+                    )
+                with col2:
+                    st.download_button(
+                        label="📥 Download Redacted Markdown", 
+                        data=redacted_md,
+                        file_name=f"{selected_file}_redacted.md",
+                        mime="text/markdown",
+                        use_container_width=True
+                    )
+                with col3:
+                    # Create a detailed diff view
+                    if st.button("🔍 Show Detailed Differences", use_container_width=True):
+                        st.session_state.show_diff = True
+                
+                # Show detailed diff if requested
+                if st.session_state.get("show_diff", False):
+                    st.markdown("---")
+                    st.subheader("🔍 Detailed Content Differences")
+                    
+                    # Simple diff visualization
+                    original_lines = original_md.split('\n')
+                    redacted_lines = redacted_md.split('\n')
+                    
+                    # Find removed lines
+                    removed_lines = []
+                    for line in original_lines:
+                        if line.strip() and line not in redacted_lines:
+                            removed_lines.append(line)
+                    
+                    if removed_lines:
+                        st.warning(f"**Removed {len(removed_lines)} lines containing medication information:**")
+                        for i, line in enumerate(removed_lines[:10]):  # Show first 10 removed lines
+                            st.text(f"❌ {line[:100]}{'...' if len(line) > 100 else ''}")
+                        if len(removed_lines) > 10:
+                            st.text(f"... and {len(removed_lines) - 10} more lines")
+                    else:
+                        st.info("No significant differences detected in the text content")
+            
+            with tab2:
+                st.subheader("Document Structure Analysis")
+                
+                # Show JSON structure comparison
+                col1, col2 = st.columns(2)
+                
+                with col1:
+                    st.markdown("**📊 Original Structure (JSON)**")
+                    st.json(structured_json)
+                
+                with col2:
+                    st.markdown("**🔒 Redacted Structure (JSON)**")
+                    st.json(redacted_json)
+            
+            with tab3:
+                st.subheader("Processing Details")
+                
+                # Show what was removed
+                if removed_count > 0:
+                    st.info(f"**Removed {removed_count} text elements from the document structure.**")
+                    
+                    # Show the removed text elements
+                    st.subheader("Removed Text Elements:")
+                    removed_texts = []
+                    for i, text_elem in enumerate(original_texts):
+                        if i >= len(redacted_texts) or text_elem.get("text", "") != redacted_texts[i].get("text", ""):
+                            removed_texts.append((i, text_elem.get("text", "")[:100] + "..." if len(text_elem.get("text", "")) > 100 else text_elem.get("text", "")))
+                    
+                    for idx, text in removed_texts:
+                        st.text(f"Text {idx}: {text}")
+                else:
+                    st.info("No text elements were removed during processing.")
+                
+                # Show processing logs
+                st.subheader("Processing Logs")
+                st.text_area(
+                    label="Processing logs",
+                    value=st.session_state.logs.get(selected_file, ""),
+                    height=300,
+                    label_visibility="collapsed"
+                )

src/utils/logging_utils.py

@@ -0,0 +1,25 @@
+import logging
+
+class ListHandler(logging.Handler):
+    """Logging handler that appends log messages to an internal list (for in-memory capture)."""
+    def __init__(self, log_list):
+        super().__init__()
+        self.log_list = log_list
+        # Use a simple formatter with time, level, and message
+        formatter = logging.Formatter(fmt="%(asctime)s - %(levelname)s - %(message)s", 
+                                      datefmt="%H:%M:%S")
+        self.setFormatter(formatter)
+    
+    def emit(self, record):
+        log_entry = self.format(record)
+        self.log_list.append(log_entry)
+
+def get_log_handler():
+    """
+    Create a ListHandler and return (handler, log_list).
+    This allows capturing logs in a list for a specific operation.
+    """
+    log_list = []
+    handler = ListHandler(log_list)
+    handler.setLevel(logging.DEBUG)
+    return handler, log_list