Upload 11 files

.gitignore

@@ -0,0 +1,89 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Installer logs
+debug.log
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# VS Code
+.vscode/
+
+# mypy
+.mypy_cache/
+.dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Output and models
+
+output/
+models/
+
+# Environment files
+.env
+.env.*
+*.env
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# Log files
+*.log
+
+# Docker
+*.pid
+*.sock
+
+# Ignore thumbnails
+thumbnails/
+
+# Ignore docker build cache
+__docker_build_cache__/
+
+# Ignore Miniconda
+/opt/conda/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 TIGER Lab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,10 +1,348 @@
----
-title: T2m
-emoji: 🏃
-colorFrom: green
-colorTo: pink
-sdk: docker
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# TheoremExplainAgent (TEA) 🍵
+[![arXiv](https://img.shields.io/badge/arXiv-2502.19400-b31b1b.svg)](https://arxiv.org/abs/2502.19400)
+<a href='https://huggingface.co/papers/2502.19400'><img src='https://img.shields.io/static/v1?label=Paper&message=Huggingface&color=orange'></a> 
+
+[**🌐 Homepage**](https://tiger-ai-lab.github.io/TheoremExplainAgent/)  | [**📖 arXiv**](https://arxiv.org/abs/2502.19400) | [**🤗 HuggingFace Dataset**](https://huggingface.co/datasets/TIGER-Lab/TheoremExplainBench) 
+
+[![contributors](https://img.shields.io/github/contributors/TIGER-AI-Lab/TheoremExplainAgent)](https://github.com/TIGER-AI-Lab/TheoremExplainAgent/graphs/contributors)
+[![license](https://img.shields.io/github/license/TIGER-AI-Lab/TheoremExplainAgent.svg)](https://github.com/TIGER-AI-Lab/TheoremExplainAgent/blob/main/LICENSE)
+[![GitHub](https://img.shields.io/github/stars/TIGER-AI-Lab/TheoremExplainAgent?style=social)](https://github.com/TIGER-AI-Lab/TheoremExplainAgent)
+[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FTIGER-AI-Lab%2FTheoremExplainAgent&count_bg=%23C83DB9&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=visitors&edge_flat=false)](https://hits.seeyoufarm.com)
+
+This repo contains the codebase for our paper [TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem Understanding](https://arxiv.org/abs/2502.19400)
+
+## Introduction
+TheoremExplainAgent is an AI system that generates long-form Manim videos to visually explain theorems, proving its deep understanding while uncovering reasoning flaws that text alone often hides.
+
+
+
+https://github.com/user-attachments/assets/17f2f4f2-8f2c-4abc-b377-ac92ebda69f3
+
+
+## 📰 News
+* 2025 Mar 3: Generation code and Evaluation code released. Thanks for the wait!
+<!--* 2025 Mar 3: Reach 404 stars without code.-->
+* 2025 Feb 27: Paper available on [Arxiv](https://arxiv.org/abs/2502.19400). Thanks AK for putting our paper on [HF Daily](https://huggingface.co/papers/2502.19400).
+
+## Installation
+
+> **Look at the [FAQ section in this README doc](https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#-faq) if you encountered any errors. If that didnt help, create a issue**<br>
+
+1. Setting up conda environment
+```shell
+conda create --name tea python=3.12.8
+conda activate tea
+pip install -r requirements.txt
+```
+
+2. You may also need to install latex and other dependencies for Manim Community. Look at [Manim Installation Docs](https://docs.manim.community/en/stable/installation.html) for more details.
+```shell
+# You might need these dependencies if you are using Linux Ubuntu:
+sudo apt-get install portaudio19-dev
+sudo apt-get install libsdl-pango-dev
+```
+
+3. Then Download the Kokoro model and voices using the commands to enable TTS service.
+
+```shell
+mkdir -p models && wget -P models https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/kokoro-v0_19.onnx && wget -P models https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/voices.bin
+```
+
+4. Create `.env` based on `.env.template`, filling in the environmental variables according to the models you choose to use.
+See [LiteLLM](https://docs.litellm.ai/docs/providers) for reference.
+
+```shell
+touch .env
+```
+Then open the `.env` file and edit it with whatever text editor you like.
+
+Your `.env` file should look like the following:
+```shell
+# OpenAI
+OPENAI_API_KEY=""
+
+# Azure OpenAI
+AZURE_API_KEY=""
+AZURE_API_BASE=""
+AZURE_API_VERSION=""
+
+# Google Vertex AI
+VERTEXAI_PROJECT=""
+VERTEXAI_LOCATION=""
+GOOGLE_APPLICATION_CREDENTIALS=""
+
+# Google Gemini
+GEMINI_API_KEY=""
+
+...
+
+# Kokoro TTS Settings
+KOKORO_MODEL_PATH="models/kokoro-v0_19.onnx"
+KOKORO_VOICES_PATH="models/voices.bin"
+KOKORO_DEFAULT_VOICE="af"
+KOKORO_DEFAULT_SPEED="1.0"
+KOKORO_DEFAULT_LANG="en-us"
+```
+Fill in the API keys according to the model you wanted to use.
+
+5. Configure Python path. Note that you need to configure the python path to make it work. Otherwise you may encounter import issues (like not being able to import src etc.)
+```shell
+export PYTHONPATH=$(pwd):$PYTHONPATH
+```
+
+6. (Optional) To setup RAG, See [https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#generation-with-rag](https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#generation-with-rag).
+
+> **Look at the [FAQ section in this README doc](https://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#-faq) if you encountered any errors. If that didnt help, create a issue**<br>
+
+## Generation
+
+### Supported Models
+<!--You can customize the allowed models by editing the `src/utils/allowed_models.json` file. This file specifies which `model` and `helper_model` the system is permitted to use.--> 
+The model naming follows the LiteLLM convention. For details on how models should be named, please refer to the [LiteLLM documentation](https://docs.litellm.ai/docs/providers).
+
+### Generation (Single topic)
+```shell
+python generate_video.py \
+      --model "openai/o3-mini" \
+      --helper_model "openai/o3-mini" \
+      --output_dir "output/your_exp_name" \
+      --topic "your_topic" \
+      --context "description of your topic, e.g. 'This is a topic about the properties of a triangle'" \
+```
+
+Example:
+```shell
+python generate_video.py \
+      --model "openai/o3-mini" \
+      --helper_model "openai/o3-mini" \
+      --output_dir "output/my_exp_name" \
+      --topic "Big O notation" \
+      --context "most common type of asymptotic notation in computer science used to measure worst case complexity" \
+```
+
+### Generation (in batch)
+```shell
+python generate_video.py \
+      --model "openai/o3-mini" \
+      --helper_model "openai/o3-mini" \
+      --output_dir "output/my_exp_name" \
+      --theorems_path data/thb_easy/math.json \
+      --max_scene_concurrency 7 \
+      --max_topic_concurrency 20 \
+```
+
+### Generation with RAG
+Before using RAG, download the RAG documentation from this [Google Drive link](https://drive.google.com/file/d/1Tn6J_JKVefFZRgZbjns93KLBtI9ullRv/view?usp=sharing). After downloading, unzip the file. For example, if you unzip it to `data/rag/manim_docs`, then you should set `--manim_docs_path` to `data/rag/manim_docs`. The vector database will be created the first time you run with RAG.
+
+```shell
+python generate_video.py \
+            --model "openai/o3-mini" \
+            --helper_model "openai/o3-mini" \
+            --output_dir "output/with_rag/o3-mini/vtutorbench_easy/math" \
+            --topic "Big O notation" \
+            --context "most common type of asymptotic notation in computer science used to measure worst case complexity" \
+            --use_rag \
+            --chroma_db_path "data/rag/chroma_db" \
+            --manim_docs_path "data/rag/manim_docs" \
+            --embedding_model "vertex_ai/text-embedding-005"
+```
+
+We support more options for generation, see below for more details:
+```shell
+usage: generate_video.py [-h]
+                         [--model]
+                         [--topic TOPIC] [--context CONTEXT]
+                         [--helper_model]
+                         [--only_gen_vid] [--only_combine] [--peek_existing_videos] [--output_dir OUTPUT_DIR] [--theorems_path THEOREMS_PATH]
+                         [--sample_size SAMPLE_SIZE] [--verbose] [--max_retries MAX_RETRIES] [--use_rag] [--use_visual_fix_code]
+                         [--chroma_db_path CHROMA_DB_PATH] [--manim_docs_path MANIM_DOCS_PATH]
+                         [--embedding_model {azure/text-embedding-3-large,vertex_ai/text-embedding-005}] [--use_context_learning]
+                         [--context_learning_path CONTEXT_LEARNING_PATH] [--use_langfuse] [--max_scene_concurrency MAX_SCENE_CONCURRENCY]
+                         [--max_topic_concurrency MAX_TOPIC_CONCURRENCY] [--debug_combine_topic DEBUG_COMBINE_TOPIC] [--only_plan] [--check_status]
+                         [--only_render] [--scenes SCENES [SCENES ...]]
+
+Generate Manim videos using AI
+
+options:
+  -h, --help            show this help message and exit
+  --model               Select the AI model to use
+  --topic TOPIC         Topic to generate videos for
+  --context CONTEXT     Context of the topic
+  --helper_model        Select the helper model to use
+  --only_gen_vid        Only generate videos to existing plans
+  --only_combine        Only combine videos
+  --peek_existing_videos, --peek
+                        Peek at existing videos
+  --output_dir OUTPUT_DIR
+                        Output directory
+  --theorems_path THEOREMS_PATH
+                        Path to theorems json file
+  --sample_size SAMPLE_SIZE, --sample SAMPLE_SIZE
+                        Number of theorems to sample
+  --verbose             Print verbose output
+  --max_retries MAX_RETRIES
+                        Maximum number of retries for code generation
+  --use_rag, --rag      Use Retrieval Augmented Generation
+  --use_visual_fix_code, --visual_fix_code
+                        Use VLM to fix code with rendered visuals
+  --chroma_db_path CHROMA_DB_PATH
+                        Path to Chroma DB
+  --manim_docs_path MANIM_DOCS_PATH
+                        Path to manim docs
+  --embedding_model {azure/text-embedding-3-large,vertex_ai/text-embedding-005}
+                        Select the embedding model to use
+  --use_context_learning
+                        Use context learning with example Manim code
+  --context_learning_path CONTEXT_LEARNING_PATH
+                        Path to context learning examples
+  --use_langfuse        Enable Langfuse logging
+  --max_scene_concurrency MAX_SCENE_CONCURRENCY
+                        Maximum number of scenes to process concurrently
+  --max_topic_concurrency MAX_TOPIC_CONCURRENCY
+                        Maximum number of topics to process concurrently
+  --debug_combine_topic DEBUG_COMBINE_TOPIC
+                        Debug combine videos
+  --only_plan           Only generate scene outline and implementation plans
+  --check_status        Check planning and code status for all theorems
+  --only_render         Only render scenes without combining videos
+  --scenes SCENES [SCENES ...]
+                        Specific scenes to process (if theorems_path is provided)
+```
+
+## Evaluation
+Note that Gemini and GPT4o is required for evaluation.
+
+Currently, evaluation requires a video file and a subtitle file (SRT format).
+
+Video evaluation:
+```shell
+usage: evaluate.py [-h]
+                   [--model_text {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}]
+                   [--model_video {gemini/gemini-1.5-pro-002,gemini/gemini-2.0-flash-exp,gemini/gemini-2.0-pro-exp-02-05}]
+                   [--model_image {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}]
+                   [--eval_type {text,video,image,all}] --file_path FILE_PATH --output_folder OUTPUT_FOLDER [--retry_limit RETRY_LIMIT] [--combine] [--bulk_evaluate] [--target_fps TARGET_FPS]
+                   [--use_parent_folder_as_topic] [--max_workers MAX_WORKERS]
+
+Automatic evaluation of theorem explanation videos with LLMs
+
+options:
+  -h, --help            show this help message and exit
+  --model_text {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}
+                        Select the AI model to use for text evaluation
+  --model_video {gemini/gemini-1.5-pro-002,gemini/gemini-2.0-flash-exp,gemini/gemini-2.0-pro-exp-02-05}
+                        Select the AI model to use for video evaluation
+  --model_image {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}
+                        Select the AI model to use for image evaluation
+  --eval_type {text,video,image,all}
+                        Type of evaluation to perform
+  --file_path FILE_PATH
+                        Path to a file or a theorem folder
+  --output_folder OUTPUT_FOLDER
+                        Directory to store the evaluation files
+  --retry_limit RETRY_LIMIT
+                        Number of retry attempts for each inference
+  --combine             Combine all results into a single JSON file
+  --bulk_evaluate       Evaluate a folder of theorems together
+  --target_fps TARGET_FPS
+                        Target FPS for video processing. If not set, original video FPS will be used
+  --use_parent_folder_as_topic
+                        Use parent folder name as topic name for single file evaluation
+  --max_workers MAX_WORKERS
+                        Maximum number of concurrent workers for parallel processing
+```
+* For `file_path`, it is recommended to pass a folder containing both an MP4 file and an SRT file.
+
+## Misc: Modify the system prompt in TheoremExplainAgent
+
+If you want to modify the system prompt, you need to:
+
+1. Modify files in `task_generator/prompts_raw` folder.
+2. Run `task_generator/parse_prompt.py` to rebuild the `__init__.py` file.
+
+```python
+cd task_generator
+python parse_prompt.py
+cd ..
+```
+
+## TheoremExplainBench (TEB)
+
+TheoremExplainBench can be found on https://huggingface.co/datasets/TIGER-Lab/TheoremExplainBench.
+
+How to use:
+```python
+import datasets
+dataset = datasets.load_dataset("TIGER-Lab/TheoremExplainBench")
+```
+
+Dataset info:
+```shell
+DatasetDict({
+    train: Dataset({
+        features: ['uid', 'subject', 'difficulty', 'theorem', 'description', 'subfield'],
+        num_rows: 240
+    })
+})
+```
+
+## ❓ FAQ
+
+The FAQ should cover the most common errors you could encounter. If you see something new, report it on issues.
+
+Q: Error `src.utils.kokoro_voiceover import KokoroService  # You MUST import like this as this is our custom voiceover service. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'src'`. <br>
+A: Please run `export PYTHONPATH=$(pwd):$PYTHONPATH` when you start a new terminal. <br>
+
+Q: Error `Files not found` <br>
+A: Check your Manim installation. <br>
+
+Q: Error `latex ...` <br>
+A: Check your latex installation. <br>
+
+Q: The output log is not showing response? <br>
+A: It could be API-related issues. Make sure your `.env` file is properly configured (fill in your API keys), or you can enable litellm debug mode to figure out the issues. <be>
+
+Q: Plans / Scenes are missing? <br>
+A: It could be API-related issues. Make sure your `.env` file is properly configured (fill in your API keys), or you can enable litellm debug mode to figure out the issues. <br>
+
+
+## 🖊️ Citation
+
+Please kindly cite our paper if you use our code, data, models or results:
+```bibtex
+@misc{ku2025theoremexplainagentmultimodalexplanationsllm,
+      title={TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem Understanding}, 
+      author={Max Ku and Thomas Chong and Jonathan Leung and Krish Shah and Alvin Yu and Wenhu Chen},
+      year={2025},
+      eprint={2502.19400},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2502.19400}, 
+}
+```
+
+## 🎫 License
+
+This project is released under the [the MIT License](LICENSE).
+
+## ⭐ Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=TIGER-AI-Lab/TheoremExplainAgent&type=Date)](https://star-history.com/#TIGER-AI-Lab/TheoremExplainAgent&Date)
+
+## 💞 Acknowledgements
+
+We want to thank [Votee AI](https://votee.ai/) for sponsoring API keys to access the close-sourced models.
+
+The code is built upon the below repositories, we thank all the contributors for open-sourcing.
+* [Manim Community](https://www.manim.community/)
+* [kokoro-manim-voiceover](https://github.com/xposed73/kokoro-manim-voiceover)
+* [manim-physics](https://github.com/Matheart/manim-physics)
+* [manim-Chemistry](https://github.com/UnMolDeQuimica/manim-Chemistry)
+* [ManimML](https://github.com/helblazer811/ManimML)
+* [manim-dsa](https://github.com/F4bbi/manim-dsa)
+* [manim-circuit](https://github.com/Mr-FuzzyPenguin/manim-circuit)
+
+## 🚨 Disclaimer
+
+**This work is intended for research purposes only. The authors do not encourage or endorse the use of this codebase for commercial applications. The code is provided "as is" without any warranties, and users assume all responsibility for its use.**
+
+Tested Environment: MacOS, Linux

create_embeddings.py

@@ -0,0 +1,491 @@
+#!/usr/bin/env python3
+"""
+Script to create embeddings and populate the vector store with Manim documentation.
+"""
+
+import os
+import sys
+import time
+import json
+import logging
+from pathlib import Path
+from typing import List, Dict, Any
+
+from src.rag.vector_store import EnhancedRAGVectorStore as RAGVectorStore
+from src.config.config import Config
+
+# Setup logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+def download_manim_docs():
+    """Download Manim documentation if not available locally."""
+    print("📥 Checking for Manim documentation...")
+    
+    manim_docs_path = Config.MANIM_DOCS_PATH
+    os.makedirs(manim_docs_path, exist_ok=True)
+    
+    # Check if we have any documentation files
+    doc_files = []
+    for ext in ['.py', '.md', '.rst', '.txt']:
+        doc_files.extend(list(Path(manim_docs_path).rglob(f'*{ext}')))
+    
+    if len(doc_files) == 0:
+        print("📝 No documentation files found. Creating sample documentation...")
+        create_sample_manim_docs(manim_docs_path)
+    else:
+        print(f"✅ Found {len(doc_files)} documentation files")
+
+def create_sample_manim_docs(docs_path: str):
+    """Create sample Manim documentation for testing."""
+    
+    # Create core documentation
+    core_docs = {
+        "circle.py": '''
+"""Circle mobject in Manim.
+
+The Circle class is one of the basic geometric shapes in Manim.
+It can be used to create circular objects that can be animated.
+
+Example:
+    circle = Circle(radius=1, color=BLUE)
+    self.add(circle)
+    self.play(Create(circle))
+
+Attributes:
+    radius (float): The radius of the circle
+    color (str): The color of the circle
+    stroke_width (float): The width of the circle's outline
+"""
+
+from manim import *
+
+class Circle(Arc):
+    \"\"\"A circle mobject.
+    
+    Parameters
+    ----------
+    radius : float
+        The radius of the circle.
+    color : str
+        The color of the circle.
+    \"\"\"
+    
+    def __init__(self, radius=1, **kwargs):
+        super().__init__(start_angle=0, angle=TAU, radius=radius, **kwargs)
+        
+    def animate_creation(self):
+        \"\"\"Animate the creation of the circle.\"\"\"
+        return Create(self)
+        
+    def move_to_position(self, position):
+        \"\"\"Move circle to a specific position.\"\"\"
+        return self.move_to(position)
+''',
+        
+        "text.py": '''
+"""Text mobjects in Manim.
+
+Text and Tex classes for displaying text and mathematical expressions.
+
+Example:
+    text = Text("Hello World", font_size=48)
+    equation = MathTex(r"E = mc^2")
+    
+    self.play(Write(text))
+    self.play(Transform(text, equation))
+"""
+
+from manim import *
+
+class Text(SVGMobject):
+    \"\"\"Text mobject for displaying regular text.
+    
+    Parameters
+    ----------
+    text : str
+        The text to display
+    font_size : int
+        Size of the font
+    color : str
+        Color of the text
+    \"\"\"
+    
+    def __init__(self, text, font_size=48, **kwargs):
+        self.text = text
+        self.font_size = font_size
+        super().__init__(**kwargs)
+        
+class MathTex(SVGMobject):
+    \"\"\"Mathematical text using LaTeX.
+    
+    Parameters
+    ----------
+    tex_string : str
+        LaTeX string for mathematical expression
+    \"\"\"
+    
+    def __init__(self, tex_string, **kwargs):
+        self.tex_string = tex_string
+        super().__init__(**kwargs)
+''',
+
+        "animation.py": '''
+"""Animation classes in Manim.
+
+Core animation classes for creating smooth transitions and movements.
+
+Example:
+    circle = Circle()
+    
+    # Basic animations
+    self.play(Create(circle))
+    self.play(FadeIn(circle))
+    self.play(Transform(circle, square))
+"""
+
+from manim import *
+
+class Create(Animation):
+    \"\"\"Animation that creates a mobject by drawing it.
+    
+    Parameters
+    ----------
+    mobject : Mobject
+        The mobject to create
+    run_time : float
+        Duration of the animation
+    \"\"\"
+    
+    def __init__(self, mobject, run_time=1, **kwargs):
+        super().__init__(mobject, run_time=run_time, **kwargs)
+
+class Transform(Animation):
+    \"\"\"Transform one mobject into another.
+    
+    Parameters
+    ----------
+    mobject : Mobject
+        Source mobject
+    target_mobject : Mobject
+        Target mobject to transform into
+    \"\"\"
+    
+    def __init__(self, mobject, target_mobject, **kwargs):
+        self.target_mobject = target_mobject
+        super().__init__(mobject, **kwargs)
+
+class FadeIn(Animation):
+    \"\"\"Fade in animation.\"\"\"
+    pass
+
+class FadeOut(Animation):
+    \"\"\"Fade out animation.\"\"\"
+    pass
+''',
+
+        "scene.py": '''
+"""Scene class - the foundation of Manim animations.
+
+Every Manim animation is built using Scene classes.
+
+Example:
+    class MyScene(Scene):
+        def construct(self):
+            circle = Circle()
+            self.play(Create(circle))
+            self.wait(1)
+"""
+
+from manim import *
+
+class Scene:
+    \"\"\"Base class for all Manim scenes.
+    
+    The Scene class provides the foundation for creating animations.
+    Override the construct() method to define your animation.
+    \"\"\"
+    
+    def __init__(self):
+        self.mobjects = []
+        
+    def construct(self):
+        \"\"\"Override this method to create your animation.\"\"\"
+        pass
+        
+    def add(self, *mobjects):
+        \"\"\"Add mobjects to the scene.\"\"\"
+        for mobject in mobjects:
+            self.mobjects.append(mobject)
+            
+    def play(self, *animations, run_time=1):
+        \"\"\"Play animations.
+        
+        Parameters
+        ----------
+        animations : Animation
+            Animations to play
+        run_time : float
+            Duration to run animations
+        \"\"\"
+        for animation in animations:
+            animation.run_time = run_time
+            
+    def wait(self, duration=1):
+        \"\"\"Wait for specified duration.\"\"\"
+        time.sleep(duration)
+''',
+
+        "plotting.py": '''
+"""Plotting utilities in Manim.
+
+Classes for creating mathematical plots and graphs.
+
+Example:
+    axes = Axes(x_range=[-3, 3], y_range=[-2, 2])
+    graph = axes.plot(lambda x: x**2, color=BLUE)
+    
+    self.play(Create(axes))
+    self.play(Create(graph))
+"""
+
+from manim import *
+
+class Axes(VGroup):
+    \"\"\"Coordinate axes for plotting.
+    
+    Parameters
+    ----------
+    x_range : list
+        Range for x-axis [min, max, step]
+    y_range : list  
+        Range for y-axis [min, max, step]
+    \"\"\"
+    
+    def __init__(self, x_range=[-1, 1, 1], y_range=[-1, 1, 1], **kwargs):
+        self.x_range = x_range
+        self.y_range = y_range
+        super().__init__(**kwargs)
+        
+    def plot(self, function, color=WHITE, **kwargs):
+        \"\"\"Plot a mathematical function.
+        
+        Parameters
+        ----------
+        function : callable
+            Function to plot
+        color : str
+            Color of the plot
+        \"\"\"
+        return ParametricFunction(function, color=color, **kwargs)
+        
+    def get_graph(self, function, **kwargs):
+        \"\"\"Get graph of function.\"\"\"
+        return self.plot(function, **kwargs)
+
+class NumberPlane(Axes):
+    \"\"\"A coordinate plane with grid lines.\"\"\"
+    
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+'''
+    }
+    
+    # Create the core documentation files
+    for filename, content in core_docs.items():
+        file_path = os.path.join(docs_path, filename)
+        with open(file_path, 'w', encoding='utf-8') as f:
+            f.write(content)
+    
+    # Create a README file
+    readme_content = '''# Manim Documentation
+
+This directory contains documentation for Manim (Mathematical Animation Engine).
+
+## Core Classes
+
+- **Circle**: Create circular shapes and animations
+- **Text/MathTex**: Display text and mathematical expressions  
+- **Scene**: Base class for all animations
+- **Animation**: Classes for creating smooth transitions
+- **Axes/NumberPlane**: Plotting and coordinate systems
+
+## Basic Usage
+
+```python
+from manim import *
+
+class MyScene(Scene):
+    def construct(self):
+        # Create objects
+        circle = Circle(radius=1, color=BLUE)
+        text = Text("Hello Manim!")
+        
+        # Animate
+        self.play(Create(circle))
+        self.play(Write(text))
+        self.wait(1)
+```
+
+For more information, visit: https://docs.manim.community/
+'''
+    
+    readme_path = os.path.join(docs_path, "README.md")
+    with open(readme_path, 'w', encoding='utf-8') as f:
+        f.write(readme_content)
+    
+    print(f"✅ Created {len(core_docs) + 1} sample documentation files")
+
+def create_vector_store_with_progress():
+    """Create and populate the vector store with progress tracking."""
+    
+    print("🚀 Initializing Enhanced RAG Vector Store...")
+    print(f"📁 ChromaDB Path: {Config.CHROMA_DB_PATH}")
+    print(f"📁 Manim Docs Path: {Config.MANIM_DOCS_PATH}")
+    print(f"🧠 Embedding Model: {Config.EMBEDDING_MODEL}")
+    print(f"📊 Expected Embedding Dimensions: 384")
+    
+    # Ensure ChromaDB directory exists
+    os.makedirs(Config.CHROMA_DB_PATH, exist_ok=True)
+    
+    start_time = time.time()
+    
+    try:
+        # Create vector store - this will automatically process documents and create embeddings
+        vector_store = RAGVectorStore(
+            chroma_db_path=Config.CHROMA_DB_PATH,
+            manim_docs_path=Config.MANIM_DOCS_PATH,
+            embedding_model=Config.EMBEDDING_MODEL,
+            session_id="embedding_creation",
+            use_langfuse=False
+        )
+        
+        print("✅ Vector store initialized successfully!")
+        
+        # Check document count
+        if hasattr(vector_store, 'core_vector_store') and vector_store.core_vector_store:
+            try:
+                collection = vector_store.core_vector_store._collection
+                doc_count = collection.count()
+                print(f"📊 Total documents in vector store: {doc_count}")
+                
+                if doc_count > 0:
+                    # Test the embedding dimensions
+                    embedding_function = vector_store._get_embedding_function()
+                    test_embedding = embedding_function.embed_query("test query")
+                    print(f"🧠 Embedding dimensions: {len(test_embedding)}")
+                    
+                    # Get sample documents
+                    sample_docs = collection.peek(limit=3)
+                    print(f"📋 Sample document IDs: {sample_docs.get('ids', [])[:3]}")
+                    
+                    # Test search functionality
+                    print("\n🔍 Testing search functionality...")
+                    results = vector_store.core_vector_store.similarity_search_with_relevance_scores(
+                        query="How to create a circle?",
+                        k=3,
+                        score_threshold=0.0
+                    )
+                    
+                    print(f"🎯 Search results: {len(results)} documents found")
+                    for i, (doc, score) in enumerate(results):
+                        print(f"   [{i+1}] Score: {score:.4f} | Source: {doc.metadata.get('source', 'unknown')}")
+                        print(f"       Content: {doc.page_content[:100]}...")
+                else:
+                    print("⚠️ Warning: No documents found in vector store")
+                    
+            except Exception as e:
+                print(f"❌ Error checking document count: {e}")
+        
+        elapsed_time = time.time() - start_time
+        print(f"\n⏱️ Vector store creation completed in {elapsed_time:.2f} seconds")
+        
+        return vector_store
+        
+    except Exception as e:
+        print(f"❌ Error creating vector store: {e}")
+        import traceback
+        traceback.print_exc()
+        return None
+
+def verify_embeddings():
+    """Verify that embeddings were created successfully."""
+    
+    print("\n🔬 Verifying embeddings...")
+    
+    try:
+        # Load the vector store
+        vector_store = RAGVectorStore(
+            chroma_db_path=Config.CHROMA_DB_PATH,
+            manim_docs_path=Config.MANIM_DOCS_PATH,
+            embedding_model=Config.EMBEDDING_MODEL,
+            session_id="verification",
+            use_langfuse=False
+        )
+        
+        # Test queries
+        test_queries = [
+            "How to create a circle in Manim?",
+            "Text animation examples",
+            "Mathematical plotting with axes",
+            "Scene construction basics"
+        ]
+        
+        print("🎯 Testing search with various queries:")
+        
+        for query in test_queries:
+            results = vector_store.core_vector_store.similarity_search_with_relevance_scores(
+                query=query,
+                k=2,
+                score_threshold=0.0
+            )
+            
+            print(f"\nQuery: '{query}'")
+            print(f"Results: {len(results)} documents")
+            
+            for i, (doc, score) in enumerate(results):
+                print(f"  [{i+1}] Score: {score:.4f}")
+                print(f"      Source: {doc.metadata.get('source', 'unknown')}")
+                print(f"      Content: {doc.page_content[:80]}...")
+        
+        print("\n✅ Embedding verification completed successfully!")
+        return True
+        
+    except Exception as e:
+        print(f"❌ Embedding verification failed: {e}")
+        return False
+
+def main():
+    """Main function to create embeddings."""
+    
+    print("🎬 MANIM RAG EMBEDDING CREATION")
+    print("="*50)
+    
+    # Step 1: Download/create documentation
+    download_manim_docs()
+    
+    # Step 2: Create vector store and embeddings
+    vector_store = create_vector_store_with_progress()
+    
+    if vector_store is None:
+        print("❌ Failed to create vector store. Exiting.")
+        return False
+    
+    # Step 3: Verify embeddings
+    success = verify_embeddings()
+    
+    if success:
+        print("\n🎉 SUCCESS! Embeddings created and verified.")
+        print(f"📁 ChromaDB location: {Config.CHROMA_DB_PATH}")
+        print("🔍 You can now run the RAG system tests.")
+        print("\nNext steps:")
+        print("  1. Run: python test_rag_system.py")
+        print("  2. Run: python debug_retrieval.py")
+    else:
+        print("\n❌ FAILED! Something went wrong during verification.")
+    
+    return success
+
+if __name__ == "__main__":
+    success = main()
+    sys.exit(0 if success else 1)

docker-compose.yml

@@ -0,0 +1,80 @@
+version: '3.8'
+
+services:
+  theoremexplain:
+    build:
+      context: .
+      dockerfile: dockerfile
+    container_name: theoremexplain-agent
+    ports:
+      - "7860:7860"
+    volumes:
+      # Mount output directory to persist generated videos
+      - ./output:/app/output
+      # Mount models directory if you want to use local models
+      - ./models:/app/models
+      # Mount data directory for RAG and datasets
+      - ./data:/app/data
+    environment:
+      # Copy environment variables from host .env file
+      - OPENAI_API_KEY=${OPENAI_API_KEY}
+      - AZURE_API_KEY=${AZURE_API_KEY}
+      - AZURE_API_BASE=${AZURE_API_BASE}
+      - AZURE_API_VERSION=${AZURE_API_VERSION}
+      - VERTEXAI_PROJECT=${VERTEXAI_PROJECT}
+      - VERTEXAI_LOCATION=${VERTEXAI_LOCATION}
+      - GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
+      - GEMINI_API_KEY=${GEMINI_API_KEY}
+      # Kokoro TTS settings
+      - KOKORO_MODEL_PATH=models/kokoro-v0_19.onnx
+      - KOKORO_VOICES_PATH=models/voices.bin
+      - KOKORO_DEFAULT_VOICE=af
+      - KOKORO_DEFAULT_SPEED=1.0
+      - KOKORO_DEFAULT_LANG=en-us
+      # Python path
+      - PYTHONPATH=/app:$PYTHONPATH
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "conda", "run", "-n", "tea", "python", "-c", "import src; import manim; print('Health check passed')"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+
+  # Optional: Add a service for running batch generation
+  theoremexplain-batch:
+    build:
+      context: .
+      dockerfile: dockerfile
+    container_name: theoremexplain-batch
+    profiles:
+      - batch
+    volumes:
+      - ./output:/app/output
+      - ./models:/app/models
+      - ./data:/app/data
+    environment:
+      # Same environment variables as main service
+      - OPENAI_API_KEY=${OPENAI_API_KEY}
+      - AZURE_API_KEY=${AZURE_API_KEY}
+      - AZURE_API_BASE=${AZURE_API_BASE}
+      - AZURE_API_VERSION=${AZURE_API_VERSION}
+      - VERTEXAI_PROJECT=${VERTEXAI_PROJECT}
+      - VERTEXAI_LOCATION=${VERTEXAI_LOCATION}
+      - GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}
+      - GEMINI_API_KEY=${GEMINI_API_KEY}
+      - KOKORO_MODEL_PATH=models/kokoro-v0_19.onnx
+      - KOKORO_VOICES_PATH=models/voices.bin
+      - KOKORO_DEFAULT_VOICE=af
+      - KOKORO_DEFAULT_SPEED=1.0
+      - KOKORO_DEFAULT_LANG=en-us
+      - PYTHONPATH=/app:$PYTHONPATH
+    command: >
+      conda run --no-capture-output -n tea python generate_video.py
+      --model "openai/gpt-4o-mini"
+      --helper_model "openai/gpt-4o-mini"
+      --output_dir "output/batch_generation"
+      --theorems_path "data/thb_easy/math.json"
+      --max_scene_concurrency 3
+      --max_topic_concurrency 5
+    restart: no

dockerfile

@@ -0,0 +1,126 @@
+# Stage 1: Builder
+FROM python:3.12-slim AS builder
+
+WORKDIR /app
+
+# Set environment variables
+ENV DEBIAN_FRONTEND=noninteractive \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_DISABLE_PIP_VERSION_CHECK=1
+
+# Install build dependencies in a single layer with version pinning where critical
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    wget=1.21.* \
+    curl \
+    gcc \
+    bzip2 \
+    ca-certificates \
+    gnupg \
+    git \
+    python3-dev \
+    build-essential \
+    pkg-config \
+    portaudio19-dev \
+    libsdl-pango-dev \
+    libcairo2-dev \
+    libpango1.0-dev \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+# Install TinyTeX with error handling and cleanup
+RUN wget -qO- "https://yihui.org/tinytex/install-bin-unix.sh" | sh \
+    && ~/.TinyTeX/bin/*/tlmgr path add \
+    && ~/.TinyTeX/bin/*/tlmgr install \
+        amsmath babel-english cbfonts-fd cm-super count1to ctex \
+        doublestroke dvisvgm everysel fontspec frcursive fundus-calligra \
+        gnu-freefont jknapltx latex-bin mathastext microtype multitoc \
+        physics preview prelim2e ragged2e relsize rsfs setspace \
+        standalone tipa wasy wasysym xcolor xetex xkeyval \
+    && rm -rf ~/.TinyTeX/texmf-var/web2c/tlmgr.log* \
+    && rm -rf ~/.TinyTeX/texmf-var/web2c/tlmgr-commands.log* \
+    && find ~/.TinyTeX -name "*.log" -delete \
+    && find ~/.TinyTeX -name "*.aux" -delete
+
+# Copy and install Python dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir --prefix=/install -r requirements.txt \
+    && find /install -name "*.pyc" -delete \
+    && find /install -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
+
+# Download models with checksums and error handling
+RUN mkdir -p /models \
+    && cd /models \
+    && wget --progress=dot:giga -O kokoro-v0_19.onnx \
+        "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/kokoro-v0_19.onnx" \
+    && wget --progress=dot:giga -O voices.bin \
+        "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/voices.bin" \
+    && ls -la /models
+
+# Stage 2: Runtime
+FROM python:3.12-slim AS runtime
+
+# Create non-root user for security
+RUN groupadd -r appuser && useradd -r -g appuser -d /app -s /sbin/nologin appuser
+
+WORKDIR /app
+
+# Set environment variables
+ENV DEBIAN_FRONTEND=noninteractive \
+    PYTHONUNBUFFERED=1 \
+    PYTHONPATH=/app \
+    PATH="/root/.TinyTeX/bin/x86_64-linux:$PATH"
+
+# Install runtime dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    portaudio19-dev \
+    libasound2-dev \
+    libsdl-pango-dev \
+    libcairo2-dev \
+    libpango1.0-dev \
+    sox \
+    ffmpeg \
+    tini \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+# Copy TinyTeX from builder stage
+COPY --from=builder /root/.TinyTeX /root/.TinyTeX
+
+# Copy Python packages from builder stage
+COPY --from=builder /install /usr/local
+
+# Copy models from builder stage
+COPY --from=builder /models /app/models
+
+# Copy application files (be more selective to reduce layer size)
+COPY --chown=appuser:appuser .env gradio_app.py ./
+COPY --chown=appuser:appuser src/ ./src/
+
+# Create output directory with proper permissions
+RUN mkdir -p output \
+    && chown -R appuser:appuser /app
+
+# Switch to non-root user
+USER appuser
+
+# Add labels for better maintainability
+LABEL maintainer="[email protected]" \
+      version="1.0" \
+      description="Multi-stage Docker image for ML application"
+
+# Expose port
+EXPOSE 7860
+
+# Use tini as PID 1 for proper signal handling
+ENTRYPOINT ["tini", "--"]
+
+# Improved health check with more specific validation
+HEALTHCHECK --interval=30s --timeout=15s --start-period=60s --retries=3 \
+    CMD python -c "import sys; import src; import manim; \
+                   import requests; \
+                   r = requests.get('http://localhost:7860/health', timeout=10); \
+                   sys.exit(0 if r.status_code == 200 else 1)" || exit 1
+
+# Start the application
+CMD ["python", "gradio_app.py"]

evaluate.py

@@ -0,0 +1,474 @@
+import os
+import json
+import argparse
+import tempfile
+from typing import Dict, List, Union
+from datetime import datetime
+
+from dotenv import load_dotenv
+from moviepy import VideoFileClip
+
+from mllm_tools.litellm import LiteLLMWrapper
+from mllm_tools.gemini import GeminiWrapper
+from eval_suite.utils import calculate_geometric_mean
+from eval_suite.text_utils import parse_srt_to_text, fix_transcript, evaluate_text
+from eval_suite.video_utils import evaluate_video_chunk_new
+from eval_suite.image_utils import evaluate_sampled_images
+
+load_dotenv()
+
+with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "src", "utils", "allowed_models.json")) as f:
+    ALLOWED_MODELS = json.load(f)["allowed_models"]
+
+
+def combine_results(output_folder: str, combined_file: str, results: Dict[str, Dict]) -> None:
+    """
+    Combine all evaluation results into a single file.
+
+    Args:
+        output_folder (str): Directory to store the combined file.
+        combined_file (str): Name of the combined file.
+        results (Dict[str, Dict]): Dictionary of evaluation results with file names as keys.
+
+    Returns:
+        None
+    """
+    combined_path = os.path.join(output_folder, combined_file)
+    with open(combined_path, 'w') as output_file:
+        json.dump(results, output_file, indent=4)
+
+
+def save_individual_result(output_folder: str, file_name: str, result: Dict) -> None:
+    """
+    Save individual evaluation result to a file.
+
+    Args:
+        output_folder (str): Directory to store the evaluation file.
+        file_name (str): Name of the file.
+        result (Dict): Evaluation result.
+
+    Returns:
+        None
+    """
+    current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
+    result_file = f"evaluation_{file_name}_{current_time}.json"
+    os.makedirs(output_folder, exist_ok=True)
+    result_path = os.path.join(output_folder, result_file)
+    with open(result_path, 'w') as output_file:
+        json.dump(result, output_file, indent=4)
+
+
+def evaluate_text_file(model, transcript_path, retry_limit):
+    """
+    Evaluate a text file using the provided model.
+
+    Args:
+        model: The model to use for evaluation.
+        transcript_path (str): Path to the transcript file (.srt or .txt).
+        retry_limit (int): Number of retry attempts for evaluation.
+
+    Returns:
+        Dict or None: Evaluation results if successful, None if file format unsupported.
+    """
+    if not transcript_path.endswith(('.srt', '.txt')):
+        print(f"Skipping {transcript_path}: Unsupported file format for text evaluation.")
+        return None
+
+    if transcript_path.endswith(".srt"):
+        transcript = parse_srt_to_text(transcript_path)
+    elif transcript_path.endswith(".txt"):
+        with open(transcript_path) as f:
+            transcript = f.read().strip()
+    else:
+        raise ValueError("Unrecognized transcript file format.")
+
+    capital_letter_proportion = sum(1 for c in transcript if c.isupper()) / sum(1 for c in transcript if c.isalpha())
+    if capital_letter_proportion < 0.01:
+        transcript = fix_transcript(model, transcript)
+
+    print(f"Performing text evaluation: {os.path.basename(transcript_path)}")
+    result = evaluate_text(model, transcript, retry_limit)
+    return result
+
+
+def evaluate_video_file(model, video_path, transcript_path, description_path, target_fps=None, output_folder=None):
+    """
+    Evaluate a video file using the provided model.
+
+    Args:
+        model: The model to use for evaluation.
+        video_path (str): Path to the video file.
+        transcript_path (str): Path to the transcript file.
+        description_path (str): Path to the description file.
+        target_fps (int, optional): Target frames per second for video processing.
+        output_folder (str, optional): Directory to store output files.
+
+    Returns:
+        Dict or None: Evaluation results if successful, None if file format unsupported.
+    """
+    if not video_path.endswith(('.mp4', '.mkv')):
+        print(f"Skipping {video_path}: Unsupported file format for video evaluation.")
+        return None
+
+    moviepy_temp_dir = os.path.join(output_folder, "moviepy_temp")
+
+    # Chunking
+    num_chunks = 10
+    with VideoFileClip(video_path) as clip:
+        duration = clip.duration
+        chunk_duration = duration / num_chunks
+        results = []
+        
+        # Create a temporary directory in the output_folder
+        temp_dir_parent = output_folder or os.getcwd()
+        with tempfile.TemporaryDirectory(dir=temp_dir_parent) as temp_dir:
+            for i in range(10):
+                start = i * chunk_duration
+                end = min(start + chunk_duration, duration)
+                chunk = clip.subclipped(start, end)
+                chunk_path = os.path.join(temp_dir, f"chunk_{i+1}.mp4")
+                # Explicitly set the temp_audiofile path with matching codec
+                temp_audiofile = os.path.join(moviepy_temp_dir, f"temp_audio_chunk_{i+1}.m4a")
+                chunk.write_videofile(
+                    chunk_path,
+                    codec="libx264",
+                    audio_codec="aac",
+                    temp_audiofile=temp_audiofile,
+                    audio_bitrate="192k",
+                    preset="ultrafast",  # Speed up encoding
+                    logger=None
+                )
+                # Create processed videos folder inside output_folder
+                processed_videos_dir = os.path.join(output_folder, "processed_videos")
+                save_path = os.path.join(processed_videos_dir, f"processed_chunk_{i+1}.mp4")
+                result = evaluate_video_chunk_new(
+                    model,
+                    chunk_path,
+                    transcript_path,
+                    description_path,
+                    target_fps=target_fps,
+                    save_processed_video=save_path
+                )
+                results.append(result)
+
+    score_dict = {}
+    for key in results[0]["evaluation"].keys():
+        score_dict[key] = []
+        for result in results:
+            score_dict[key].append(result["evaluation"][key]["score"])
+
+    evaluation = {}
+    for key, scores in score_dict.items():
+        evaluation[key] = {"score": calculate_geometric_mean(scores)}
+
+    result_json = {
+        "evaluation": evaluation,
+        "video_chunks": results
+    }
+    return result_json
+
+
+def extract_scores(data: Union[Dict, List]) -> List[int]:
+    """
+    Extract all score values from a nested dictionary or list structure.
+
+    Args:
+        data (Union[Dict, List]): The data structure to extract scores from.
+
+    Returns:
+        List[int]: List of extracted score values.
+    """
+    scores = []
+    if isinstance(data, dict):
+        for key, value in data.items():
+            if "chunks" in key:
+                continue
+            elif isinstance(value, dict) or isinstance(value, list):
+                scores.extend(extract_scores(value))
+            elif key == 'score':
+                scores.append(value)
+    elif isinstance(data, list):
+        for item in data:
+            scores.extend(extract_scores(item))
+    return scores
+
+
+def calculate_overall_score(result: Dict) -> float:
+    """
+    Calculate the overall score from evaluation results.
+
+    Args:
+        result (Dict): Dictionary containing evaluation results.
+
+    Returns:
+        float: The calculated overall score.
+    """
+    scores = extract_scores(result)
+    overall_score = calculate_geometric_mean(scores)
+    return overall_score
+
+
+def process_topic_name(topic_name: str) -> str:
+    """
+    Process a topic name by capitalizing words and handling special characters.
+
+    Args:
+        topic_name (str): The topic name to process.
+
+    Returns:
+        str: The processed topic name.
+    """
+    words = topic_name.replace("_s_", "'s_").split("_")
+    return " ".join([word.capitalize() for word in words])
+
+
+def merge_dicts(dict1: dict, dict2: dict) -> dict:
+    """
+    Recursively merge two dictionaries.
+
+    Args:
+        dict1 (dict): First dictionary.
+        dict2 (dict): Second dictionary.
+
+    Returns:
+        dict: Merged dictionary.
+    """
+    merged = dict1.copy()
+    for key, value in dict2.items():
+        if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):
+            merged[key] = merge_dicts(merged[key], value)
+        else:
+            merged[key] = value
+    return merged
+
+
+def process_theorem(models, file_path: str, eval_type: str, retry_limit: int,
+                    target_fps: int = None, use_parent_folder_as_topic: bool = False,
+                    output_folder: str = None) -> tuple[str, dict]:
+    """
+    Process a theorem file or directory for evaluation.
+
+    Args:
+        models: Dictionary of models for different evaluation types.
+        file_path (str): Path to the file or directory to evaluate.
+        eval_type (str): Type of evaluation to perform.
+        retry_limit (int): Number of retry attempts.
+        target_fps (int, optional): Target frames per second for video processing.
+        use_parent_folder_as_topic (bool, optional): Use parent folder name as topic.
+        output_folder (str, optional): Directory to store output files.
+
+    Returns:
+        tuple[str, dict]: Tuple of file name and evaluation results.
+    """
+    ext_map = {
+        'text': ('.txt', '.srt'),
+        'video': ('.mp4', '.mkv')
+    }
+
+    # Handle single file evaluation
+    if os.path.isfile(file_path):
+        file_ext = os.path.splitext(file_path)[1].lower()
+        file_name = os.path.basename(file_path)
+
+        if eval_type == "text" and file_ext in ext_map['text']:
+            return file_name, evaluate_text_file(models['text'], file_path, retry_limit)
+        elif eval_type == "video" and file_ext in ext_map['video']:
+            if use_parent_folder_as_topic:
+                topic_name = os.path.basename(os.path.dirname(file_path))
+            else:
+                topic_name = None
+            topic_name = process_topic_name(topic_name)
+            return file_name, evaluate_video_file(models['video'], file_path, None, topic_name, target_fps, output_folder)
+        elif eval_type == "image" and file_ext in ext_map['video']:
+            if use_parent_folder_as_topic:
+                topic_name = os.path.basename(os.path.dirname(file_path))
+            else:
+                topic_name = None
+            topic_name = process_topic_name(topic_name)
+            return file_name, evaluate_sampled_images(models['image'], file_path, topic_name, num_chunks=10, output_folder=output_folder)
+        elif eval_type == "all":
+            raise ValueError("Evaluation type 'all' is not supported for a single file. Try passing a folder with both a video and a subtitle file.")
+        else:
+            raise ValueError(f"File type of {file_path} does not match evaluation type {eval_type!r}")
+
+    # Handle directory evaluation
+    theorem_dir = file_path
+    all_files = os.listdir(theorem_dir)
+
+    # Look for transcript files, prioritizing .srt over .txt if both exist
+    transcript_file_candidates = [f for f in all_files if f.endswith(ext_map['text']) and not f.endswith('_scene_outline.txt')]
+    srt_files = [f for f in transcript_file_candidates if f.endswith('.srt')]
+    txt_files = [f for f in transcript_file_candidates if f.endswith('.txt')]
+
+    transcript_path = None
+    if srt_files:
+        transcript_path = os.path.join(theorem_dir, srt_files[0])
+    elif txt_files:
+        transcript_path = os.path.join(theorem_dir, txt_files[0])
+
+    video_file_candidates = [f for f in all_files if f.endswith(ext_map['video'])]
+    video_path = os.path.join(theorem_dir, video_file_candidates[0]) if len(video_file_candidates) == 1 else None
+
+    topic_name = os.path.basename(theorem_dir)
+    topic_name = process_topic_name(topic_name)
+
+    if not video_path:
+        print(f"Skipping {theorem_dir}: No video file found")
+        return None, None
+
+    text_result = video_result = image_result = None
+    if eval_type == "text" or eval_type == "all":
+        if transcript_path is None:
+            print(f"Warning: No suitable transcript file found in {theorem_dir}")
+        else:
+            text_result = evaluate_text_file(models['text'], transcript_path, retry_limit)
+    if eval_type == "video" or eval_type == "all":
+        assert video_path is not None, f"Expected 1 video file, got {len(video_file_candidates)} for {theorem_dir}"
+        video_result = evaluate_video_file(models['video'], video_path, transcript_path, topic_name, target_fps, output_folder)
+    if eval_type == "image" or eval_type == "all":
+        assert video_path is not None, f"Expected 1 video file, got {len(video_file_candidates)} for {theorem_dir}"
+        image_result = evaluate_sampled_images(models['image'], video_path, topic_name, num_chunks=10, output_folder=output_folder)
+    
+    if eval_type == "all":
+        result = {}
+        if text_result:
+            result = merge_dicts(result, text_result)
+        if video_result:
+            result = merge_dicts(result, video_result)
+        if image_result:
+            result = merge_dicts(result, image_result)
+        if result:
+            result["evaluation"]["overall_score"] = calculate_overall_score(result)
+    else:
+        result = text_result if eval_type == "text" else video_result if eval_type == "video" else image_result if eval_type == "image" else None
+
+    file_name = os.path.basename(theorem_dir)
+    return file_name, result
+
+
+def main():
+    """
+    Main function to run the evaluation script.
+
+    Parses command line arguments and orchestrates the evaluation process
+    for text, video, and image content using specified AI models.
+    """
+    parser = argparse.ArgumentParser(description='Automatic evaluation of theorem explanation videos with LLMs')
+    parser.add_argument('--model_text', type=str, 
+                       choices=ALLOWED_MODELS,
+                       default='azure/gpt-4o',
+                       help='Select the AI model to use for text evaluation')
+    parser.add_argument('--model_video', type=str,
+                       choices=['gemini/gemini-1.5-pro-002',
+                                'gemini/gemini-2.0-flash-exp',
+                                'gemini/gemini-2.0-pro-exp-02-05'],
+                       default='gemini/gemini-1.5-pro-002',
+                       help='Select the AI model to use for video evaluation')
+    parser.add_argument('--model_image', type=str,
+                       choices=ALLOWED_MODELS,
+                       default='azure/gpt-4o',
+                       help='Select the AI model to use for image evaluation')
+    parser.add_argument('--eval_type', type=str, choices=['text', 'video', 'image', 'all'], default='all', help='Type of evaluation to perform')
+    parser.add_argument('--file_path', type=str, help='Path to a file or a theorem folder', required=True)
+    parser.add_argument('--output_folder', type=str, help='Directory to store the evaluation files', required=True)
+    parser.add_argument('--retry_limit', type=int, default=3, help='Number of retry attempts for each inference')
+    parser.add_argument('--combine', action='store_true', help='Combine all results into a single JSON file')
+    parser.add_argument('--bulk_evaluate', action='store_true', help='Evaluate a folder of theorems together', default=False)
+    parser.add_argument('--target_fps', type=int, help='Target FPS for video processing. If not set, original video FPS will be used', required=False)
+    parser.add_argument('--use_parent_folder_as_topic', action='store_true', help='Use parent folder name as topic name for single file evaluation', default=True)
+    parser.add_argument('--max_workers', type=int, default=4, help='Maximum number of concurrent workers for parallel processing')
+
+    args = parser.parse_args()
+
+    # Initialize separate models
+    text_model = LiteLLMWrapper(
+        model_name=args.model_text,
+        temperature=0.0,
+    )
+    video_model = GeminiWrapper(
+        model_name=args.model_video,
+        temperature=0.0,
+    )
+    image_model = LiteLLMWrapper(
+        model_name=args.model_image,
+        temperature=0.0,
+    )
+
+    models = {
+        'text': text_model,
+        'video': video_model,
+        'image': image_model
+    }
+
+    theorem_dirs = []
+    if args.bulk_evaluate:
+        assert os.path.isdir(args.file_path), "File path must be a folder for --bulk_evaluate"
+        for root, dirnames, _ in os.walk(args.file_path):
+            if not any(f.endswith(".mp4") for f in os.listdir(root)):
+                continue
+
+            theorem_dirs.append(root)
+    elif os.path.isdir(args.file_path):
+        assert any(f.endswith(".mp4") for f in os.listdir(args.file_path)), "The provided folder must contain a video file"
+
+        theorem_dirs.append(args.file_path)
+
+    # Create output directory and its temp subdirectories if it doesn't exist
+    os.makedirs(args.output_folder, exist_ok=True)
+    moviepy_temp_dir = os.path.join(args.output_folder, "moviepy_temp")
+    os.makedirs(moviepy_temp_dir, exist_ok=True)
+    VideoFileClip.DEFAULT_TEMP_DIR = moviepy_temp_dir
+
+    processed_videos_dir = os.path.join(args.output_folder, "processed_videos")
+    os.makedirs(processed_videos_dir, exist_ok=True)
+
+    results = {}
+    if theorem_dirs:
+        for theorem_dir in theorem_dirs:
+            file_name, result = process_theorem(
+                models,
+                theorem_dir,
+                args.eval_type,
+                args.retry_limit,
+                args.target_fps,
+                args.use_parent_folder_as_topic,
+                args.output_folder
+            )
+
+            if result is not None:
+                results[file_name] = result
+
+                if not args.combine:
+                    save_individual_result(args.output_folder, file_name, result)
+    else:
+        file_name, result = process_theorem(
+            models, 
+            args.file_path, 
+            args.eval_type, 
+            args.retry_limit,
+            args.target_fps,
+            args.use_parent_folder_as_topic,
+            args.output_folder
+        )
+        
+        if result is not None:
+            results[file_name] = result
+
+            if not args.combine:
+                save_individual_result(args.output_folder, file_name, result)
+
+    if args.combine:
+        if len(results) > 1:
+            current_time = datetime.now().strftime("%Y%m%d_%H%M%S")
+            combined_file = f"evaluation_{current_time}.json"
+            combine_results(args.output_folder, combined_file, results)
+            print("Combining results completed.")
+        else:
+            for file_name, result in results.items():
+                save_individual_result(args.output_folder, file_name, result)
+    
+    os.rmdir(moviepy_temp_dir)
+
+
+if __name__ == "__main__":
+    main()

generate_video.py

@@ -0,0 +1,990 @@
+import os
+import json
+import asyncio
+import uuid
+from typing import Union, List, Dict, Optional, Protocol
+from dataclasses import dataclass
+from abc import ABC, abstractmethod
+import argparse
+import re
+from dotenv import load_dotenv
+
+from mllm_tools.litellm import LiteLLMWrapper
+from mllm_tools.openrouter import OpenRouterWrapper
+from src.core.video_planner import EnhancedVideoPlanner
+from src.core.code_generator import CodeGenerator  # Use existing CodeGenerator
+from src.core.video_renderer import VideoRenderer  # Use existing VideoRenderer
+from src.utils.utils import extract_xml
+from src.config.config import Config
+from task_generator import get_banned_reasonings
+from task_generator.prompts_raw import (_code_font_size, _code_disable, _code_limit, _prompt_manim_cheatsheet)
+
+# Load configuration
+load_dotenv(override=True)
+
+# Load allowed models
+allowed_models_path = os.path.join(os.path.dirname(__file__), 'src', 'utils', 'allowed_models.json')
+with open(allowed_models_path, 'r') as f:
+    allowed_models_data = json.load(f)
+    allowed_models = allowed_models_data.get("allowed_models", [])
+
+@dataclass
+class VideoGenerationConfig:
+    """Configuration for video generation pipeline."""
+    planner_model: str
+    scene_model: Optional[str] = None
+    helper_model: Optional[str] = None
+    output_dir: str = "output"
+    verbose: bool = False
+    use_rag: bool = False
+    use_context_learning: bool = False
+    context_learning_path: str = "data/context_learning"
+    chroma_db_path: str = "data/rag/chroma_db"
+    manim_docs_path: str = "data/rag/manim_docs"
+    embedding_model: str = "hf:ibm-granite/granite-embedding-30m-english"
+    use_visual_fix_code: bool = False
+    use_langfuse: bool = True
+    max_scene_concurrency: int = 5
+    max_topic_concurrency: int = 1
+    max_retries: int = 5
+    
+    # Renderer optimizations
+    enable_caching: bool = True
+    default_quality: str = "medium"
+    use_gpu_acceleration: bool = False
+    preview_mode: bool = False
+    max_concurrent_renders: int = 4
+
+# Protocols for dependency injection (Interface Segregation Principle)
+class ModelProvider(Protocol):
+    """Protocol for AI model providers."""
+    def __call__(self, prompt: str, **kwargs) -> str: ...
+
+class PlannerInterface(Protocol):
+    """Interface for video planners."""
+    async def generate_scene_outline(self, topic: str, description: str, session_id: str) -> str: ...
+    async def generate_scene_implementation_concurrently_enhanced(
+        self, topic: str, description: str, plan: str, session_id: str
+    ) -> List[str]: ...
+
+class CodeGeneratorInterface(Protocol):
+    """Interface for code generators."""
+    def generate_manim_code(self, **kwargs) -> tuple: ...
+    def fix_code_errors(self, **kwargs) -> tuple: ...
+    def visual_self_reflection(self, **kwargs) -> tuple: ...
+
+class RendererInterface(Protocol):
+    """Interface for video renderers."""
+    async def render_scene_optimized(self, **kwargs) -> tuple: ...
+    async def combine_videos_optimized(self, topic: str, **kwargs) -> str: ...
+
+# Factory for creating components (Factory Pattern)
+class ComponentFactory:
+    """Factory for creating video generation components."""
+    
+    @staticmethod
+    def create_model(model_name: str, config: VideoGenerationConfig) -> ModelProvider:
+        """Create AI model wrapper."""
+        # Use OpenRouter wrapper for OpenRouter models
+        if model_name.startswith('openrouter/'):
+            return OpenRouterWrapper(
+                model_name=model_name,
+                temperature=0.7,
+                print_cost=True,
+                verbose=config.verbose,
+                use_langfuse=config.use_langfuse
+            )
+        else:
+            # Use LiteLLM wrapper for other models
+            return LiteLLMWrapper(
+                model_name=model_name,
+                temperature=0.7,
+                print_cost=True,
+                verbose=config.verbose,
+                use_langfuse=config.use_langfuse
+            )
+    
+    @staticmethod
+    def create_planner(planner_model: ModelProvider, helper_model: ModelProvider, 
+                      config: VideoGenerationConfig, session_id: str) -> PlannerInterface:
+        """Create video planner with enhanced capabilities."""
+        return EnhancedVideoPlanner(
+            planner_model=planner_model,
+            helper_model=helper_model,
+            output_dir=config.output_dir,
+            print_response=config.verbose,
+            use_context_learning=config.use_context_learning,
+            context_learning_path=config.context_learning_path,
+            use_rag=config.use_rag,
+            session_id=session_id,
+            chroma_db_path=config.chroma_db_path,
+            manim_docs_path=config.manim_docs_path,
+            embedding_model=config.embedding_model,
+            use_langfuse=config.use_langfuse,
+            max_scene_concurrency=config.max_scene_concurrency,
+            max_step_concurrency=3,
+            enable_caching=config.enable_caching
+        )
+    
+    @staticmethod
+    def create_code_generator(scene_model: ModelProvider, helper_model: ModelProvider,
+                            config: VideoGenerationConfig, session_id: str) -> CodeGeneratorInterface:
+        """Create code generator with existing implementation."""
+        return CodeGenerator(  # Use existing CodeGenerator
+            scene_model=scene_model,
+            helper_model=helper_model,
+            output_dir=config.output_dir,
+            print_response=config.verbose,
+            use_rag=config.use_rag,
+            use_context_learning=config.use_context_learning,
+            context_learning_path=config.context_learning_path,
+            chroma_db_path=config.chroma_db_path,
+            manim_docs_path=config.manim_docs_path,
+            embedding_model=config.embedding_model,
+            use_visual_fix_code=config.use_visual_fix_code,
+            use_langfuse=config.use_langfuse,
+            session_id=session_id
+        )
+    
+    @staticmethod
+    def create_renderer(config: VideoGenerationConfig) -> RendererInterface:
+        """Create video renderer with existing implementation."""
+        return VideoRenderer(  # Use existing VideoRenderer
+            output_dir=config.output_dir,
+            print_response=config.verbose,
+            use_visual_fix_code=config.use_visual_fix_code
+        )
+
+# Enhanced VideoRenderer wrapper to add async methods
+class AsyncVideoRendererWrapper:
+    """Wrapper to add async functionality to existing VideoRenderer."""
+    
+    def __init__(self, renderer: VideoRenderer, config: VideoGenerationConfig):
+        self.renderer = renderer
+        self.config = config
+        self.render_stats = {'cache_hits': 0, 'total_renders': 0}
+    
+    async def render_scene_optimized(self, **kwargs) -> tuple:
+        """Async wrapper for scene rendering with intelligent error handling."""
+        # Extract parameters
+        code = kwargs.get('code')
+        file_prefix = kwargs.get('file_prefix')
+        curr_scene = kwargs.get('curr_scene')
+        curr_version = kwargs.get('curr_version', 1)
+        code_dir = kwargs.get('code_dir')
+        media_dir = kwargs.get('media_dir')
+        code_generator = kwargs.get('code_generator')
+        scene_implementation = kwargs.get('scene_implementation')
+        description = kwargs.get('description')
+        scene_outline = kwargs.get('scene_outline')
+        scene_trace_id = kwargs.get('scene_trace_id')
+        topic = kwargs.get('topic')
+        session_id = kwargs.get('session_id')
+        
+        # Use existing render_scene method with all parameters
+        loop = asyncio.get_event_loop()
+        result = await loop.run_in_executor(
+            None,
+            self.renderer.render_scene,
+            code,
+            file_prefix,
+            curr_scene,
+            curr_version,
+            code_dir,
+            media_dir,
+            False,  # use_visual_fix_code
+            None,   # visual_self_reflection_func
+            None,   # banned_reasonings
+            scene_trace_id,
+            topic,
+            session_id,
+            code_generator,
+            scene_implementation,
+            description,
+            scene_outline
+        )
+        
+        self.render_stats['total_renders'] += 1
+        return result
+    
+    async def render_multiple_scenes_parallel(self, scene_configs: List[Dict], 
+                                           max_concurrent: int = None) -> List[tuple]:
+        """Render multiple scenes in parallel."""
+        max_concurrent = max_concurrent or self.config.max_concurrent_renders
+        semaphore = asyncio.Semaphore(max_concurrent)
+        
+        async def render_single_scene(config):
+            async with semaphore:
+                return await self.render_scene_optimized(**config)
+        
+        print(f"🚀 Starting parallel rendering of {len(scene_configs)} scenes (max concurrent: {max_concurrent})")
+        
+        tasks = [render_single_scene(config) for config in scene_configs]
+        results = await asyncio.gather(*tasks, return_exceptions=True)
+        
+        successful = sum(1 for r in results if not isinstance(r, Exception))
+        print(f"📊 Render results: {successful}/{len(results)} scenes successful")
+        
+        return results
+    
+    async def combine_videos_optimized(self, topic: str, **kwargs) -> str:
+        """Async wrapper for video combination."""
+        loop = asyncio.get_event_loop()
+        return await loop.run_in_executor(
+            None,
+            self.renderer.combine_videos,
+            topic
+        )
+
+# Service classes (Single Responsibility Principle)
+class SessionManager:
+    """Manages session IDs for video generation."""
+    
+    def __init__(self, output_dir: str):
+        self.output_dir = output_dir
+    
+    def load_or_create_session_id(self) -> str:
+        """Load existing session ID or create new one."""
+        session_file = os.path.join(self.output_dir, "session_id.txt")
+        
+        if os.path.exists(session_file):
+            with open(session_file, 'r') as f:
+                session_id = f.read().strip()
+                print(f"📋 Loaded existing session ID: {session_id}")
+                return session_id
+        
+        session_id = str(uuid.uuid4())
+        os.makedirs(self.output_dir, exist_ok=True)
+        with open(session_file, 'w') as f:
+            f.write(session_id)
+        print(f"🆕 Created new session ID: {session_id}")
+        return session_id
+    
+    def save_topic_session_id(self, topic: str, session_id: str) -> None:
+        """Save session ID for specific topic."""
+        file_prefix = re.sub(r'[^a-z0-9_]+', '_', topic.lower())
+        topic_dir = os.path.join(self.output_dir, file_prefix)
+        os.makedirs(topic_dir, exist_ok=True)
+        
+        session_file = os.path.join(topic_dir, "session_id.txt")
+        with open(session_file, 'w') as f:
+            f.write(session_id)
+
+class SceneAnalyzer:
+    """Analyzes and manages scene information."""
+    
+    def __init__(self, output_dir: str):
+        self.output_dir = output_dir
+    
+    def load_implementation_plans(self, topic: str) -> Dict[int, Optional[str]]:
+        """Load implementation plans for each scene."""
+        file_prefix = re.sub(r'[^a-z0-9_]+', '_', topic.lower())
+        scene_outline_path = os.path.join(self.output_dir, file_prefix, f"{file_prefix}_scene_outline.txt")
+        
+        if not os.path.exists(scene_outline_path):
+            return {}
+        
+        with open(scene_outline_path, "r") as f:
+            scene_outline = f.read()
+        
+        scene_outline_content = extract_xml(scene_outline)
+        scene_count = len(re.findall(r'<SCENE_(\d+)>[^<]', scene_outline_content))
+        
+        implementation_plans = {}
+        for i in range(1, scene_count + 1):
+            plan_path = os.path.join(
+                self.output_dir, file_prefix, f"scene{i}", 
+                f"{file_prefix}_scene{i}_implementation_plan.txt"
+            )
+            if os.path.exists(plan_path):
+                with open(plan_path, "r") as f:
+                    implementation_plans[i] = f.read()
+                print(f"📄 Found existing implementation plan for scene {i}")
+            else:
+                implementation_plans[i] = None
+                print(f"❌ Missing implementation plan for scene {i}")
+        
+        return implementation_plans
+    
+    def analyze_scene_status(self, topic: str) -> Dict:
+        """Analyze status of all scenes for a topic."""
+        file_prefix = re.sub(r'[^a-z0-9_]+', '_', topic.lower())
+        
+        # Check scene outline
+        scene_outline_path = os.path.join(self.output_dir, file_prefix, f"{file_prefix}_scene_outline.txt")
+        has_scene_outline = os.path.exists(scene_outline_path)
+        
+        num_scenes = 0
+        if has_scene_outline:
+            with open(scene_outline_path, "r") as f:
+                scene_outline = f.read()
+            scene_outline_content = extract_xml(scene_outline)
+            num_scenes = len(re.findall(r'<SCENE_(\d+)>[^<]', scene_outline_content))
+        
+        # Analyze each scene
+        scene_status = []
+        implementation_plans = code_files = rendered_scenes = 0
+        
+        for i in range(1, num_scenes + 1):
+            scene_dir = os.path.join(self.output_dir, file_prefix, f"scene{i}")
+            
+            # Check implementation plan
+            plan_path = os.path.join(scene_dir, f"{file_prefix}_scene{i}_implementation_plan.txt")
+            has_plan = os.path.exists(plan_path)
+            if has_plan:
+                implementation_plans += 1
+            
+            # Check code files
+            code_dir = os.path.join(scene_dir, "code")
+            has_code = os.path.exists(code_dir) and any(f.endswith('.py') for f in os.listdir(code_dir))
+            if has_code:
+                code_files += 1
+            
+            # Check rendered videos
+            has_render = os.path.exists(os.path.join(scene_dir, "succ_rendered.txt"))
+            if has_render:
+                rendered_scenes += 1
+            
+            scene_status.append({
+                'scene_number': i,
+                'has_plan': has_plan,
+                'has_code': has_code,
+                'has_render': has_render
+            })
+        
+        # Check combined video
+        combined_video_path = os.path.join(self.output_dir, file_prefix, f"{file_prefix}_combined.mp4")
+        has_combined_video = os.path.exists(combined_video_path)
+        
+        return {
+            'topic': topic,
+            'has_scene_outline': has_scene_outline,
+            'total_scenes': num_scenes,
+            'implementation_plans': implementation_plans,
+            'code_files': code_files,
+            'rendered_scenes': rendered_scenes,
+            'has_combined_video': has_combined_video,
+            'scene_status': scene_status
+        }
+
+# Scene rendering wrapper for existing render_scene method
+class SceneRenderingService:
+    """Service for rendering individual scenes with existing VideoRenderer."""
+    
+    def __init__(self, renderer: VideoRenderer, code_generator: CodeGenerator, 
+                 banned_reasonings: List[str], config: VideoGenerationConfig):
+        self.renderer = renderer
+        self.code_generator = code_generator
+        self.banned_reasonings = banned_reasonings
+        self.config = config
+    
+    async def render_scene_with_code_generation(self, topic: str, description: str,
+                                              scene_outline: str, scene_implementation: str,
+                                              scene_number: int, file_prefix: str,
+                                              code_dir: str, media_dir: str,
+                                              scene_trace_id: str, session_id: str) -> tuple:
+        """Render a scene with code generation and error handling."""
+        
+        print(f"🎬 Processing scene {scene_number} for {topic}")
+        
+        try:
+            # Step 1: Generate Manim code
+            print(f"⚡ Generating code for scene {scene_number}")
+            code, _ = self.code_generator.generate_manim_code(
+                topic=topic,
+                description=description,
+                scene_outline=scene_outline,
+                scene_implementation=scene_implementation,
+                scene_number=scene_number,
+                scene_trace_id=scene_trace_id,
+                session_id=session_id
+            )
+            
+            # Step 2: Render with intelligent error handling (single attempt - renderer handles retries and fixes)
+            current_version = 1
+            
+            print(f"🎞️ Rendering scene {scene_number} with intelligent error handling")
+            
+            try:
+                # Use existing render_scene method with enhanced error handling
+                loop = asyncio.get_event_loop()
+                result_code, error = await loop.run_in_executor(
+                    None,
+                    self.renderer.render_scene,
+                    code,
+                    file_prefix,
+                    scene_number,
+                    current_version,
+                    code_dir,
+                    media_dir,
+                    False,  # use_visual_fix_code
+                    None,   # visual_self_reflection_func
+                    self.banned_reasonings,
+                    scene_trace_id,
+                    topic,
+                    session_id,
+                    self.code_generator,  # Pass code generator for intelligent error handling
+                    scene_implementation,  # Pass implementation for context
+                    description,  # Pass description for context
+                    scene_outline  # Pass scene outline for context
+                )
+                
+                if error is None:
+                    # Success - mark as rendered
+                    scene_dir = os.path.join(self.config.output_dir, file_prefix, f"scene{scene_number}")
+                    success_file = os.path.join(scene_dir, "succ_rendered.txt")
+                    with open(success_file, 'w') as f:
+                        f.write(f"Successfully rendered with intelligent error handling")
+                    
+                    print(f"✅ Scene {scene_number} rendered successfully")
+                    return result_code, None
+                else:
+                    # Error occurred even with intelligent retry/fix attempts
+                    print(f"❌ Scene {scene_number} failed after intelligent error handling: {error}")
+                    return result_code, error
+                
+            except Exception as e:
+                print(f"❌ Exception during scene {scene_number} rendering: {e}")
+                return code, str(e)
+            
+        except Exception as e:
+            print(f"❌ Fatal error in scene {scene_number}: {e}")
+            return None, str(e)
+
+# Main Video Generator (Open/Closed Principle - extensible via composition)
+class EnhancedVideoGenerator:
+    """Enhanced video generator following SOLID principles."""
+    
+    def __init__(self, config: VideoGenerationConfig):
+        self.config = config
+        self.session_manager = SessionManager(config.output_dir)
+        self.scene_analyzer = SceneAnalyzer(config.output_dir)
+        self.banned_reasonings = get_banned_reasonings()
+        
+        # Initialize session
+        self.session_id = self.session_manager.load_or_create_session_id()
+        
+        # Create AI models
+        self.planner_model = ComponentFactory.create_model(config.planner_model, config)
+        self.scene_model = ComponentFactory.create_model(
+            config.scene_model or config.planner_model, config
+        )
+        self.helper_model = ComponentFactory.create_model(
+            config.helper_model or config.planner_model, config
+        )
+        
+        # Create components using dependency injection
+        self.planner = ComponentFactory.create_planner(
+            self.planner_model, self.helper_model, config, self.session_id
+        )
+        self.code_generator = ComponentFactory.create_code_generator(
+            self.scene_model, self.helper_model, config, self.session_id
+        )
+        
+        # Create renderer with async wrapper
+        base_renderer = ComponentFactory.create_renderer(config)
+        self.renderer = AsyncVideoRendererWrapper(base_renderer, config)
+        
+        # Create scene rendering service
+        self.scene_service = SceneRenderingService(
+            base_renderer, self.code_generator, self.banned_reasonings, config
+        )
+        
+        # Concurrency control
+        self.scene_semaphore = asyncio.Semaphore(config.max_scene_concurrency)
+        
+        print(f"🚀 Enhanced VideoGenerator initialized with:")
+        print(f"   Planner: {config.planner_model}")
+        print(f"   Scene: {config.scene_model or config.planner_model}")
+        print(f"   Helper: {config.helper_model or config.planner_model}")
+        print(f"   Max Scene Concurrency: {config.max_scene_concurrency}")
+        print(f"   Caching: {'✅' if config.enable_caching else '❌'}")
+        print(f"   GPU Acceleration: {'✅' if config.use_gpu_acceleration else '❌'}")
+
+    async def generate_scene_outline(self, topic: str, description: str) -> str:
+        """Generate scene outline for topic."""
+        print(f"📝 Generating scene outline for: {topic}")
+        return await self.planner.generate_scene_outline(topic, description, self.session_id)
+
+    async def generate_video_pipeline(self, topic: str, description: str, 
+                                    only_plan: bool = False, 
+                                    specific_scenes: List[int] = None) -> None:
+        """Complete video generation pipeline with enhanced performance."""
+        
+        print(f"🎬 Starting enhanced video pipeline for: {topic}")
+        self.session_manager.save_topic_session_id(topic, self.session_id)
+        
+        file_prefix = re.sub(r'[^a-z0-9_]+', '_', topic.lower())
+        
+        # Step 1: Load or generate scene outline
+        scene_outline = await self._load_or_generate_outline(topic, description, file_prefix)
+        
+        # Step 2: Generate implementation plans
+        implementation_plans = await self._generate_implementation_plans(
+            topic, description, scene_outline, file_prefix, specific_scenes
+        )
+        
+        if only_plan:
+            print(f"📋 Plan-only mode completed for: {topic}")
+            return
+        
+        # Step 3: Render scenes with optimization
+        await self._render_scenes_optimized(
+            topic, description, scene_outline, implementation_plans, file_prefix
+        )
+        
+        # Step 4: Combine videos
+        await self._combine_videos_optimized(topic)
+        
+        print(f"✅ Enhanced video pipeline completed for: {topic}")
+
+    async def _load_or_generate_outline(self, topic: str, description: str, file_prefix: str) -> str:
+        """Load existing outline or generate new one."""
+        scene_outline_path = os.path.join(self.config.output_dir, file_prefix, f"{file_prefix}_scene_outline.txt")
+        
+        if os.path.exists(scene_outline_path):
+            with open(scene_outline_path, "r") as f:
+                scene_outline = f.read()
+            print(f"📄 Loaded existing scene outline for: {topic}")
+            
+            # Detect plugins if RAG is enabled
+            if self.config.use_rag and hasattr(self.planner, 'rag_integration'):
+                plugins = self.planner.rag_integration.detect_relevant_plugins(topic, description)
+                if plugins:
+                    self.planner.rag_integration.set_relevant_plugins(plugins)
+                    print(f"🔌 Detected relevant plugins: {plugins}")
+        else:
+            print(f"📝 Generating new scene outline for: {topic}")
+            scene_outline = await self.planner.generate_scene_outline(topic, description, self.session_id)
+            
+            os.makedirs(os.path.join(self.config.output_dir, file_prefix), exist_ok=True)
+            with open(scene_outline_path, "w") as f:
+                f.write(scene_outline)
+        
+        return scene_outline
+
+    async def _generate_implementation_plans(self, topic: str, description: str, 
+                                           scene_outline: str, file_prefix: str,
+                                           specific_scenes: List[int] = None) -> Dict[int, str]:
+        """Generate missing implementation plans."""
+        
+        # First, ensure the topic directory exists
+        topic_dir = os.path.join(self.config.output_dir, file_prefix)
+        os.makedirs(topic_dir, exist_ok=True)
+        
+        try:
+            implementation_plans_dict = self.scene_analyzer.load_implementation_plans(topic)
+            
+            if not implementation_plans_dict:
+                print(f"No existing implementation plans found for {topic}. Generating all plans from scratch.")
+                scene_outline_content = extract_xml(scene_outline)
+                scene_count = len(re.findall(r'<SCENE_(\d+)>[^<]', scene_outline_content))
+                
+                if scene_count == 0:
+                    print(f"⚠️ Warning: No scenes found in scene outline. Check the regex pattern and scene outline format.")
+                    print(f"Scene outline content: {scene_outline_content[:100]}...")
+                
+                print(f"Found {scene_count} scenes in the outline.")
+                implementation_plans_dict = {i: None for i in range(1, scene_count + 1)}
+            
+            # Find missing scenes
+            missing_scenes = [
+                scene_num for scene_num, plan in implementation_plans_dict.items()
+                if plan is None and (specific_scenes is None or scene_num in specific_scenes)
+            ]
+            
+            if missing_scenes:
+                print(f"📋 Generating implementation plans for scenes: {missing_scenes}")
+                
+                # Make sure scene directories exist for missing scenes
+                for scene_num in missing_scenes:
+                    scene_dir = os.path.join(topic_dir, f"scene{scene_num}")
+                    os.makedirs(scene_dir, exist_ok=True)
+                
+                # Use enhanced concurrent generation if available
+                if hasattr(self.planner, 'generate_scene_implementation_concurrently_enhanced'):
+                    try:
+                        all_plans = await self.planner.generate_scene_implementation_concurrently_enhanced(
+                            topic, description, scene_outline, self.session_id
+                        )
+                        
+                        if not all_plans:
+                            print(f"❌ Error: No implementation plans were returned!")
+                            return implementation_plans_dict
+                            
+                        # Update missing plans
+                        updated_count = 0
+                        for i, scene_num in enumerate(sorted(missing_scenes)):
+                            if i < len(all_plans):
+                                plan = all_plans[i]
+                                if isinstance(plan, str) and plan.strip():
+                                    implementation_plans_dict[scene_num] = plan
+                                    updated_count += 1
+                                else:
+                                    print(f"⚠️ Warning: Empty or invalid plan for scene {scene_num}")
+                        
+                        print(f"✅ Generated {updated_count}/{len(missing_scenes)} implementation plans")
+                        
+                    except Exception as e:
+                        print(f"❌ Error generating implementation plans: {str(e)}")
+                        
+                else:
+                    # Fallback to sequential generation
+                    print("⚠️ Using fallback sequential plan generation")
+                    # Implement sequential generation if needed
+            else:
+                print("✅ All implementation plans already exist.")
+                
+            return implementation_plans_dict
+            
+        except Exception as e:
+            print(f"❌ Fatal error in implementation plan generation: {str(e)}")
+            raise
+
+    async def _render_scenes_optimized(self, topic: str, description: str, 
+                                     scene_outline: str, implementation_plans: Dict[int, str],
+                                     file_prefix: str) -> None:
+        """Render scenes with enhanced optimization."""
+        
+        # Determine which scenes need processing
+        scenes_to_process = self._get_scenes_to_process(implementation_plans, file_prefix)
+        
+        if not scenes_to_process:
+            print(f"✅ No scenes need processing for: {topic}")
+            return
+        
+        print(f"🎬 Rendering {len(scenes_to_process)} scenes with optimization...")
+        
+        # Create render tasks
+        render_tasks = []
+        for scene_num, implementation_plan in scenes_to_process:
+            task = self._create_scene_render_task(
+                topic, description, scene_outline, implementation_plan,
+                scene_num, file_prefix
+            )
+            render_tasks.append(task)
+        
+        # Execute with concurrency control
+        semaphore = asyncio.Semaphore(self.config.max_concurrent_renders)
+        
+        async def execute_render_task(task):
+            async with semaphore:
+                return await task
+        
+        # Run all render tasks
+        results = await asyncio.gather(
+            *[execute_render_task(task) for task in render_tasks],
+            return_exceptions=True
+        )
+        
+        # Process results
+        successful_renders = sum(1 for r in results if not isinstance(r, Exception) and r[1] is None)
+        print(f"📊 Render results: {successful_renders}/{len(results)} scenes successful")
+
+    def _get_scenes_to_process(self, implementation_plans: Dict[int, str], 
+                             file_prefix: str) -> List[tuple]:
+        """Determine which scenes need processing."""
+        scenes_to_process = []
+        
+        for scene_num, implementation_plan in implementation_plans.items():
+            if implementation_plan is None:
+                continue
+                
+            scene_dir = os.path.join(self.config.output_dir, file_prefix, f"scene{scene_num}")
+            
+            # Check if scene already successfully rendered
+            if not os.path.exists(os.path.join(scene_dir, "succ_rendered.txt")):
+                scenes_to_process.append((scene_num, implementation_plan))
+        
+        return scenes_to_process
+
+    def _create_scene_render_task(self, topic: str, description: str, scene_outline: str,
+                                      implementation_plan: str, scene_num: int, file_prefix: str):
+        """Create render task for a scene."""
+        
+        # Generate or load scene trace ID
+        scene_dir = os.path.join(self.config.output_dir, file_prefix, f"scene{scene_num}")
+        subplan_dir = os.path.join(scene_dir, "subplans")
+        os.makedirs(subplan_dir, exist_ok=True)
+        
+        scene_trace_id_path = os.path.join(subplan_dir, "scene_trace_id.txt")
+        try:
+            with open(scene_trace_id_path, 'r') as f:
+                scene_trace_id = f.read().strip()
+        except FileNotFoundError:
+            scene_trace_id = str(uuid.uuid4())
+            with open(scene_trace_id_path, 'w') as f:
+                f.write(scene_trace_id)
+        
+        # Create directories
+        code_dir = os.path.join(scene_dir, "code")
+        media_dir = os.path.join(self.config.output_dir, file_prefix, "media")
+        os.makedirs(code_dir, exist_ok=True)
+        
+        # Return coroutine that will be awaited later
+        return self.scene_service.render_scene_with_code_generation(
+            topic=topic,
+            description=description,
+            scene_outline=scene_outline,
+            scene_implementation=implementation_plan,
+            scene_number=scene_num,
+            file_prefix=file_prefix,
+            code_dir=code_dir,
+            media_dir=media_dir,
+            scene_trace_id=scene_trace_id,
+            session_id=self.session_id
+        )
+
+    async def _combine_videos_optimized(self, topic: str) -> None:
+        """Combine videos with hardware acceleration."""
+        print(f"🎞️ Combining videos for: {topic}")
+        
+        try:
+            output_path = await self.renderer.combine_videos_optimized(
+                topic, use_hardware_acceleration=self.config.use_gpu_acceleration
+            )
+            print(f"✅ Combined video saved to: {output_path}")
+        except Exception as e:
+            print(f"❌ Error combining videos: {e}")
+
+    async def process_multiple_topics(self, topics_data: List[Dict], 
+                                    only_plan: bool = False,
+                                    specific_scenes: List[int] = None) -> None:
+        """Process multiple topics concurrently."""
+        
+        topic_semaphore = asyncio.Semaphore(self.config.max_topic_concurrency)
+        
+        async def process_single_topic(topic_data):
+            async with topic_semaphore:
+                topic = topic_data['theorem']
+                description = topic_data['description']
+                print(f"🎯 Processing topic: {topic}")
+                
+                try:
+                    await self.generate_video_pipeline(
+                        topic, description, only_plan=only_plan, 
+                        specific_scenes=specific_scenes
+                    )
+                    print(f"✅ Completed topic: {topic}")
+                except Exception as e:
+                    print(f"❌ Error processing {topic}: {e}")
+        
+        tasks = [process_single_topic(topic_data) for topic_data in topics_data]
+        await asyncio.gather(*tasks, return_exceptions=True)
+
+    def get_status_summary(self, topics_data: List[Dict]) -> None:
+        """Print comprehensive status summary."""
+        print("\n📊 Comprehensive Status Summary")
+        print("=" * 160)
+        
+        all_statuses = [
+            self.scene_analyzer.analyze_scene_status(topic_data['theorem']) 
+            for topic_data in topics_data
+        ]
+        
+        # Print header
+        print(f"{'Topic':<40} {'Outline':<8} {'Total':<8} {'Status (Plan/Code/Render)':<50} {'Combined':<10} {'Missing Components':<40}")
+        print("-" * 160)
+        
+        # Print each topic status
+        for status in all_statuses:
+            scene_status_str = ""
+            for scene in status['scene_status']:
+                scene_str = (
+                    ("P" if scene['has_plan'] else "-") +
+                    ("C" if scene['has_code'] else "-") +
+                    ("R" if scene['has_render'] else "-") + " "
+                )
+                scene_status_str += scene_str
+            
+            # Collect missing components
+            missing_components = self._format_missing_components(status['scene_status'])
+            
+            print(f"{status['topic'][:37]+'...' if len(status['topic'])>37 else status['topic']:<40} "
+                f"{'✓' if status['has_scene_outline'] else '✗':<8} "
+                f"{status['total_scenes']:<8} "
+                f"{scene_status_str[:47]+'...' if len(scene_status_str)>47 else scene_status_str:<50} "
+                f"{'✓' if status['has_combined_video'] else '✗':<10} "
+                f"{missing_components[:37]+'...' if len(missing_components)>37 else missing_components:<40}")
+        
+        # Print summary statistics
+        self._print_summary_statistics(all_statuses, len(topics_data))
+
+    def _format_missing_components(self, scene_status: List[Dict]) -> str:
+        """Format missing components string."""
+        missing_plans = [str(s['scene_number']) for s in scene_status if not s['has_plan']]
+        missing_code = [str(s['scene_number']) for s in scene_status if not s['has_code']]
+        missing_renders = [str(s['scene_number']) for s in scene_status if not s['has_render']]
+        
+        missing_str = []
+        if missing_plans:
+            missing_str.append(f"P:{','.join(missing_plans)}")
+        if missing_code:
+            missing_str.append(f"C:{','.join(missing_code)}")
+        if missing_renders:
+            missing_str.append(f"R:{','.join(missing_renders)}")
+        
+        return ' '.join(missing_str)
+
+    def _print_summary_statistics(self, all_statuses: List[Dict], total_topics: int) -> None:
+        """Print summary statistics."""
+        total_scenes = sum(status['total_scenes'] for status in all_statuses)
+        total_plans = sum(status['implementation_plans'] for status in all_statuses)
+        total_code = sum(status['code_files'] for status in all_statuses)
+        total_renders = sum(status['rendered_scenes'] for status in all_statuses)
+        total_combined = sum(1 for status in all_statuses if status['has_combined_video'])
+        
+        print("\n📈 Summary Statistics:")
+        print(f"   Total topics: {total_topics}")
+        print(f"   Total scenes: {total_scenes}")
+        print(f"   Completion rates:")
+        print(f"     Plans: {total_plans}/{total_scenes} ({total_plans/max(1,total_scenes)*100:.1f}%)")
+        print(f"     Code: {total_code}/{total_scenes} ({total_code/max(1,total_scenes)*100:.1f}%)")
+        print(f"     Renders: {total_renders}/{total_scenes} ({total_renders/max(1,total_scenes)*100:.1f}%)")
+        print(f"     Combined videos: {total_combined}/{total_topics} ({total_combined/max(1,total_topics)*100:.1f}%)")
+
+# Command-line interface
+class VideoGeneratorCLI:
+    """Command-line interface for video generation."""
+    
+    @staticmethod
+    def create_argument_parser() -> argparse.ArgumentParser:
+        """Create argument parser with all options."""
+        parser = argparse.ArgumentParser(description='Enhanced Manim Video Generator')
+        
+        # Model configuration
+        parser.add_argument('--model', type=str, choices=allowed_models,
+                          default='gemini/gemini-2.5-flash-preview-04-17', help='AI model to use')
+        parser.add_argument('--scene_model', type=str, choices=allowed_models,
+                          help='Specific model for scene generation')
+        parser.add_argument('--helper_model', type=str, choices=allowed_models,
+                          help='Helper model for additional tasks')
+        
+        # Input/Output
+        parser.add_argument('--topic', type=str, help='Single topic to process')
+        parser.add_argument('--context', type=str, help='Context for the topic')
+        parser.add_argument('--theorems_path', type=str, help='Path to theorems JSON file')
+        parser.add_argument('--output_dir', type=str, default=Config.OUTPUT_DIR, help='Output directory')
+        
+        # Processing options
+        parser.add_argument('--sample_size', type=int, help='Number of theorems to sample')
+        parser.add_argument('--scenes', nargs='+', type=int, help='Specific scenes to process')
+        parser.add_argument('--max_retries', type=int, default=5, help='Maximum retries for code generation')
+        
+        # Mode flags
+        parser.add_argument('--only_plan', action='store_true', help='Only generate plans')
+        parser.add_argument('--only_render', action='store_true', help='Only render scenes')
+        parser.add_argument('--only_combine', action='store_true', help='Only combine videos')
+        parser.add_argument('--check_status', action='store_true', help='Check status of all topics')
+        
+        # Performance options
+        parser.add_argument('--max_scene_concurrency', type=int, default=5, help='Max concurrent scenes')
+        parser.add_argument('--max_topic_concurrency', type=int, default=1, help='Max concurrent topics')
+        parser.add_argument('--max_concurrent_renders', type=int, default=4, help='Max concurrent renders')
+        parser.add_argument('--quality', choices=['preview', 'low', 'medium', 'high', 'production'],
+                          default='medium', help='Render quality preset')
+        
+        # Feature flags
+        parser.add_argument('--verbose', action='store_true', help='Verbose output')
+        parser.add_argument('--use_rag', action='store_true', help='Use RAG')
+        parser.add_argument('--use_context_learning', action='store_true', help='Use context learning')
+        parser.add_argument('--use_visual_fix_code', action='store_true', help='Use visual code fixing')
+        parser.add_argument('--use_langfuse', action='store_true', help='Enable Langfuse logging')
+        parser.add_argument('--enable_caching', action='store_true', default=True, help='Enable caching')
+        parser.add_argument('--use_gpu_acceleration', action='store_true', default=False, help='Use GPU acceleration')
+        parser.add_argument('--preview_mode', action='store_true', help='Enable preview mode')
+        
+        # Paths
+        parser.add_argument('--chroma_db_path', type=str, default=Config.CHROMA_DB_PATH, help='ChromaDB path')
+        parser.add_argument('--manim_docs_path', type=str, default=Config.MANIM_DOCS_PATH, help='Manim docs path')
+        parser.add_argument('--context_learning_path', type=str, default=Config.CONTEXT_LEARNING_PATH, help='Context learning path')
+        parser.add_argument('--embedding_model', type=str, default=Config.EMBEDDING_MODEL, help='Embedding model')
+        
+        return parser
+    
+    @staticmethod
+    def create_config_from_args(args) -> VideoGenerationConfig:
+        """Create configuration from command-line arguments."""
+        return VideoGenerationConfig(
+            planner_model=args.model,
+            scene_model=args.scene_model,
+            helper_model=args.helper_model,
+            output_dir=args.output_dir,
+            verbose=args.verbose,
+            use_rag=args.use_rag,
+            use_context_learning=args.use_context_learning,
+            context_learning_path=args.context_learning_path,
+            chroma_db_path=args.chroma_db_path,
+            manim_docs_path=args.manim_docs_path,
+            embedding_model=args.embedding_model,
+            use_visual_fix_code=args.use_visual_fix_code,
+            use_langfuse=args.use_langfuse,
+            max_scene_concurrency=args.max_scene_concurrency,
+            max_topic_concurrency=args.max_topic_concurrency,
+            max_retries=args.max_retries,
+            enable_caching=args.enable_caching,
+            default_quality=args.quality,
+            use_gpu_acceleration=args.use_gpu_acceleration,
+            preview_mode=args.preview_mode,
+            max_concurrent_renders=args.max_concurrent_renders
+        )
+
+async def main():
+    """Enhanced main function with improved error handling and performance."""
+    parser = VideoGeneratorCLI.create_argument_parser()
+    args = parser.parse_args()
+    
+    # Create configuration
+    config = VideoGeneratorCLI.create_config_from_args(args)
+    
+    # Initialize enhanced video generator
+    video_generator = EnhancedVideoGenerator(config)
+    
+    try:
+        if args.theorems_path:
+            await handle_multiple_topics(video_generator, args)
+        elif args.topic and args.context:
+            await handle_single_topic(video_generator, args)
+        else:
+            print("❌ Please provide either (--theorems_path) or (--topic and --context)")
+            return
+            
+    except Exception as e:
+        print(f"❌ Fatal error: {e}")
+        raise
+
+async def handle_multiple_topics(video_generator: EnhancedVideoGenerator, args):
+    """Handle processing of multiple topics."""
+    with open(args.theorems_path, "r") as f:
+        theorems = json.load(f)
+    
+    if args.sample_size:
+        theorems = theorems[:args.sample_size]
+    
+    if args.check_status:
+        video_generator.get_status_summary(theorems)
+        return
+    
+    if args.only_combine:
+        for theorem in theorems:
+            await video_generator._combine_videos_optimized(theorem['theorem'])
+    else:
+        await video_generator.process_multiple_topics(
+            theorems, 
+            only_plan=args.only_plan,
+            specific_scenes=args.scenes
+        )
+
+async def handle_single_topic(video_generator: EnhancedVideoGenerator, args):
+    """Handle processing of single topic."""
+    if args.only_combine:
+        await video_generator._combine_videos_optimized(args.topic)
+    else:
+        await video_generator.generate_video_pipeline(
+            args.topic,
+            args.context,
+            only_plan=args.only_plan
+        )
+
+if __name__ == "__main__":
+    asyncio.run(main())

gradio_app.py

@@ -0,0 +1,925 @@
+import os
+import gradio as gr
+import asyncio
+import json
+import uuid
+import threading
+import time
+from datetime import datetime
+import logging
+import traceback
+import re
+from typing import Dict, List, Optional
+
+from mllm_tools.litellm import LiteLLMWrapper
+from src.config.config import Config
+from generate_video import EnhancedVideoGenerator, VideoGenerationConfig, allowed_models
+
+# Configure logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+    handlers=[
+        logging.FileHandler("gradio_app.log"),
+        logging.StreamHandler()
+    ]
+)
+logger = logging.getLogger(__name__)
+
+# Create necessary directories
+os.makedirs("thumbnails", exist_ok=True)
+
+# Global dictionary to track job status
+job_status = {}
+
+# Model descriptions for better user understanding
+MODEL_DESCRIPTIONS = {
+    "gemini/gemini-1.5-pro-002": "🧠 Advanced reasoning, excellent for complex mathematical concepts",
+    "gemini/gemini-2.5-flash-preview-04-17": "⚡ Fast processing, good for quick prototypes",
+    "openai/gpt-4": "🎯 Reliable and consistent, great for educational content",
+    "openai/gpt-4o": "🚀 Latest OpenAI model with enhanced capabilities",
+    "anthropic/claude-3-5-sonnet-20241022": "📚 Excellent at detailed explanations and structured content",
+    "openrouter/openai/gpt-4o": "🌐 GPT-4o via OpenRouter - Powerful and versatile",
+    "openrouter/openai/gpt-4o-mini": "🌐 GPT-4o Mini via OpenRouter - Fast and cost-effective",
+    "openrouter/anthropic/claude-3.5-sonnet": "🌐 Claude 3.5 Sonnet via OpenRouter - Excellent reasoning",
+    "openrouter/anthropic/claude-3-haiku": "🌐 Claude 3 Haiku via OpenRouter - Quick responses",
+    "openrouter/google/gemini-pro-1.5": "🌐 Gemini Pro 1.5 via OpenRouter - Google's advanced model",
+    "openrouter/deepseek/deepseek-chat": "🌐 DeepSeek Chat via OpenRouter - Advanced conversation",
+    "openrouter/qwen/qwen-2.5-72b-instruct": "🌐 Qwen 2.5 72B via OpenRouter - Alibaba's flagship model",
+    "openrouter/meta-llama/llama-3.1-8b-instruct:free": "🌐 Llama 3.1 8B via OpenRouter - Free open source model",
+    "openrouter/microsoft/phi-3-mini-128k-instruct:free": "🌐 Phi-3 Mini via OpenRouter - Free Microsoft model"
+}
+
+def cancel_job(job_id):
+    """Cancel a running job."""
+    if job_id and job_id in job_status:
+        if job_status[job_id]['status'] in ['pending', 'initializing', 'planning', 'running']:
+            job_status[job_id]['status'] = 'cancelled'
+            job_status[job_id]['message'] = 'Job cancelled by user'
+            return f"Job {job_id} has been cancelled"
+    return "Job not found or cannot be cancelled"
+
+def delete_job(job_id):
+    """Delete a job from history."""
+    if job_id and job_id in job_status:
+        # Remove output files if they exist
+        job = job_status[job_id]
+        if job.get('output_file') and os.path.exists(job['output_file']):
+            try:
+                # Remove the entire output directory for this job
+                output_dir = os.path.dirname(job['output_file'])
+                import shutil
+                shutil.rmtree(output_dir, ignore_errors=True)
+            except Exception as e:
+                logger.error(f"Error removing output files: {e}")
+        
+        # Remove thumbnail
+        if job.get('thumbnail') and os.path.exists(job['thumbnail']):
+            try:
+                os.remove(job['thumbnail'])
+            except Exception as e:
+                logger.error(f"Error removing thumbnail: {e}")
+        
+        # Remove from job status
+        del job_status[job_id]
+        return f"Job {job_id} deleted successfully"
+    return "Job not found"
+
+def get_job_statistics():
+    """Get statistics about jobs."""
+    total_jobs = len(job_status)
+    completed_jobs = sum(1 for job in job_status.values() if job.get('status') == 'completed')
+    failed_jobs = sum(1 for job in job_status.values() if job.get('status') == 'failed')
+    running_jobs = sum(1 for job in job_status.values() if job.get('status') in ['pending', 'initializing', 'planning', 'running'])
+    
+    return {
+        'total': total_jobs,
+        'completed': completed_jobs,
+        'failed': failed_jobs,
+        'running': running_jobs
+    }
+
+def init_video_generator(params):
+    """Initialize the EnhancedVideoGenerator with the given parameters."""
+    model_name = params.get('model', 'gemini/gemini-2.5-flash-preview-04-17')
+    helper_model_name = params.get('helper_model', model_name)
+    verbose = params.get('verbose', True)  # Set verbose to True by default for better debugging
+    max_scene_concurrency = params.get('max_scene_concurrency', 1)
+    
+    # Create configuration for the enhanced video generator
+    config = VideoGenerationConfig(
+        planner_model=model_name,
+        scene_model=model_name,
+        helper_model=helper_model_name,
+        output_dir=params.get('output_dir', Config.OUTPUT_DIR),
+        verbose=verbose,
+        use_rag=params.get('use_rag', False),
+        use_context_learning=params.get('use_context_learning', False),
+        context_learning_path=params.get('context_learning_path', Config.CONTEXT_LEARNING_PATH),
+        chroma_db_path=params.get('chroma_db_path', Config.CHROMA_DB_PATH),
+        manim_docs_path=params.get('manim_docs_path', Config.MANIM_DOCS_PATH),
+        embedding_model=params.get('embedding_model', Config.EMBEDDING_MODEL),
+        use_visual_fix_code=params.get('use_visual_fix_code', True),  # Enable visual fix code by default
+        use_langfuse=params.get('use_langfuse', False),
+        max_scene_concurrency=max_scene_concurrency,
+        max_retries=params.get('max_retries', 3)
+    )
+    
+    # Initialize EnhancedVideoGenerator
+    video_generator = EnhancedVideoGenerator(config)
+    
+    return video_generator
+
+async def process_video_generation(job_id, params):
+    """Process video generation asynchronously."""
+    try:
+        # Update job status
+        job_status[job_id]['status'] = 'initializing'
+        job_status[job_id]['progress'] = 5
+        job_status[job_id]['message'] = 'Initializing video generator...'
+        
+        # Initialize video generator
+        video_generator = init_video_generator(params)
+        
+        # Extract video generation parameters
+        topic = params.get('topic')
+        description = params.get('description')
+        max_retries = int(params.get('max_retries', 3))
+        only_plan = params.get('only_plan', False)
+        
+        # Log job start
+        logger.info(f"Starting job {job_id} for topic: {topic}")
+        job_status[job_id]['status'] = 'planning'
+        job_status[job_id]['progress'] = 10
+        job_status[job_id]['message'] = 'Planning video scenes...'
+        
+        # Generate video pipeline
+        start_time = datetime.now()
+        logger.info(f"Running generate_video_pipeline for topic: {topic}")
+        
+        # Create an event loop for the async process
+        def update_progress_callback(progress, message):
+            job_status[job_id]['progress'] = progress
+            job_status[job_id]['message'] = message
+            logger.info(f"Job {job_id} progress: {progress}% - {message}")
+        
+        # Start a background task to periodically update progress
+        async def progress_update_task():
+            stages = [
+                (15, 'Creating scene outline...'),
+                (25, 'Generating implementation plans...'),
+                (35, 'Generating code for scenes...'),
+                (45, 'Compiling Manim code...'),
+                (60, 'Rendering scenes...'),
+                (80, 'Combining videos...'),
+                (90, 'Finalizing video...')
+            ]
+            
+            for progress, message in stages:
+                update_progress_callback(progress, message)
+                await asyncio.sleep(5)  # Wait between updates
+                
+                # Stop updating if job is complete or failed
+                if job_status[job_id]['status'] in ['completed', 'failed']:
+                    break
+        
+        # Start progress update task
+        progress_task = asyncio.create_task(progress_update_task())
+        
+        # Run the main video generation task with detailed logging
+        try:
+            logger.info(f"Starting video generation pipeline for job {job_id}")
+            update_progress_callback(15, 'Starting video generation pipeline...')
+            
+            await video_generator.generate_video_pipeline(
+                topic=topic,
+                description=description,
+                only_plan=only_plan
+            )
+                
+            logger.info(f"Video generation pipeline completed for job {job_id}")
+        except Exception as e:
+            logger.error(f"Error in video generation pipeline for job {job_id}: {str(e)}")
+            logger.error(traceback.format_exc())
+            raise
+        
+        # Cancel progress update task
+        if not progress_task.done():
+            progress_task.cancel()
+        
+        # Calculate processing time
+        end_time = datetime.now()
+        processing_time = (end_time - start_time).total_seconds()
+        
+        # Get output file path
+        file_prefix = topic.lower()
+        file_prefix = re.sub(r'[^a-z0-9_]+', '_', file_prefix)
+        output_file = os.path.join(
+            params.get('output_dir', Config.OUTPUT_DIR),
+            file_prefix,
+            f"{file_prefix}_combined.mp4"
+        )
+        
+        # Check if output file actually exists
+        if not os.path.exists(output_file):
+            alternative_output = None
+            # Look for any MP4 files that might have been generated
+            scene_dir = os.path.join(params.get('output_dir', Config.OUTPUT_DIR), file_prefix)
+            if os.path.exists(scene_dir):
+                for root, dirs, files in os.walk(scene_dir):
+                    for file in files:
+                        if file.endswith('.mp4'):
+                            alternative_output = os.path.join(root, file)
+                            logger.info(f"Combined video not found, but found alternative: {alternative_output}")
+                            break
+                    if alternative_output:
+                        break
+            
+            if alternative_output:
+                output_file = alternative_output
+            else:
+                logger.error(f"No video output file found for job {job_id}")
+                raise Exception("No video output was generated. Check Manim execution logs.")
+        
+        # Create a thumbnail from the video if it exists
+        thumbnail_path = None
+        if os.path.exists(output_file):
+            thumbnail_path = os.path.join("thumbnails", f"{job_id}.jpg")
+            try:
+                import subprocess
+                result = subprocess.run([
+                    'ffmpeg', '-i', output_file, 
+                    '-ss', '00:00:05', '-frames:v', '1', 
+                    thumbnail_path
+                ], capture_output=True, text=True)
+                
+                if result.returncode != 0:
+                    logger.error(f"Error creating thumbnail: {result.stderr}")
+                    thumbnail_path = None
+            except Exception as e:
+                logger.error(f"Error creating thumbnail: {str(e)}")
+                thumbnail_path = None
+        
+        # Get scene snapshots
+        scene_snapshots = []
+        scene_dir = os.path.join(params.get('output_dir', Config.OUTPUT_DIR), file_prefix)
+        if os.path.exists(scene_dir):
+            for i in range(1, 10):  # Check up to 10 possible scenes
+                scene_snapshot_dir = os.path.join(scene_dir, f"scene{i}")
+                if os.path.exists(scene_snapshot_dir):
+                    img_files = [f for f in os.listdir(scene_snapshot_dir) if f.endswith('.png')]
+                    if img_files:
+                        img_path = os.path.join(scene_snapshot_dir, img_files[-1])  # Get the last image
+                        scene_snapshots.append(img_path)
+        
+        # Update job status to completed
+        job_status[job_id].update({
+            'status': 'completed',
+            'progress': 100,
+            'message': 'Video generation completed',
+            'output_file': output_file if os.path.exists(output_file) else None,
+            'processing_time': processing_time,
+            'thumbnail': thumbnail_path,
+            'scene_snapshots': scene_snapshots
+        })
+        
+        logger.info(f"Job {job_id} completed successfully in {processing_time:.2f} seconds")
+        
+    except Exception as e:
+        # Handle exceptions
+        error_msg = str(e)
+        stack_trace = traceback.format_exc()
+        logger.error(f"Error in job {job_id}: {error_msg}\n{stack_trace}")
+        
+        job_status[job_id].update({
+            'status': 'failed',
+            'error': error_msg,
+            'stack_trace': stack_trace,
+            'message': f'Error: {error_msg[:100]}...' if len(error_msg) > 100 else f'Error: {error_msg}'
+        })
+
+def start_async_job(job_id, params):
+    """Start an async job in a separate thread."""
+    def run_async():
+        asyncio.run(process_video_generation(job_id, params))
+    
+    thread = threading.Thread(target=run_async)
+    thread.daemon = True
+    thread.start()
+    return thread
+
+def submit_job(topic, description, model, helper_model, max_retries, use_rag, use_visual_fix_code, temperature, use_context_learning, verbose, max_scene_concurrency):
+    """Submit a new video generation job."""
+    # Input validation
+    if not topic.strip():
+        return "❌ Error: Topic is required", None, gr.update(visible=False)
+    
+    if not description.strip():
+        return "❌ Error: Description is required", None, gr.update(visible=False)
+    
+    if len(topic.strip()) < 3:
+        return "❌ Error: Topic must be at least 3 characters long", None, gr.update(visible=False)
+    
+    if len(description.strip()) < 10:
+        return "❌ Error: Description must be at least 10 characters long", None, gr.update(visible=False)
+    
+    try:
+        # Generate job ID
+        job_id = str(uuid.uuid4())
+        
+        # Initialize job status
+        job_status[job_id] = {
+            'id': job_id,
+            'status': 'pending',
+            'topic': topic,
+            'description': description,
+            'model': model,
+            'start_time': datetime.now().isoformat(),
+            'progress': 0,
+            'message': 'Job submitted, waiting to start...'
+        }
+        
+        # Prepare parameters
+        params = {
+            'topic': topic,
+            'description': description,
+            'model': model,
+            'helper_model': helper_model,
+            'max_retries': max_retries,
+            'use_rag': use_rag,
+            'use_visual_fix_code': use_visual_fix_code,
+            'temperature': temperature,
+            'use_context_learning': use_context_learning,
+            'verbose': verbose,
+            'max_scene_concurrency': max_scene_concurrency,
+            'output_dir': Config.OUTPUT_DIR,
+        }
+        
+        # Start job asynchronously
+        start_async_job(job_id, params)
+        
+        return f"✅ Job submitted successfully. Job ID: {job_id}", job_id, gr.update(visible=True)
+    
+    except Exception as e:
+        logger.error(f"Error submitting job: {str(e)}")
+        return f"❌ Error: {str(e)}", None, gr.update(visible=False)
+
+def check_job_status(job_id):
+    """Check the status of a job."""
+    if not job_id or job_id not in job_status:
+        return {"status": "not_found", "message": "Job not found"}
+    
+    return job_status[job_id]
+
+def get_video_details(job_id):
+    """Get details of a completed video job."""
+    if not job_id or job_id not in job_status:
+        return None, None, None, [], "Job not found"
+    
+    job = job_status[job_id]
+    
+    if job['status'] != 'completed':
+        return None, None, None, [], f"Video not ready. Current status: {job['status']}"
+    
+    # Get video path, processing time, thumbnail and scene snapshots
+    video_path = job.get('output_file')
+    processing_time = job.get('processing_time', 0)
+    thumbnail = job.get('thumbnail')
+    scene_snapshots = job.get('scene_snapshots', [])
+    
+    if not video_path or not os.path.exists(video_path):
+        return None, None, None, [], "Video file not found"
+    
+    return video_path, processing_time, thumbnail, scene_snapshots, None
+
+def get_job_list():
+    """Get a list of all jobs."""
+    job_list = []
+    for job_id, job in job_status.items():
+        job_list.append({
+            'id': job_id,
+            'topic': job.get('topic', 'Unknown'),
+            'status': job.get('status', 'unknown'),
+            'start_time': job.get('start_time', ''),
+            'progress': job.get('progress', 0),
+            'message': job.get('message', '')
+        })
+    
+    # Sort by start time, most recent first
+    job_list.sort(key=lambda x: x.get('start_time', ''), reverse=True)
+    return job_list
+
+def format_status_message(job):
+    """Format status message for display."""
+    if not job:
+        return "No job selected"
+    
+    status = job.get('status', 'unknown')
+    progress = job.get('progress', 0)
+    message = job.get('message', '')
+    
+    status_emoji = {
+        'pending': '⏳',
+        'initializing': '🔄',
+        'planning': '🧠',
+        'running': '⚙️',
+        'completed': '✅',
+        'failed': '❌',
+        'unknown': '❓'
+    }.get(status, '❓')
+    
+    return f"{status_emoji} Status: {status.title()} ({progress}%)\n{message}"
+
+def update_status_display(job_id):
+    """Update the status display for a job."""
+    if not job_id:
+        return ("No job selected", 
+                gr.update(value=None), 
+                gr.update(visible=False), 
+                gr.update(visible=False), 
+                gr.update(value=[]),
+                gr.update(visible=False),
+                gr.update(visible=False))
+    
+    job = check_job_status(job_id)
+    status_message = format_status_message(job)
+    
+    # Check if the job is completed to show the video
+    if job.get('status') == 'completed' and job.get('output_file') and os.path.exists(job.get('output_file')):
+        video_path = job.get('output_file')
+        video_vis = True
+        thumbnail = job.get('thumbnail')
+        scene_snapshots = job.get('scene_snapshots', [])
+        processing_time = job.get('processing_time', 0)
+        
+        return (status_message, 
+                gr.update(value=video_path), 
+                gr.update(visible=video_vis), 
+                gr.update(visible=thumbnail is not None, value=thumbnail), 
+                gr.update(value=scene_snapshots),
+                gr.update(visible=True, value=f"⏱️ Processing Time: {processing_time:.2f} seconds"),
+                gr.update(visible=job.get('status') in ['pending', 'initializing', 'planning', 'running']))
+    
+    return (status_message, 
+            gr.update(value=None), 
+            gr.update(visible=False), 
+            gr.update(visible=False), 
+            gr.update(value=[]),
+            gr.update(visible=False),
+            gr.update(visible=job.get('status') in ['pending', 'initializing', 'planning', 'running']))
+
+# Create Gradio interface
+with gr.Blocks(
+    title="Theory2Manim Video Generator", 
+    theme=gr.themes.Soft(
+        primary_hue="blue",
+        secondary_hue="slate",
+        neutral_hue="slate",
+        font=gr.themes.GoogleFont("Inter")
+    ),
+    css="""
+    .main-header {
+        text-align: center;
+        background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+        color: white;
+        padding: 2rem;
+        border-radius: 1rem;
+        margin-bottom: 2rem;
+    }
+    .status-card {
+        border: 1px solid #e1e5e9;
+        border-radius: 0.5rem;
+        padding: 1rem;
+        background: #f8f9fa;
+    }
+    .metric-card {
+        border: 1px solid #e1e5e9;
+        border-radius: 0.5rem;
+        padding: 1rem;
+        text-align: center;
+        background: white;
+    }
+    .job-actions {
+        gap: 0.5rem;
+    }
+    """
+) as app:
+    
+    # Header
+    with gr.Row():
+        with gr.Column():
+            gr.HTML("""
+                <div class="main-header">
+                    <h1>🎬 Theory2Manim Video Generator</h1>
+                    <p>Transform mathematical and scientific concepts into engaging educational videos</p>
+                </div>
+            """)
+    
+    # Statistics Dashboard
+    with gr.Row():
+        stats_total = gr.Textbox(label="📊 Total Jobs", interactive=False, scale=1)
+        stats_completed = gr.Textbox(label="✅ Completed", interactive=False, scale=1)
+        stats_running = gr.Textbox(label="⚙️ Running", interactive=False, scale=1)
+        stats_failed = gr.Textbox(label="❌ Failed", interactive=False, scale=1)
+    
+    with gr.Tab("🎥 Generate Video"):
+        with gr.Row():
+            with gr.Column(scale=2):
+                with gr.Group():
+                    gr.Markdown("### 📝 Content Configuration")
+                    topic_input = gr.Textbox(
+                        label="📚 Topic", 
+                        placeholder="e.g., Fourier Transform, Calculus Derivatives, Quantum Mechanics",
+                        info="Enter the main topic for your educational video"
+                    )
+                    description_input = gr.Textbox(
+                        label="📋 Detailed Description", 
+                        placeholder="Provide a comprehensive description of what you want the video to cover, including specific concepts, examples, and target audience level...",
+                        lines=6,
+                        info="The more detailed your description, the better the AI can generate relevant content"
+                    )
+            
+            with gr.Column(scale=1):
+                with gr.Group():
+                    gr.Markdown("### ⚙️ AI Model Settings")
+                    model_input = gr.Dropdown(
+                        label="🤖 Primary AI Model", 
+                        choices=list(MODEL_DESCRIPTIONS.keys()),
+                        value="gemini/gemini-2.5-flash-preview-04-17",
+                        info="Choose the AI model for content generation"
+                    )
+                    model_description = gr.Markdown(MODEL_DESCRIPTIONS["gemini/gemini-2.5-flash-preview-04-17"])
+                    
+                    helper_model_input = gr.Dropdown(
+                        label="🔧 Helper Model", 
+                        choices=list(MODEL_DESCRIPTIONS.keys()),
+                        value="gemini/gemini-2.5-flash-preview-04-17",
+                        info="Model for auxiliary tasks"
+                    )
+                    
+                    temperature_input = gr.Slider(
+                        label="🌡️ Creativity (Temperature)", 
+                        minimum=0.0, 
+                        maximum=1.0, 
+                        value=0.7, 
+                        step=0.1,
+                        info="Lower = more focused, Higher = more creative"
+                    )
+        
+        with gr.Row():
+            with gr.Column():
+                with gr.Group():
+                    gr.Markdown("### 🔧 Advanced Settings")
+                    with gr.Row():
+                        max_retries_input = gr.Slider(
+                            label="🔄 Max Retries", 
+                            minimum=1, 
+                            maximum=10, 
+                            value=3, 
+                            step=1,
+                            info="Number of retry attempts for failed operations"
+                        )
+                        max_scene_concurrency_input = gr.Slider(
+                            label="⚡ Scene Concurrency", 
+                            minimum=1, 
+                            maximum=5, 
+                            value=1, 
+                            step=1,
+                            info="Number of scenes to process simultaneously"
+                        )
+                    
+                    with gr.Row():
+                        use_rag_input = gr.Checkbox(
+                            label="📚 Use RAG (Retrieval Augmented Generation)", 
+                            value=False,
+                            info="Enhance generation with relevant knowledge retrieval"
+                        )
+                        use_visual_fix_code_input = gr.Checkbox(
+                            label="🎨 Use Visual Code Fixing", 
+                            value=True,
+                            info="Automatically fix visual rendering issues"
+                        )
+                        use_context_learning_input = gr.Checkbox(
+                            label="🧠 Use Context Learning", 
+                            value=False,
+                            info="Learn from previous successful videos"
+                        )
+                        verbose_input = gr.Checkbox(
+                            label="📝 Verbose Logging", 
+                            value=True,
+                            info="Enable detailed logging for debugging"
+                        )
+        
+        with gr.Row():
+            with gr.Column(scale=3):
+                submit_btn = gr.Button("🚀 Generate Video", variant="primary", size="lg")
+            with gr.Column(scale=1):
+                clear_form_btn = gr.Button("🧹 Clear Form", variant="secondary")
+        
+        result_text = gr.Textbox(label="📋 Status", interactive=False)
+        job_id_output = gr.Textbox(label="Job ID", visible=False)
+        
+        with gr.Column(visible=False) as status_container:
+            with gr.Group():
+                gr.Markdown("### 📊 Job Progress")
+                with gr.Row():
+                    with gr.Column(scale=3):
+                        status_text = gr.Textbox(label="Current Status", interactive=False, elem_classes=["status-card"])
+                        processing_time_text = gr.Textbox(label="Processing Information", visible=False, interactive=False)
+                    with gr.Column(scale=1):
+                        with gr.Group():
+                            refresh_btn = gr.Button("🔄 Refresh Status", variant="secondary")
+                            cancel_btn = gr.Button("⏹️ Cancel Job", variant="stop", visible=False)
+                
+                with gr.Row():
+                    with gr.Column(scale=2):
+                        video_output = gr.Video(
+                            label="🎬 Generated Video", 
+                            interactive=False, 
+                            visible=False,
+                            show_download_button=True
+                        )
+                        thumbnail_preview = gr.Image(
+                            label="🖼️ Video Thumbnail", 
+                            visible=False,
+                            height=200
+                        )
+                    
+                    with gr.Column(scale=1):
+                        scene_gallery = gr.Gallery(
+                            label="🎨 Scene Previews", 
+                            columns=2, 
+                            object_fit="contain", 
+                            height=400,
+                            show_download_button=True
+                        )
+    
+    with gr.Tab("📂 Job History & Management"):
+        with gr.Row():
+            with gr.Column(scale=3):
+                refresh_jobs_btn = gr.Button("🔄 Refresh Job List", variant="secondary")
+            with gr.Column(scale=1):
+                clear_completed_btn = gr.Button("🧹 Clear Completed Jobs", variant="secondary")
+                clear_all_btn = gr.Button("🗑️ Clear All Jobs", variant="stop")
+        
+        jobs_table = gr.Dataframe(
+            headers=["ID", "Topic", "Status", "Progress (%)", "Start Time", "Message"],
+            datatype=["str", "str", "str", "number", "str", "str"],
+            interactive=False,
+            label="📋 Job History",
+            wrap=True
+        )
+        
+        with gr.Row():
+            with gr.Column():
+                select_job_btn = gr.Button("👁️ View Selected Job", variant="primary")
+                selected_job_id = gr.Textbox(label="Selected Job ID", visible=False)
+            with gr.Column():
+                delete_job_btn = gr.Button("🗑️ Delete Selected Job", variant="stop")
+                download_job_btn = gr.Button("💾 Download Job Results", variant="secondary")
+    
+    with gr.Tab("ℹ️ Help & Documentation"):
+        gr.Markdown("""
+        ## 🎯 How to Use Theory2Manim
+        
+        ### 📝 Step 1: Content Planning
+        - **Topic**: Enter a clear, specific topic (e.g., "Linear Algebra: Matrix Multiplication")
+        - **Description**: Provide detailed context about what you want covered:
+          - Target audience level (beginner, intermediate, advanced)
+          - Specific concepts to include
+          - Examples or applications to demonstrate
+          - Preferred video length or depth
+        
+        ### 🤖 Step 2: Model Selection
+        - **Gemini 1.5 Pro**: Best for complex mathematical reasoning
+        - **Gemini 2.0 Flash**: Fastest processing, good for simple topics
+        - **GPT-4**: Reliable and consistent output
+        - **Claude**: Excellent for detailed explanations
+        
+        ### ⚙️ Step 3: Advanced Settings
+        - **Temperature**: 0.3-0.5 for factual content, 0.7-0.9 for creative explanations
+        - **RAG**: Enable for topics requiring external knowledge
+        - **Visual Code Fixing**: Recommended for better video quality
+        - **Context Learning**: Use previous successful videos as examples
+        
+        ### 📊 Step 4: Monitor Progress
+        - Check the **Job History** tab to monitor all your video generation tasks
+        - Use **Refresh Status** to get real-time updates
+        - **Cancel** jobs if needed during processing
+        
+        ### 🎬 Step 5: Review Results
+        - Preview generated videos directly in the interface
+        - View scene breakdowns and thumbnails
+        - Download videos for offline use
+        
+        ## 💡 Tips for Best Results
+        1. **Be Specific**: Detailed descriptions lead to better videos
+        2. **Start Simple**: Try basic topics first to understand the system
+        3. **Use Examples**: Mention specific examples you want included
+        4. **Set Context**: Specify the educational level and background needed
+        5. **Review Settings**: Adjust temperature and models based on your content type
+        
+        ## 🔧 Troubleshooting
+        - **Job Stuck**: Try canceling and resubmitting with different settings
+        - **Poor Quality**: Use higher temperature or enable Visual Code Fixing
+        - **Missing Content**: Provide more detailed descriptions
+        - **Errors**: Check the verbose logs in the status messages
+        """)
+    
+    # Event handlers with improved functionality
+    def clear_form():
+        return ("", "", 0.7, False, True, False, True, 1, 1, "Form cleared! Ready for new input.")
+    
+    def update_model_description(model):
+        return MODEL_DESCRIPTIONS.get(model, "No description available")
+    
+    def update_stats():
+        stats = get_job_statistics()
+        return (f"{stats['total']}", 
+                f"{stats['completed']}", 
+                f"{stats['running']}", 
+                f"{stats['failed']}")
+    
+    def clear_completed_jobs():
+        completed_jobs = [job_id for job_id, job in job_status.items() 
+                         if job.get('status') == 'completed']
+        for job_id in completed_jobs:
+            delete_job(job_id)
+        return f"Cleared {len(completed_jobs)} completed jobs"
+    
+    def clear_all_jobs():
+        count = len(job_status)
+        job_status.clear()
+        return f"Cleared all {count} jobs"
+    
+    # Connect event handlers
+    model_input.change(
+        fn=update_model_description,
+        inputs=[model_input],
+        outputs=[model_description]
+    )
+    
+    clear_form_btn.click(
+        fn=clear_form,
+        outputs=[topic_input, description_input, temperature_input, 
+                use_rag_input, use_visual_fix_code_input, use_context_learning_input, 
+                verbose_input, max_retries_input, max_scene_concurrency_input, result_text]
+    )
+    
+    submit_btn.click(
+        fn=submit_job,
+        inputs=[
+            topic_input, description_input, model_input, helper_model_input, max_retries_input,
+            use_rag_input, use_visual_fix_code_input, temperature_input, use_context_learning_input,
+            verbose_input, max_scene_concurrency_input
+        ],
+        outputs=[result_text, job_id_output, status_container]
+    ).then(
+        fn=update_status_display,
+        inputs=[job_id_output],
+        outputs=[status_text, video_output, video_output, thumbnail_preview, scene_gallery, processing_time_text, cancel_btn]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    refresh_btn.click(
+        fn=update_status_display,
+        inputs=[job_id_output],
+        outputs=[status_text, video_output, video_output, thumbnail_preview, scene_gallery, processing_time_text, cancel_btn]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    cancel_btn.click(
+        fn=cancel_job,
+        inputs=[job_id_output],
+        outputs=[result_text]
+    ).then(
+        fn=update_status_display,
+        inputs=[job_id_output],
+        outputs=[status_text, video_output, video_output, thumbnail_preview, scene_gallery, processing_time_text, cancel_btn]
+    )
+    
+    # Job history tab functions
+    def load_job_list():
+        jobs = get_job_list()
+        rows = []
+        for job in jobs:
+            start_time = job.get('start_time', '')
+            if start_time:
+                try:
+                    dt = datetime.fromisoformat(start_time.replace('Z', '+00:00'))
+                    formatted_time = dt.strftime('%Y-%m-%d %H:%M:%S')
+                except:
+                    formatted_time = start_time
+            else:
+                formatted_time = 'Unknown'
+            
+            rows.append([
+                job['id'][:8] + '...', 
+                job['topic'][:50] + ('...' if len(job['topic']) > 50 else ''), 
+                job['status'].title(), 
+                job['progress'], 
+                formatted_time,
+                job['message'][:100] + ('...' if len(job['message']) > 100 else '')
+            ])
+        return rows
+    
+    def select_job(evt: gr.EventData):
+        if not evt:
+            return "", "No job selected"
+        
+        selected_row = evt.index[0] if hasattr(evt, 'index') and evt.index else 0
+        jobs = get_job_list()
+        if selected_row < len(jobs):
+            return jobs[selected_row]['id'], f"Selected job: {jobs[selected_row]['topic']}"
+        return "", "No job selected"
+    
+    def delete_selected_job(job_id):
+        if job_id:
+            result = delete_job(job_id)
+            return result, ""
+        return "No job selected", ""
+    
+    refresh_jobs_btn.click(
+        fn=load_job_list,
+        outputs=[jobs_table]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    jobs_table.select(
+        fn=select_job,
+        outputs=[selected_job_id, result_text]
+    )
+    
+    select_job_btn.click(
+        fn=lambda x: gr.update(visible=True) if x else gr.update(visible=False),
+        inputs=[selected_job_id],
+        outputs=[status_container]
+    ).then(
+        fn=update_status_display,
+        inputs=[selected_job_id],
+        outputs=[status_text, video_output, video_output, thumbnail_preview, scene_gallery, processing_time_text, cancel_btn]
+    )
+    
+    delete_job_btn.click(
+        fn=delete_selected_job,
+        inputs=[selected_job_id],
+        outputs=[result_text, selected_job_id]
+    ).then(
+        fn=load_job_list,
+        outputs=[jobs_table]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    clear_completed_btn.click(
+        fn=clear_completed_jobs,
+        outputs=[result_text]
+    ).then(
+        fn=load_job_list,
+        outputs=[jobs_table]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    clear_all_btn.click(
+        fn=clear_all_jobs,
+        outputs=[result_text]
+    ).then(
+        fn=load_job_list,
+        outputs=[jobs_table]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    # Set up polling for status updates
+    app.load(
+        fn=load_job_list,
+        outputs=[jobs_table]
+    ).then(
+        fn=update_stats,
+        outputs=[stats_total, stats_completed, stats_running, stats_failed]
+    )
+    
+    # Load on app start
+    def on_app_start():
+        if not os.path.exists("thumbnails"):
+            os.makedirs("thumbnails", exist_ok=True)
+        return "🎬 Welcome to Theory2Manim Video Generator! Ready to create amazing educational videos."
+    
+    app.load(
+        fn=on_app_start,
+        outputs=[result_text]
+    )
+
+# Launch the app
+if __name__ == "__main__":
+    app.queue().launch(server_name="127.0.0.1", share=False)

requirements.txt

@@ -0,0 +1,99 @@
+annotated-types~=0.7.0
+azure-cognitiveservices-speech~=1.41.1
+cachetools~=5.5.0
+certifi~=2024.8.30
+charset-normalizer~=3.4.0
+click~=8.1.7
+cloup~=3.0.5
+Cython~=3.0.11
+decorator~=5.1.1
+glcontext~=3.0.0
+google-ai-generativelanguage~=0.6.10
+google-api-core~=2.22.0
+google-api-python-client~=2.151.0
+google-auth~=2.35.0
+google-auth-httplib2~=0.2.0
+google-generativeai~=0.8.3
+googleapis-common-protos~=1.65.0
+grpcio~=1.67.1
+grpcio-status~=1.67.1
+gTTS~=2.5.3
+httplib2~=0.22.0
+idna~=3.10
+isosurfaces~=0.1.2
+manim~=0.18.1
+manim-voiceover~=0.3.7
+ManimPango~=0.6.0 # sudo apt-get install libsdl-pango-dev    if you dont have pangocairo
+mapbox_earcut~=1.0.2
+markdown-it-py~=3.0.0
+mdurl~=0.1.2
+moderngl~=5.12.0
+multipledispatch~=1.0.0
+mutagen~=1.47.0
+networkx~=3.4.2
+numpy~=2.2.2
+pillow
+proto-plus~=1.25.0
+protobuf~=5.28.3
+pyasn1~=0.6.1
+pyasn1_modules~=0.4.1
+PyAudio~=0.2.14 #required brew install portaudio for mac
+pycairo~=1.27.0
+pydantic~=2.9.2
+pydantic_core~=2.23.4
+pydub~=0.25.1
+pyglet~=2.0.18
+Pygments~=2.18.0
+#pyobjc-core~=10.3.1 # only for mac
+#pyobjc-framework-Cocoa~=10.3.1 # only for mac
+pyparsing~=3.2.0
+pyrr~=0.10.3
+python-dotenv~=0.21.1
+python-slugify~=8.0.4
+requests~=2.32.3
+rich~=13.9.3
+rsa~=4.9
+scipy~=1.14.1
+screeninfo~=0.8.1
+skia-pathops~=0.8.0.post2
+sox~=1.5.0
+srt~=3.5.3
+svgelements~=1.9.6
+text-unidecode~=1.3
+tqdm~=4.66.5
+typing_extensions~=4.12.2
+uritemplate~=4.1.1
+urllib3~=2.2.3
+watchdog~=5.0.3
+inquirer
+openai~=1.61.0
+tiktoken~=0.8.0
+timm
+sentencepiece
+transformers
+sentence-transformers
+litellm~=1.60.5
+pysrt
+moviepy~=2.1.2
+yt-dlp
+imageio_ffmpeg~=0.5.1
+langchain~=0.3.14
+langchain_community~=0.3.14
+SpeechRecognition~=3.14.1
+boto3~=1.36.9
+manim-physics~=0.4.0
+manim-ml~=0.0.24
+manim-chemistry~=0.4.4
+manim-dsa~=0.2.0
+manim-circuit~=0.0.3
+langfuse~=2.58.1
+chromadb~=0.6.3
+google-cloud-aiplatform~=1.79.0
+cairosvg
+pylatexenc~=2.10
+ffmpeg-python~=0.2.0
+kokoro-onnx # if you have a GPU, otherwise kokoro-onnx
+soundfile~=0.13.1
+krippendorff~=0.8.1
+statsmodels~=0.14.4
+opencv-python~=4.11.0