|
|
--- |
|
|
license: mit |
|
|
title: Data Extractor Using Gemini |
|
|
sdk: docker |
|
|
emoji: 🏃 |
|
|
colorFrom: yellow |
|
|
colorTo: blue |
|
|
--- |
|
|
# 📊 Financial Data Extractor Using Gemini |
|
|
|
|
|
A powerful AI-driven financial document analysis system that automatically extracts, organizes, and generates professional Excel reports from financial documents using Google's Gemini AI models. |
|
|
|
|
|
## 🚀 Features |
|
|
|
|
|
### Core Functionality |
|
|
|
|
|
- **📄 Multi-format Document Support**: PDF, DOCX, TXT, and image files |
|
|
- **🔍 Intelligent Data Extraction**: AI-powered extraction of financial data points |
|
|
- **📊 Smart Data Organization**: Automatic categorization into 12+ financial categories |
|
|
- **💻 Excel Report Generation**: Professional multi-worksheet Excel reports with charts |
|
|
- **🎯 Real-time Processing**: Live streaming interface with progress tracking |
|
|
|
|
|
### Advanced Capabilities |
|
|
|
|
|
- **🤖 Multi-Agent Workflow**: Specialized AI agents for extraction, arrangement, and code generation |
|
|
- **💾 Session Management**: Persistent storage with SQLite caching |
|
|
- **🔄 Auto-shutdown**: Intelligent resource management for cloud deployments |
|
|
- **📱 Modern UI**: Beautiful Gradio-based web interface |
|
|
- **🌐 Cross-platform**: Works on Windows, Mac, and Linux |
|
|
- **🐳 Docker Support**: Containerized deployment ready |
|
|
|
|
|
## 🏗️ Architecture |
|
|
|
|
|
The system uses a sophisticated multi-agent workflow powered by the Agno framework: |
|
|
|
|
|
``` |
|
|
📄 Document Upload |
|
|
↓ |
|
|
🔍 Data Extractor Agent |
|
|
↓ (Structured Financial Data) |
|
|
📊 Data Arranger Agent |
|
|
↓ (Organized Categories) |
|
|
💻 Code Generator Agent |
|
|
↓ (Python Excel Code) |
|
|
📊 Excel Report Output |
|
|
``` |
|
|
|
|
|
### Agent Specialization |
|
|
|
|
|
- **Data Extractor**: Extracts financial data points with confidence scoring |
|
|
- **Data Arranger**: Organizes data into 12+ professional categories |
|
|
- **Code Generator**: Creates Python code for Excel report generation |
|
|
|
|
|
## 📋 Requirements |
|
|
|
|
|
### System Requirements |
|
|
|
|
|
- Python 3.8+ |
|
|
- Google API Key (for Gemini models) |
|
|
- 2GB+ RAM recommended |
|
|
- Cross-platform compatible |
|
|
|
|
|
### Dependencies |
|
|
|
|
|
``` |
|
|
agno>=1.7.4 |
|
|
gradio |
|
|
google-generativeai |
|
|
PyPDF2 |
|
|
Pillow |
|
|
python-dotenv |
|
|
pandas |
|
|
matplotlib |
|
|
openpyxl |
|
|
python-docx |
|
|
lxml |
|
|
markdown |
|
|
requests |
|
|
seaborn |
|
|
sqlalchemy |
|
|
websockets |
|
|
``` |
|
|
|
|
|
## 🚀 Quick Start |
|
|
|
|
|
### 1. Clone the Repository |
|
|
|
|
|
```bash |
|
|
git clone <repository-url> |
|
|
cd Data_Extractor_Using_Gemini |
|
|
``` |
|
|
|
|
|
### 2. Install Dependencies |
|
|
|
|
|
```bash |
|
|
pip install -r requirements.txt |
|
|
``` |
|
|
|
|
|
### 3. Configure Environment |
|
|
|
|
|
Create a `.env` file: |
|
|
|
|
|
```env |
|
|
GOOGLE_API_KEY=your_gemini_api_key_here |
|
|
``` |
|
|
|
|
|
### 4. Run the Application |
|
|
|
|
|
```bash |
|
|
python app.py |
|
|
``` |
|
|
|
|
|
The application will be available at `http://localhost:7860` |
|
|
|
|
|
## 🐳 Docker Deployment |
|
|
|
|
|
### Build and Run |
|
|
|
|
|
```bash |
|
|
docker build -t financial-extractor . |
|
|
docker run -p 7860:7860 --env-file .env financial-extractor |
|
|
``` |
|
|
|
|
|
### Environment Variables |
|
|
|
|
|
- `GOOGLE_API_KEY`: Your Google Gemini API key |
|
|
- `INACTIVITY_TIMEOUT_MINUTES`: Auto-shutdown timeout (default: 30) |
|
|
|
|
|
## 📖 Usage Guide |
|
|
|
|
|
### 1. Upload Document |
|
|
|
|
|
- Drag and drop or select your financial document |
|
|
- Supported formats: PDF, DOCX, TXT, PNG, JPG, JPEG |
|
|
|
|
|
### 2. Select Processing Mode |
|
|
|
|
|
- **Quick Analysis**: Standard extraction and organization |
|
|
- **Custom Prompts**: Use predefined prompt templates for specific document types |
|
|
|
|
|
### 3. Monitor Progress |
|
|
|
|
|
- Real-time streaming interface shows each processing step |
|
|
- Progress indicators for all workflow stages |
|
|
- Live terminal output for code execution |
|
|
|
|
|
### 4. Download Results |
|
|
|
|
|
- Professional Excel report with multiple worksheets |
|
|
- Organized data categories with charts and formatting |
|
|
- All intermediate files available for download |
|
|
|
|
|
## 📊 Output Structure |
|
|
|
|
|
The generated Excel reports include: |
|
|
|
|
|
### Worksheets |
|
|
|
|
|
- **Summary**: Executive overview with key metrics |
|
|
- **Revenue**: Income and revenue streams |
|
|
- **Expenses**: Operating and non-operating expenses |
|
|
- **Assets**: Current and non-current assets |
|
|
- **Liabilities**: Short-term and long-term liabilities |
|
|
- **Equity**: Shareholder equity components |
|
|
- **Cash Flow**: Cash flow statements |
|
|
- **Ratios**: Financial ratio analysis |
|
|
- **Charts**: Visual representations of key data |
|
|
- **Raw Data**: Original extracted data points |
|
|
|
|
|
### Features |
|
|
|
|
|
- Professional formatting with consistent styling |
|
|
- Interactive charts and visualizations |
|
|
- Dynamic period handling (auto-detects years/quarters) |
|
|
- Cross-referenced data validation |
|
|
- Print-ready layouts |
|
|
|
|
|
## 🔧 Configuration |
|
|
|
|
|
### Model Settings |
|
|
|
|
|
Configure AI models in `config/settings.py`: |
|
|
|
|
|
- Data Extractor Model |
|
|
- Data Arranger Model |
|
|
- Code Generator Model |
|
|
- Thinking budgets and retry settings |
|
|
|
|
|
### Prompt Customization |
|
|
|
|
|
Customize agent instructions in `instructions/agents/`: |
|
|
|
|
|
- `data_extractor.md`: Data extraction prompts |
|
|
- `data_arranger.md`: Data organization prompts |
|
|
- `code_generator.md`: Excel generation prompts |
|
|
|
|
|
### Workflow Configuration |
|
|
|
|
|
Modify workflow behavior in `workflow/financial_workflow.py`: |
|
|
|
|
|
- Agent configurations |
|
|
- Tool assignments |
|
|
- Output formats |
|
|
|
|
|
## 🛠️ Development |
|
|
|
|
|
### Project Structure |
|
|
|
|
|
``` |
|
|
├── app.py # Main Gradio application |
|
|
├── workflow/ # Core workflow implementation |
|
|
├── instructions/ # Agent instruction templates |
|
|
├── prompts/ # Prompt gallery configurations |
|
|
├── config/ # Application settings |
|
|
├── utils/ # Utility functions |
|
|
├── static/ # Static assets |
|
|
├── models/ # Data models |
|
|
└── terminal_stream.py # Real-time terminal streaming |
|
|
``` |
|
|
|
|
|
### Key Components |
|
|
|
|
|
- **WorkflowUI**: Main interface controller |
|
|
- **FinancialDocumentWorkflow**: Core processing pipeline |
|
|
- **AutoShutdownManager**: Resource management |
|
|
- **TerminalLogHandler**: Real-time logging |
|
|
- **PromptGallery**: Template management |
|
|
|
|
|
## 🔒 Security & Privacy |
|
|
|
|
|
- **Local Processing**: All document processing happens locally |
|
|
- **No Data Storage**: Documents are processed and cleaned up automatically |
|
|
- **API Key Security**: Environment-based configuration |
|
|
- **Session Isolation**: Each session has isolated temporary directories |
|
|
|
|
|
## 🌐 Deployment Options |
|
|
|
|
|
### Local Development |
|
|
|
|
|
```bash |
|
|
python app.py |
|
|
``` |
|
|
|
|
|
### Production (Gunicorn) |
|
|
|
|
|
```bash |
|
|
gunicorn -w 4 -b 0.0.0.0:7860 app:app |
|
|
``` |
|
|
|
|
|
### Cloud Platforms |
|
|
|
|
|
- **Hugging Face Spaces**: Ready for deployment |
|
|
- **Google Cloud Run**: Containerized deployment |
|
|
- **AWS/Azure**: Standard container deployment |
|
|
|
|
|
## 🤝 Contributing |
|
|
|
|
|
1. Fork the repository |
|
|
2. Create a feature branch |
|
|
3. Make your changes |
|
|
4. Add tests if applicable |
|
|
5. Submit a pull request |
|
|
|
|
|
## 📝 License |
|
|
|
|
|
This project is licensed under the MIT License - see the LICENSE file for details. |
|
|
|
|
|
## 🆘 Support |
|
|
|
|
|
### Common Issues |
|
|
|
|
|
- **API Key Errors**: Ensure your Google API key is valid and has Gemini access |
|
|
- **Memory Issues**: Increase system RAM or reduce document size |
|
|
- **Processing Timeouts**: Check network connectivity and API quotas |
