README.md

---
title: AI Data Analysis with SmoLagent
emoji: 🤖
colorFrom: purple
colorTo: green
sdk: gradio
sdk_version: 5.37.0
app_file: app.py
pinned: false
---

# Singtel Use Case 1: OutSystems Log Analysis with SmoLagent AI

An intelligent data analysis application specifically designed for analyzing OutSystems application logs using SmoLagent AI framework. This project focuses on extracting insights from 6 months of OutSystems log data to identify patterns, errors, and optimization opportunities.

## Features

🤖 **AI-Powered Log Analysis**: Uses SmoLagent CodeAgent with DuckDuckGo search for intelligent log analysis  
📊 **Interactive Visualizations**: Correlation heatmaps, distribution plots, and error pattern analysis  
📈 **OutSystems-Specific Analytics**: Log level analysis, error module tracking, and performance insights  
🌐 **Multiple Interfaces**: Command-line, interactive menu, and web-based Gradio interface  
🔧 **Flexible AI Models**: OpenAI, Ollama (local), and HuggingFace Transformers support  
🚨 **Error Pattern Detection**: Automated identification of error trends and module issues

## Quick Start

1. **Install Dependencies**

   ```bash
   pip install -r requirements.txt
   ```

2. **Configure Your AI Model** (Choose one option)

   **Option A: Free Local AI (Ollama)**

   ```bash
   # Install Ollama from https://ollama.ai/
   ollama pull llama2
   ollama serve
   python setup_ollama.py  # Test setup
   ```

   **Option B: OpenAI API**

   ```bash
   set OPENAI_API_KEY=your_api_key_here
   ```

   **Option C: Use without AI (basic analysis only)**

   ```bash
   # No setup needed - basic data analysis works without AI
   ```

3. **Run the Application**

   **Main Interactive Analysis:**

   ```bash
   python upload.py
   # Choose mode 2 for enhanced AI features
   ```

   **Quick Demo:**

   ```bash
   python quick_ai_demo.py
   ```

   **Web Interface:**

   ```bash
   python app.py
   # Open browser to displayed URL
   ```

4. **Load Your OutSystems Data**
   - Update file path in `upload.py` or `config.py`
   - Default: `outsystems_sample_logs_6months.csv`

## Files Overview

### Main Scripts

- **`upload.py`** - Main data analysis script with interactive menu and AI integration
- **`app.py`** - Gradio web interface for browser-based analysis
- **`fixed_upload.py`** - Streamlined working version with automatic model fallbacks

### Setup & Testing Scripts

- **`setup_ollama.py`** - Ollama local AI setup helper
- **`setup_free_ai.py`** - Free AI models configuration
- **`test_smolagent.py`** - SmoLagent functionality testing
- **`quick_ai_demo.py`** - Quick demo without large downloads

### Configuration

- **`config.py`** - Model and data source configuration
- **`requirements.txt`** - Python dependencies
- **`ENHANCEMENT_GUIDE.md`** - Enhancement documentation

### Analysis Examples

- **`analyze.py`** - Programmatic analysis examples
- **`demo_enhanced.py`** - Enhanced features demonstration

## Model Configuration Options

### 1. Ollama (Recommended - Free & Local)

```python
from smolagents import CodeAgent, DuckDuckGoSearchTool, OllamaModel

model = OllamaModel(model_id="llama2", base_url="http://localhost:11434")
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)
```

### 2. OpenAI Models

```python
from smolagents import CodeAgent, DuckDuckGoSearchTool, OpenAIModel

model = OpenAIModel(model_id="gpt-3.5-turbo")
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)
```

### 3. HuggingFace Transformers

```python
from smolagents import CodeAgent, DuckDuckGoSearchTool, TransformersModel

model = TransformersModel(model_id="microsoft/DialoGPT-small")
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)
```

## Usage Examples

### 1. Interactive Analysis (Recommended)

```bash
python upload.py
# Choose mode 2 for enhanced AI features
# Navigate through interactive menu:
# 1. Load and explore data
# 2. Create visualizations
# 3. Analyze data quality
# 4. AI-powered analysis
```

### 2. Quick Demo

```bash
python quick_ai_demo.py  # Basic analysis without heavy downloads
```

### 3. Web Interface

```bash
python app.py           # Start Gradio web interface
```

### 4. Automated Setup Testing

```bash
python test_smolagent.py    # Test SmoLagent setup
python setup_free_ai.py    # Configure free AI models
```

## OutSystems Log Analysis Features

### 1. Data Overview Tab

- **Dataset Structure**: 5000+ log entries with 9 columns
- **Column Analysis**: Timestamp, LogLevel, Module, User, Message, RequestUrl, SessionId, ErrorId, StackTrace
- **Missing Value Detection**: Identifies incomplete error records (ErrorId/StackTrace gaps)
- **Memory Usage**: Optimized for large log files

### 2. Log-Specific Analytics

- **Error Rate Analysis**: Info vs Error log distribution
- **Module Error Tracking**: Top error-generating modules identification
- **Session Analysis**: User session patterns and error correlation
- **Timestamp Patterns**: Time-based error trends and peak periods

### 3. Visualizations

- **Error Distribution Plots**: Visual error patterns across modules
- **Correlation Analysis**: Relationships between log parameters
- **Time Series Analysis**: Error trends over 6-month period

### 4. AI-Powered OutSystems Insights

- **Error Pattern Recognition**: AI identifies recurring error types
- **Module Risk Assessment**: Predicts which modules need attention
- **Performance Optimization**: Suggests improvements based on log patterns
- **Anomaly Detection**: Unusual log patterns and potential issues

## Example AI Queries for OutSystems Logs

Ask SmoLagent questions like:

**Error Analysis:**

- "What are the most common error types in these OutSystems logs?"
- "Which modules generate the most errors?"
- "Find patterns in error messages"
- "Identify critical errors that need immediate attention"

**Performance Insights:**

- "Analyze user session patterns for performance issues"
- "Find modules with the highest error rates"
- "Detect unusual activity patterns"
- "Suggest modules that need code review"

**Trend Analysis:**

- "Show error trends over the 6-month period"
- "Find seasonal patterns in system usage"
- "Identify peak error periods"
- "Correlate errors with specific time periods"

**Data Quality:**

- "Check for incomplete error records"
- "Find missing StackTrace patterns"
- "Validate log data completeness"
- "Suggest data quality improvements"

## Data Requirements

- **Format**: CSV format with OutSystems log structure
- **Expected Columns**: Timestamp, LogLevel, Module, User, Message, RequestUrl, SessionId, ErrorId, StackTrace
- **Default File**: `outsystems_sample_logs_6months.csv` (6 months of sample data)
- **File Location**: Update path in `upload.py` line 11 or `config.py`
- **Size**: Optimized for datasets with 5000+ log entries
- **Encoding**: UTF-8 encoding recommended

## Troubleshooting

### Common Issues:

1. **"Module not found: smolagents"**

   ```bash
   pip install 'smolagents[transformers]'
   pip install duckduckgo-search>=3.8.0
   ```

2. **"CodeAgent requires model parameter"**

   - Run `python fixed_upload.py` for automatic model fallbacks
   - Or configure Ollama: `python setup_ollama.py`

3. **Model Download Taking Too Long**

   ```bash
   python quick_ai_demo.py  # Use without heavy downloads
   ```

4. **OutSystems CSV File Not Found**

   - Update file path in `upload.py` line 11
   - Ensure CSV file exists at specified location
   - Check file permissions

5. **Ollama Connection Error**
   ```bash
   ollama serve  # Start Ollama server
   ollama pull llama2  # Download model
   ```

### Getting Help:

- **Quick Test**: Run `python test_smolagent.py`
- **Setup Issues**: Run `python setup_free_ai.py`
- **Basic Analysis**: Use Mode 1 in `upload.py` (works without AI)
- **Check Logs**: Console output shows detailed error messages

## Advanced Usage

### Custom Analysis Functions

You can extend the application by adding custom analysis functions:

```python
def custom_analysis(df):
    # Your custom analysis logic here
    return results
```

### Adding New Visualizations

Add new plotting functions to create additional visualizations:

```python
def create_custom_plot(df):
    # Your plotting logic here
    return plot_image
```

## Dependencies

**Core AI Framework:**

- `smolagents>=0.3.0` - AI agent framework with CodeAgent
- `duckduckgo-search>=3.8.0` - Web search capabilities

**Data Analysis Stack:**

- `pandas>=2.0.0` - Data manipulation and analysis
- `numpy>=1.24.0` - Numerical computing
- `scikit-learn>=1.3.0` - Machine learning tools

**Visualization:**

- `matplotlib>=3.7.0` - Basic plotting
- `seaborn>=0.12.0` - Statistical visualizations
- `plotly>=5.15.0` - Interactive visualizations

**AI Models:**

- `transformers>=4.30.0` - HuggingFace transformers
- `torch>=2.0.0` - PyTorch for model inference
- `huggingface_hub>=0.16.0` - Model hub access

**Web Interface:**

- `gradio>=5.37.0` - Web UI framework
- `Pillow>=10.0.0` - Image processing

**HTTP & Utilities:**

- `requests>=2.31.0` - HTTP requests

## Project Structure

```
Singtel_Use_Case1/
├── 📁 Main Scripts
│   ├── upload.py              # Primary interactive analysis tool
│   ├── app.py                 # Gradio web interface
│   └── fixed_upload.py        # Streamlined working version
│
├── 📁 Setup & Testing
│   ├── setup_ollama.py        # Ollama configuration helper
│   ├── setup_free_ai.py       # Free AI models setup
│   ├── test_smolagent.py      # SmoLagent testing
│   └── quick_ai_demo.py       # Quick demo script
│
├── 📁 Configuration
│   ├── config.py              # Model and data configuration
│   ├── requirements.txt       # Python dependencies
│   └── ENHANCEMENT_GUIDE.md   # Development guide
│
├── 📁 Analysis Examples
│   ├── analyze.py             # Programmatic analysis
│   └── demo_enhanced.py       # Feature demonstrations
│
└── 📁 Data (Expected)
    └── outsystems_sample_logs_6months.csv
```

## License

This project is developed for Singtel Use Case 1 analysis and is available under the MIT License.

---

## Quick Commands Reference

```bash
# 🚀 Quick Start
python upload.py              # Main interactive tool
python quick_ai_demo.py       # Fast demo without downloads
python app.py                 # Web interface

# 🔧 Setup & Testing
python setup_ollama.py        # Configure local AI
python test_smolagent.py      # Test AI setup
python setup_free_ai.py       # Free models setup

# 📊 Analysis Options
# Mode 1: Basic data exploration (no AI required)
# Mode 2: Full AI-powered analysis with interactive menu
```

**For Singtel Use Case 1**: This tool specifically analyzes OutSystems log data to identify error patterns, module performance, and system optimization opportunities using AI-powered insights.