README.md

metadata

title: AI Data Analysis with SmoLagent
emoji: 🤖
colorFrom: purple
colorTo: green
sdk: gradio
sdk_version: 5.37.0
app_file: app.py
pinned: false

Singtel Use Case 1: OutSystems Log Analysis with SmoLagent AI

An intelligent data analysis application specifically designed for analyzing OutSystems application logs using SmoLagent AI framework. This project focuses on extracting insights from 6 months of OutSystems log data to identify patterns, errors, and optimization opportunities.

Features

🤖 AI-Powered Log Analysis: Uses SmoLagent CodeAgent with DuckDuckGo search for intelligent log analysis
📊 Interactive Visualizations: Correlation heatmaps, distribution plots, and error pattern analysis
📈 OutSystems-Specific Analytics: Log level analysis, error module tracking, and performance insights
🌐 Multiple Interfaces: Command-line, interactive menu, and web-based Gradio interface
🔧 Flexible AI Models: OpenAI, Ollama (local), and HuggingFace Transformers support
🚨 Error Pattern Detection: Automated identification of error trends and module issues

Quick Start

1. Install Dependencies

   pip install -r requirements.txt
   

2. Configure Your AI Model (Choose one option)

   Option A: Free Local AI (Ollama)

   # Install Ollama from https://ollama.ai/
   ollama pull llama2
   ollama serve
   python setup_ollama.py  # Test setup
   

   Option B: OpenAI API

   set OPENAI_API_KEY=your_api_key_here
   

   Option C: Use without AI (basic analysis only)

   # No setup needed - basic data analysis works without AI
   

3. Run the Application

   Main Interactive Analysis:

   python upload.py
   # Choose mode 2 for enhanced AI features
   

   Quick Demo:

   python quick_ai_demo.py
   

   Web Interface:

   python app.py
   # Open browser to displayed URL
   

4. Load Your OutSystems Data

   • Update file path in upload.py or config.py
   • Default: outsystems_sample_logs_6months.csv

Files Overview

Main Scripts

• upload.py - Main data analysis script with interactive menu and AI integration
• app.py - Gradio web interface for browser-based analysis
• fixed_upload.py - Streamlined working version with automatic model fallbacks

Setup & Testing Scripts

• setup_ollama.py - Ollama local AI setup helper
• setup_free_ai.py - Free AI models configuration
• test_smolagent.py - SmoLagent functionality testing
• quick_ai_demo.py - Quick demo without large downloads

Configuration

• config.py - Model and data source configuration
• requirements.txt - Python dependencies
• ENHANCEMENT_GUIDE.md - Enhancement documentation

Analysis Examples

• analyze.py - Programmatic analysis examples
• demo_enhanced.py - Enhanced features demonstration

Model Configuration Options

1. Ollama (Recommended - Free & Local)

from smolagents import CodeAgent, DuckDuckGoSearchTool, OllamaModel

model = OllamaModel(model_id="llama2", base_url="http://localhost:11434")
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)

2. OpenAI Models

from smolagents import CodeAgent, DuckDuckGoSearchTool, OpenAIModel

model = OpenAIModel(model_id="gpt-3.5-turbo")
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)

3. HuggingFace Transformers

from smolagents import CodeAgent, DuckDuckGoSearchTool, TransformersModel

model = TransformersModel(model_id="microsoft/DialoGPT-small")
agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)

Usage Examples

1. Interactive Analysis (Recommended)

python upload.py
# Choose mode 2 for enhanced AI features
# Navigate through interactive menu:
# 1. Load and explore data
# 2. Create visualizations
# 3. Analyze data quality
# 4. AI-powered analysis

2. Quick Demo

python quick_ai_demo.py  # Basic analysis without heavy downloads

3. Web Interface

python app.py           # Start Gradio web interface

4. Automated Setup Testing

python test_smolagent.py    # Test SmoLagent setup
python setup_free_ai.py    # Configure free AI models

OutSystems Log Analysis Features

1. Data Overview Tab

• Dataset Structure: 5000+ log entries with 9 columns
• Column Analysis: Timestamp, LogLevel, Module, User, Message, RequestUrl, SessionId, ErrorId, StackTrace
• Missing Value Detection: Identifies incomplete error records (ErrorId/StackTrace gaps)
• Memory Usage: Optimized for large log files

2. Log-Specific Analytics

• Error Rate Analysis: Info vs Error log distribution
• Module Error Tracking: Top error-generating modules identification
• Session Analysis: User session patterns and error correlation
• Timestamp Patterns: Time-based error trends and peak periods

3. Visualizations

• Error Distribution Plots: Visual error patterns across modules
• Correlation Analysis: Relationships between log parameters
• Time Series Analysis: Error trends over 6-month period

4. AI-Powered OutSystems Insights

• Error Pattern Recognition: AI identifies recurring error types
• Module Risk Assessment: Predicts which modules need attention
• Performance Optimization: Suggests improvements based on log patterns
• Anomaly Detection: Unusual log patterns and potential issues

Example AI Queries for OutSystems Logs

Ask SmoLagent questions like:

Error Analysis:

• "What are the most common error types in these OutSystems logs?"
• "Which modules generate the most errors?"
• "Find patterns in error messages"
• "Identify critical errors that need immediate attention"

Performance Insights:

• "Analyze user session patterns for performance issues"
• "Find modules with the highest error rates"
• "Detect unusual activity patterns"
• "Suggest modules that need code review"

Trend Analysis:

• "Show error trends over the 6-month period"
• "Find seasonal patterns in system usage"
• "Identify peak error periods"
• "Correlate errors with specific time periods"

Data Quality:

• "Check for incomplete error records"
• "Find missing StackTrace patterns"
• "Validate log data completeness"
• "Suggest data quality improvements"

Data Requirements

• Format: CSV format with OutSystems log structure
• Expected Columns: Timestamp, LogLevel, Module, User, Message, RequestUrl, SessionId, ErrorId, StackTrace
• Default File: outsystems_sample_logs_6months.csv (6 months of sample data)
• File Location: Update path in upload.py line 11 or config.py
• Size: Optimized for datasets with 5000+ log entries
• Encoding: UTF-8 encoding recommended

Troubleshooting

Common Issues:

1. "Module not found: smolagents"

   pip install 'smolagents[transformers]'
   pip install duckduckgo-search>=3.8.0
   

2. "CodeAgent requires model parameter"

   • Run python fixed_upload.py for automatic model fallbacks
   • Or configure Ollama: python setup_ollama.py

3. Model Download Taking Too Long

   python quick_ai_demo.py  # Use without heavy downloads
   

4. OutSystems CSV File Not Found

   • Update file path in upload.py line 11
   • Ensure CSV file exists at specified location
   • Check file permissions

5. Ollama Connection Error

   ollama serve  # Start Ollama server
   ollama pull llama2  # Download model
   

Getting Help:

• Quick Test: Run python test_smolagent.py
• Setup Issues: Run python setup_free_ai.py
• Basic Analysis: Use Mode 1 in upload.py (works without AI)
• Check Logs: Console output shows detailed error messages

Advanced Usage

Custom Analysis Functions

You can extend the application by adding custom analysis functions:

def custom_analysis(df):
    # Your custom analysis logic here
    return results

Adding New Visualizations

Add new plotting functions to create additional visualizations:

def create_custom_plot(df):
    # Your plotting logic here
    return plot_image

Dependencies

Core AI Framework:

• smolagents>=0.3.0 - AI agent framework with CodeAgent
• duckduckgo-search>=3.8.0 - Web search capabilities

Data Analysis Stack:

• pandas>=2.0.0 - Data manipulation and analysis
• numpy>=1.24.0 - Numerical computing
• scikit-learn>=1.3.0 - Machine learning tools

Visualization:

• matplotlib>=3.7.0 - Basic plotting
• seaborn>=0.12.0 - Statistical visualizations
• plotly>=5.15.0 - Interactive visualizations

AI Models:

• transformers>=4.30.0 - HuggingFace transformers
• torch>=2.0.0 - PyTorch for model inference
• huggingface_hub>=0.16.0 - Model hub access

Web Interface:

• gradio>=5.37.0 - Web UI framework
• Pillow>=10.0.0 - Image processing

HTTP & Utilities:

• requests>=2.31.0 - HTTP requests

Project Structure

Singtel_Use_Case1/
├── 📁 Main Scripts
│   ├── upload.py              # Primary interactive analysis tool
│   ├── app.py                 # Gradio web interface
│   └── fixed_upload.py        # Streamlined working version
│
├── 📁 Setup & Testing
│   ├── setup_ollama.py        # Ollama configuration helper
│   ├── setup_free_ai.py       # Free AI models setup
│   ├── test_smolagent.py      # SmoLagent testing
│   └── quick_ai_demo.py       # Quick demo script
│
├── 📁 Configuration
│   ├── config.py              # Model and data configuration
│   ├── requirements.txt       # Python dependencies
│   └── ENHANCEMENT_GUIDE.md   # Development guide
│
├── 📁 Analysis Examples
│   ├── analyze.py             # Programmatic analysis
│   └── demo_enhanced.py       # Feature demonstrations
│
└── 📁 Data (Expected)
    └── outsystems_sample_logs_6months.csv

License

This project is developed for Singtel Use Case 1 analysis and is available under the MIT License.

---

Quick Commands Reference

# 🚀 Quick Start
python upload.py              # Main interactive tool
python quick_ai_demo.py       # Fast demo without downloads
python app.py                 # Web interface

# 🔧 Setup & Testing
python setup_ollama.py        # Configure local AI
python test_smolagent.py      # Test AI setup
python setup_free_ai.py       # Free models setup

# 📊 Analysis Options
# Mode 1: Basic data exploration (no AI required)
# Mode 2: Full AI-powered analysis with interactive menu

For Singtel Use Case 1: This tool specifically analyzes OutSystems log data to identify error patterns, module performance, and system optimization opportunities using AI-powered insights.