Add FinRL integration with comprehensive RL trading agent

- Add FinRL agent with support for PPO, A2C, DDPG, and TD3 algorithms
- Create custom trading environment compatible with Gymnasium
- Implement technical indicators integration (RSI, Bollinger Bands, MACD)
- Add comprehensive configuration system for FinRL parameters
- Create demo script with training, evaluation, and visualization
- Add comprehensive test suite for FinRL functionality
- Update requirements.txt with FinRL dependencies
- Update README with detailed FinRL documentation
- Create necessary directories for models, logs, and plots

README.md

@@ -1,6 +1,6 @@
 # Algorithmic Trading System
 
-A comprehensive algorithmic trading system with synthetic data generation, comprehensive logging, and extensive testing capabilities.
+A comprehensive algorithmic trading system with synthetic data generation, comprehensive logging, extensive testing capabilities, and FinRL reinforcement learning integration.
 
 ## Features
 
@@ -10,6 +10,15 @@ A comprehensive algorithmic trading system with synthetic data generation, compr
 - **Risk Management**: Position sizing and drawdown limits
 - **Order Execution**: Simulated broker integration with realistic execution delays
 
+### FinRL Reinforcement Learning
+- **Multiple RL Algorithms**: Support for PPO, A2C, DDPG, and TD3
+- **Custom Trading Environment**: Gymnasium-compatible environment for RL training
+- **Technical Indicators Integration**: Automatic calculation and inclusion of technical indicators
+- **Portfolio Management**: Realistic portfolio simulation with transaction costs
+- **Model Persistence**: Save and load trained models for inference
+- **TensorBoard Integration**: Training progress visualization and monitoring
+- **Comprehensive Evaluation**: Performance metrics including Sharpe ratio and total returns
+
 ### Synthetic Data Generation
 - **Realistic Market Data**: Generate OHLCV data using geometric Brownian motion
 - **Multiple Frequencies**: Support for 1min, 5min, 1H, and 1D data
@@ -226,6 +235,129 @@ logger.warning("High volatility detected")
 logger.error("Order execution failed", exc_info=True)
 ```
 
+## FinRL Integration
+
+### Overview
+The system now includes FinRL (Financial Reinforcement Learning) integration, providing state-of-the-art reinforcement learning capabilities for algorithmic trading. The FinRL agent can learn optimal trading strategies through interaction with a simulated market environment.
+
+### Supported Algorithms
+- **PPO (Proximal Policy Optimization)**: Stable policy gradient method
+- **A2C (Advantage Actor-Critic)**: Actor-critic method with advantage estimation
+- **DDPG (Deep Deterministic Policy Gradient)**: Continuous action space algorithm
+- **TD3 (Twin Delayed DDPG)**: Improved version of DDPG with twin critics
+
+### Trading Environment
+The custom trading environment provides:
+- **Action Space**: Discrete actions (0=Buy, 1=Hold, 2=Sell)
+- **Observation Space**: OHLCV data + technical indicators + portfolio state
+- **Reward Function**: Portfolio return-based rewards
+- **Transaction Costs**: Realistic trading fees and slippage
+- **Position Limits**: Maximum position constraints
+
+### Usage Examples
+
+#### Basic FinRL Training
+```python
+from agentic_ai_system.finrl_agent import FinRLAgent, FinRLConfig
+import pandas as pd
+
+# Create configuration
+config = FinRLConfig(
+    algorithm="PPO",
+    learning_rate=0.0003,
+    batch_size=64,
+    total_timesteps=100000
+)
+
+# Initialize agent
+agent = FinRLAgent(config)
+
+# Train the agent
+training_result = agent.train(
+    data=market_data,
+    total_timesteps=100000,
+    eval_freq=10000
+)
+
+# Generate predictions
+predictions = agent.predict(test_data)
+
+# Evaluate performance
+evaluation = agent.evaluate(test_data)
+print(f"Total Return: {evaluation['total_return']:.2%}")
+```
+
+#### Using Configuration File
+```python
+from agentic_ai_system.finrl_agent import create_finrl_agent_from_config
+
+# Create agent from config file
+agent = create_finrl_agent_from_config('config.yaml')
+
+# Train and evaluate
+agent.train(market_data)
+results = agent.evaluate(test_data)
+```
+
+#### Running FinRL Demo
+```bash
+# Run the complete FinRL demo
+python finrl_demo.py
+
+# This will:
+# 1. Generate synthetic training and test data
+# 2. Train a FinRL agent
+# 3. Evaluate performance
+# 4. Generate trading predictions
+# 5. Create visualization plots
+```
+
+### Configuration
+FinRL settings can be configured in `config.yaml`:
+
+```yaml
+finrl:
+  algorithm: 'PPO'  # PPO, A2C, DDPG, TD3
+  learning_rate: 0.0003
+  batch_size: 64
+  buffer_size: 1000000
+  gamma: 0.99
+  tensorboard_log: 'logs/finrl_tensorboard'
+  training:
+    total_timesteps: 100000
+    eval_freq: 10000
+    save_best_model: true
+    model_save_path: 'models/finrl_best/'
+  inference:
+    use_trained_model: false
+    model_path: 'models/finrl_best/best_model'
+```
+
+### Model Management
+```python
+# Save trained model
+agent.save_model('models/my_finrl_model')
+
+# Load pre-trained model
+agent.load_model('models/my_finrl_model')
+
+# Continue training
+agent.train(more_data, total_timesteps=50000)
+```
+
+### Performance Monitoring
+- **TensorBoard Integration**: Monitor training progress
+- **Evaluation Metrics**: Total return, Sharpe ratio, portfolio value
+- **Trading Statistics**: Buy/sell signal analysis
+- **Visualization**: Price charts with trading signals
+
+### Advanced Features
+- **Multi-timeframe Support**: Train on different data frequencies
+- **Feature Engineering**: Automatic technical indicator calculation
+- **Risk Management**: Built-in position and drawdown limits
+- **Backtesting**: Comprehensive backtesting capabilities
+- **Hyperparameter Tuning**: Easy configuration for different algorithms
+
 ## Testing
 
 ### Test Structure

agentic_ai_system/finrl_agent.py

@@ -0,0 +1,447 @@
+"""
+FinRL Agent for Algorithmic Trading
+
+This module provides a FinRL-based reinforcement learning agent that can be integrated
+with the existing algorithmic trading system. It supports various RL algorithms
+including PPO, A2C, DDPG, and TD3.
+"""
+
+import numpy as np
+import pandas as pd
+import gymnasium as gym
+from gymnasium import spaces
+from stable_baselines3 import PPO, A2C, DDPG, TD3
+from stable_baselines3.common.vec_env import DummyVecEnv
+from stable_baselines3.common.callbacks import EvalCallback
+import torch
+import logging
+from typing import Dict, List, Tuple, Optional, Any
+from dataclasses import dataclass
+import yaml
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class FinRLConfig:
+    """Configuration for FinRL agent"""
+    algorithm: str = "PPO"  # PPO, A2C, DDPG, TD3
+    learning_rate: float = 0.0003
+    batch_size: int = 64
+    buffer_size: int = 1000000
+    learning_starts: int = 100
+    gamma: float = 0.99
+    tau: float = 0.005
+    train_freq: int = 1
+    gradient_steps: int = 1
+    target_update_interval: int = 1
+    exploration_fraction: float = 0.1
+    exploration_initial_eps: float = 1.0
+    exploration_final_eps: float = 0.05
+    max_grad_norm: float = 10.0
+    verbose: int = 1
+    tensorboard_log: str = "logs/finrl_tensorboard"
+
+
+class TradingEnvironment(gym.Env):
+    """
+    Custom trading environment for FinRL
+    
+    This environment simulates a trading scenario where the agent can:
+    - Buy, sell, or hold positions
+    - Use technical indicators for decision making
+    - Manage portfolio value and risk
+    """
+    
+    def __init__(self, data: pd.DataFrame, initial_balance: float = 100000, 
+                 transaction_fee: float = 0.001, max_position: int = 100):
+        super().__init__()
+        
+        self.data = data
+        self.initial_balance = initial_balance
+        self.transaction_fee = transaction_fee
+        self.max_position = max_position
+        
+        # Reset state
+        self.reset()
+        
+        # Define action space: [-1, 0, 1] for sell, hold, buy
+        self.action_space = spaces.Discrete(3)
+        
+        # Define observation space
+        # Features: OHLCV + technical indicators + portfolio state
+        n_features = len(self._get_features(self.data.iloc[0]))
+        self.observation_space = spaces.Box(
+            low=-np.inf, high=np.inf, shape=(n_features,), dtype=np.float32
+        )
+    
+    def _get_features(self, row: pd.Series) -> np.ndarray:
+        """Extract features from market data row"""
+        features = []
+        
+        # Price features
+        features.extend([
+            row['open'], row['high'], row['low'], row['close'], row['volume']
+        ])
+        
+        # Technical indicators (if available)
+        for indicator in ['sma_20', 'sma_50', 'rsi', 'bb_upper', 'bb_lower', 'macd']:
+            if indicator in row.index:
+                features.append(row[indicator])
+            else:
+                features.append(0.0)
+        
+        # Portfolio state
+        features.extend([
+            self.balance,
+            self.position,
+            self.portfolio_value,
+            self.total_return
+        ])
+        
+        return np.array(features, dtype=np.float32)
+    
+    def _calculate_portfolio_value(self) -> float:
+        """Calculate current portfolio value"""
+        current_price = self.data.iloc[self.current_step]['close']
+        return self.balance + (self.position * current_price)
+    
+    def _calculate_reward(self) -> float:
+        """Calculate reward based on portfolio performance"""
+        current_value = self._calculate_portfolio_value()
+        previous_value = self.previous_portfolio_value
+        
+        # Calculate return
+        if previous_value > 0:
+            return (current_value - previous_value) / previous_value
+        else:
+            return 0.0
+    
+    def step(self, action: int) -> Tuple[np.ndarray, float, bool, bool, Dict]:
+        """Execute one step in the environment"""
+        
+        # Get current market data
+        current_data = self.data.iloc[self.current_step]
+        current_price = current_data['close']
+        
+        # Execute action
+        if action == 0:  # Sell
+            if self.position > 0:
+                shares_to_sell = min(self.position, self.max_position)
+                sell_value = shares_to_sell * current_price * (1 - self.transaction_fee)
+                self.balance += sell_value
+                self.position -= shares_to_sell
+        elif action == 2:  # Buy
+            if self.balance > 0:
+                max_shares = min(
+                    int(self.balance / current_price),
+                    self.max_position - self.position
+                )
+                if max_shares > 0:
+                    buy_value = max_shares * current_price * (1 + self.transaction_fee)
+                    self.balance -= buy_value
+                    self.position += max_shares
+        
+        # Update portfolio value
+        self.previous_portfolio_value = self.portfolio_value
+        self.portfolio_value = self._calculate_portfolio_value()
+        self.total_return = (self.portfolio_value - self.initial_balance) / self.initial_balance
+        
+        # Calculate reward
+        reward = self._calculate_reward()
+        
+        # Move to next step
+        self.current_step += 1
+        
+        # Check if episode is done
+        done = self.current_step >= len(self.data) - 1
+        
+        # Get observation
+        if not done:
+            observation = self._get_features(self.data.iloc[self.current_step])
+        else:
+            # Use last available data for final observation
+            observation = self._get_features(self.data.iloc[-1])
+        
+        info = {
+            'balance': self.balance,
+            'position': self.position,
+            'portfolio_value': self.portfolio_value,
+            'total_return': self.total_return,
+            'current_price': current_price
+        }
+        
+        return observation, reward, done, False, info
+    
+    def reset(self, seed: Optional[int] = None) -> Tuple[np.ndarray, Dict]:
+        """Reset the environment"""
+        super().reset(seed=seed)
+        
+        self.current_step = 0
+        self.balance = self.initial_balance
+        self.position = 0
+        self.portfolio_value = self.initial_balance
+        self.previous_portfolio_value = self.initial_balance
+        self.total_return = 0.0
+        
+        observation = self._get_features(self.data.iloc[self.current_step])
+        info = {
+            'balance': self.balance,
+            'position': self.position,
+            'portfolio_value': self.portfolio_value,
+            'total_return': self.total_return
+        }
+        
+        return observation, info
+
+
+class FinRLAgent:
+    """
+    FinRL-based reinforcement learning agent for algorithmic trading
+    """
+    
+    def __init__(self, config: FinRLConfig):
+        self.config = config
+        self.model = None
+        self.env = None
+        self.eval_env = None
+        self.callback = None
+        
+        logger.info(f"Initializing FinRL agent with algorithm: {config.algorithm}")
+    
+    def create_environment(self, data: pd.DataFrame, initial_balance: float = 100000) -> TradingEnvironment:
+        """Create trading environment from market data"""
+        return TradingEnvironment(
+            data=data,
+            initial_balance=initial_balance,
+            transaction_fee=0.001,
+            max_position=100
+        )
+    
+    def prepare_data(self, data: pd.DataFrame) -> pd.DataFrame:
+        """Prepare data with technical indicators for FinRL"""
+        df = data.copy()
+        
+        # Add technical indicators if not present
+        if 'sma_20' not in df.columns:
+            df['sma_20'] = df['close'].rolling(window=20).mean()
+        if 'sma_50' not in df.columns:
+            df['sma_50'] = df['close'].rolling(window=50).mean()
+        if 'rsi' not in df.columns:
+            df['rsi'] = self._calculate_rsi(df['close'])
+        if 'bb_upper' not in df.columns or 'bb_lower' not in df.columns:
+            bb_upper, bb_lower = self._calculate_bollinger_bands(df['close'])
+            df['bb_upper'] = bb_upper
+            df['bb_lower'] = bb_lower
+        if 'macd' not in df.columns:
+            df['macd'] = self._calculate_macd(df['close'])
+        
+        # Fill NaN values
+        df = df.fillna(method='bfill').fillna(0)
+        
+        return df
+    
+    def _calculate_rsi(self, prices: pd.Series, period: int = 14) -> pd.Series:
+        """Calculate RSI indicator"""
+        delta = prices.diff()
+        gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
+        loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
+        rs = gain / loss
+        rsi = 100 - (100 / (1 + rs))
+        return rsi
+    
+    def _calculate_bollinger_bands(self, prices: pd.Series, period: int = 20, std_dev: int = 2) -> Tuple[pd.Series, pd.Series]:
+        """Calculate Bollinger Bands"""
+        sma = prices.rolling(window=period).mean()
+        std = prices.rolling(window=period).std()
+        upper_band = sma + (std * std_dev)
+        lower_band = sma - (std * std_dev)
+        return upper_band, lower_band
+    
+    def _calculate_macd(self, prices: pd.Series, fast: int = 12, slow: int = 26, signal: int = 9) -> pd.Series:
+        """Calculate MACD indicator"""
+        ema_fast = prices.ewm(span=fast).mean()
+        ema_slow = prices.ewm(span=slow).mean()
+        macd_line = ema_fast - ema_slow
+        return macd_line
+    
+    def train(self, data: pd.DataFrame, total_timesteps: int = 100000, 
+              eval_freq: int = 10000, eval_data: Optional[pd.DataFrame] = None) -> Dict[str, Any]:
+        """Train the FinRL agent"""
+        
+        logger.info("Starting FinRL agent training")
+        
+        # Prepare data
+        train_data = self.prepare_data(data)
+        
+        # Create training environment
+        self.env = DummyVecEnv([lambda: self.create_environment(train_data)])
+        
+        # Create evaluation environment if provided
+        if eval_data is not None:
+            eval_data = self.prepare_data(eval_data)
+            self.eval_env = DummyVecEnv([lambda: self.create_environment(eval_data)])
+            self.callback = EvalCallback(
+                self.eval_env,
+                best_model_save_path="models/finrl_best/",
+                log_path="logs/finrl_eval/",
+                eval_freq=eval_freq,
+                deterministic=True,
+                render=False
+            )
+        
+        # Initialize model based on algorithm
+        if self.config.algorithm == "PPO":
+            self.model = PPO(
+                "MlpPolicy",
+                self.env,
+                learning_rate=self.config.learning_rate,
+                batch_size=self.config.batch_size,
+                gamma=self.config.gamma,
+                verbose=self.config.verbose,
+                tensorboard_log=self.config.tensorboard_log
+            )
+        elif self.config.algorithm == "A2C":
+            self.model = A2C(
+                "MlpPolicy",
+                self.env,
+                learning_rate=self.config.learning_rate,
+                gamma=self.config.gamma,
+                verbose=self.config.verbose,
+                tensorboard_log=self.config.tensorboard_log
+            )
+        elif self.config.algorithm == "DDPG":
+            self.model = DDPG(
+                "MlpPolicy",
+                self.env,
+                learning_rate=self.config.learning_rate,
+                buffer_size=self.config.buffer_size,
+                learning_starts=self.config.learning_starts,
+                gamma=self.config.gamma,
+                tau=self.config.tau,
+                train_freq=self.config.train_freq,
+                gradient_steps=self.config.gradient_steps,
+                verbose=self.config.verbose,
+                tensorboard_log=self.config.tensorboard_log
+            )
+        elif self.config.algorithm == "TD3":
+            self.model = TD3(
+                "MlpPolicy",
+                self.env,
+                learning_rate=self.config.learning_rate,
+                buffer_size=self.config.buffer_size,
+                learning_starts=self.config.learning_starts,
+                gamma=self.config.gamma,
+                tau=self.config.tau,
+                train_freq=self.config.train_freq,
+                gradient_steps=self.config.gradient_steps,
+                target_update_interval=self.config.target_update_interval,
+                verbose=self.config.verbose,
+                tensorboard_log=self.config.tensorboard_log
+            )
+        else:
+            raise ValueError(f"Unsupported algorithm: {self.config.algorithm}")
+        
+        # Train the model
+        callbacks = [self.callback] if self.callback else None
+        self.model.learn(
+            total_timesteps=total_timesteps,
+            callback=callbacks
+        )
+        
+        logger.info("FinRL agent training completed")
+        
+        return {
+            'algorithm': self.config.algorithm,
+            'total_timesteps': total_timesteps,
+            'model_path': f"models/finrl_{self.config.algorithm.lower()}"
+        }
+    
+    def predict(self, data: pd.DataFrame) -> List[int]:
+        """Generate trading predictions using the trained model"""
+        if self.model is None:
+            raise ValueError("Model not trained. Call train() first.")
+        
+        # Prepare data
+        test_data = self.prepare_data(data)
+        
+        # Create test environment
+        test_env = self.create_environment(test_data)
+        
+        predictions = []
+        obs, _ = test_env.reset()
+        
+        done = False
+        while not done:
+            action, _ = self.model.predict(obs, deterministic=True)
+            predictions.append(action)
+            obs, _, done, _, _ = test_env.step(action)
+        
+        return predictions
+    
+    def evaluate(self, data: pd.DataFrame) -> Dict[str, float]:
+        """Evaluate the trained model on test data"""
+        if self.model is None:
+            raise ValueError("Model not trained. Call train() first.")
+        
+        # Prepare data
+        test_data = self.prepare_data(data)
+        
+        # Create test environment
+        test_env = self.create_environment(test_data)
+        
+        obs, _ = test_env.reset()
+        done = False
+        total_reward = 0
+        steps = 0
+        
+        while not done:
+            action, _ = self.model.predict(obs, deterministic=True)
+            obs, reward, done, _, info = test_env.step(action)
+            total_reward += reward
+            steps += 1
+        
+        # Calculate metrics
+        final_portfolio_value = info['portfolio_value']
+        initial_balance = test_env.initial_balance
+        total_return = (final_portfolio_value - initial_balance) / initial_balance
+        
+        return {
+            'total_reward': total_reward,
+            'total_return': total_return,
+            'final_portfolio_value': final_portfolio_value,
+            'steps': steps,
+            'sharpe_ratio': total_reward / steps if steps > 0 else 0
+        }
+    
+    def save_model(self, path: str):
+        """Save the trained model"""
+        if self.model is None:
+            raise ValueError("No model to save. Train the model first.")
+        
+        self.model.save(path)
+        logger.info(f"Model saved to {path}")
+    
+    def load_model(self, path: str):
+        """Load a trained model"""
+        if self.config.algorithm == "PPO":
+            self.model = PPO.load(path)
+        elif self.config.algorithm == "A2C":
+            self.model = A2C.load(path)
+        elif self.config.algorithm == "DDPG":
+            self.model = DDPG.load(path)
+        elif self.config.algorithm == "TD3":
+            self.model = TD3.load(path)
+        else:
+            raise ValueError(f"Unsupported algorithm: {self.config.algorithm}")
+        
+        logger.info(f"Model loaded from {path}")
+
+
+def create_finrl_agent_from_config(config_path: str) -> FinRLAgent:
+    """Create FinRL agent from configuration file"""
+    with open(config_path, 'r') as file:
+        config_data = yaml.safe_load(file)
+    
+    finrl_config = FinRLConfig(**config_data.get('finrl', {}))
+    return FinRLAgent(finrl_config) 

config.yaml

@@ -33,3 +33,30 @@ logging:
   enable_file: true
   max_file_size_mb: 10
   backup_count: 5
+
+# FinRL configuration
+finrl:
+  algorithm: 'PPO'  # PPO, A2C, DDPG, TD3
+  learning_rate: 0.0003
+  batch_size: 64
+  buffer_size: 1000000
+  learning_starts: 100
+  gamma: 0.99
+  tau: 0.005
+  train_freq: 1
+  gradient_steps: 1
+  target_update_interval: 1
+  exploration_fraction: 0.1
+  exploration_initial_eps: 1.0
+  exploration_final_eps: 0.05
+  max_grad_norm: 10.0
+  verbose: 1
+  tensorboard_log: 'logs/finrl_tensorboard'
+  training:
+    total_timesteps: 100000
+    eval_freq: 10000
+    save_best_model: true
+    model_save_path: 'models/finrl_best/'
+  inference:
+    use_trained_model: false
+    model_path: 'models/finrl_best/best_model'

finrl_demo.py

@@ -0,0 +1,294 @@
+#!/usr/bin/env python3
+"""
+FinRL Demo Script
+
+This script demonstrates the integration of FinRL with the algorithmic trading system.
+It shows how to train a reinforcement learning agent and use it for trading decisions.
+"""
+
+import os
+import sys
+import yaml
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+from datetime import datetime, timedelta
+import logging
+
+# Add the project root to the path
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from agentic_ai_system.finrl_agent import FinRLAgent, FinRLConfig, create_finrl_agent_from_config
+from agentic_ai_system.synthetic_data_generator import SyntheticDataGenerator
+from agentic_ai_system.logger_config import setup_logging
+
+# Setup logging
+setup_logging()
+logger = logging.getLogger(__name__)
+
+
+def load_config(config_path: str = 'config.yaml') -> dict:
+    """Load configuration from YAML file"""
+    with open(config_path, 'r') as file:
+        return yaml.safe_load(file)
+
+
+def generate_training_data(config: dict) -> pd.DataFrame:
+    """Generate synthetic data for training"""
+    logger.info("Generating synthetic training data")
+    
+    generator = SyntheticDataGenerator(config)
+    
+    # Generate training data (longer period)
+    train_data = generator.generate_ohlcv_data(
+        symbol='AAPL',
+        start_date='2023-01-01',
+        end_date='2023-12-31',
+        frequency='1H'
+    )
+    
+    # Add technical indicators
+    train_data['sma_20'] = train_data['close'].rolling(window=20).mean()
+    train_data['sma_50'] = train_data['close'].rolling(window=50).mean()
+    train_data['rsi'] = calculate_rsi(train_data['close'])
+    bb_upper, bb_lower = calculate_bollinger_bands(train_data['close'])
+    train_data['bb_upper'] = bb_upper
+    train_data['bb_lower'] = bb_lower
+    train_data['macd'] = calculate_macd(train_data['close'])
+    
+    # Fill NaN values
+    train_data = train_data.fillna(method='bfill').fillna(0)
+    
+    logger.info(f"Generated {len(train_data)} training samples")
+    return train_data
+
+
+def generate_test_data(config: dict) -> pd.DataFrame:
+    """Generate synthetic data for testing"""
+    logger.info("Generating synthetic test data")
+    
+    generator = SyntheticDataGenerator(config)
+    
+    # Generate test data (shorter period)
+    test_data = generator.generate_ohlcv_data(
+        symbol='AAPL',
+        start_date='2024-01-01',
+        end_date='2024-03-31',
+        frequency='1H'
+    )
+    
+    # Add technical indicators
+    test_data['sma_20'] = test_data['close'].rolling(window=20).mean()
+    test_data['sma_50'] = test_data['close'].rolling(window=50).mean()
+    test_data['rsi'] = calculate_rsi(test_data['close'])
+    bb_upper, bb_lower = calculate_bollinger_bands(test_data['close'])
+    test_data['bb_upper'] = bb_upper
+    test_data['bb_lower'] = bb_lower
+    test_data['macd'] = calculate_macd(test_data['close'])
+    
+    # Fill NaN values
+    test_data = test_data.fillna(method='bfill').fillna(0)
+    
+    logger.info(f"Generated {len(test_data)} test samples")
+    return test_data
+
+
+def calculate_rsi(prices: pd.Series, period: int = 14) -> pd.Series:
+    """Calculate RSI indicator"""
+    delta = prices.diff()
+    gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
+    loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
+    rs = gain / loss
+    rsi = 100 - (100 / (1 + rs))
+    return rsi
+
+
+def calculate_bollinger_bands(prices: pd.Series, period: int = 20, std_dev: int = 2):
+    """Calculate Bollinger Bands"""
+    sma = prices.rolling(window=period).mean()
+    std = prices.rolling(window=period).std()
+    upper_band = sma + (std * std_dev)
+    lower_band = sma - (std * std_dev)
+    return upper_band, lower_band
+
+
+def calculate_macd(prices: pd.Series, fast: int = 12, slow: int = 26, signal: int = 9) -> pd.Series:
+    """Calculate MACD indicator"""
+    ema_fast = prices.ewm(span=fast).mean()
+    ema_slow = prices.ewm(span=slow).mean()
+    macd_line = ema_fast - ema_slow
+    return macd_line
+
+
+def train_finrl_agent(config: dict, train_data: pd.DataFrame, test_data: pd.DataFrame) -> FinRLAgent:
+    """Train the FinRL agent"""
+    logger.info("Starting FinRL agent training")
+    
+    # Create FinRL agent
+    finrl_config = FinRLConfig(**config['finrl'])
+    agent = FinRLAgent(finrl_config)
+    
+    # Train the agent
+    training_result = agent.train(
+        data=train_data,
+        total_timesteps=config['finrl']['training']['total_timesteps'],
+        eval_freq=config['finrl']['training']['eval_freq'],
+        eval_data=test_data
+    )
+    
+    logger.info(f"Training completed: {training_result}")
+    
+    # Save the model
+    if config['finrl']['training']['save_best_model']:
+        model_path = config['finrl']['training']['model_save_path']
+        os.makedirs(os.path.dirname(model_path), exist_ok=True)
+        agent.save_model(model_path)
+    
+    return agent
+
+
+def evaluate_agent(agent: FinRLAgent, test_data: pd.DataFrame) -> dict:
+    """Evaluate the trained agent"""
+    logger.info("Evaluating FinRL agent")
+    
+    # Evaluate on test data
+    evaluation_results = agent.evaluate(test_data)
+    
+    logger.info(f"Evaluation results: {evaluation_results}")
+    
+    return evaluation_results
+
+
+def generate_predictions(agent: FinRLAgent, test_data: pd.DataFrame) -> list:
+    """Generate trading predictions"""
+    logger.info("Generating trading predictions")
+    
+    predictions = agent.predict(test_data)
+    
+    logger.info(f"Generated {len(predictions)} predictions")
+    
+    return predictions
+
+
+def plot_results(test_data: pd.DataFrame, predictions: list, evaluation_results: dict):
+    """Plot trading results"""
+    logger.info("Creating visualization plots")
+    
+    # Create figure with subplots
+    fig, axes = plt.subplots(3, 1, figsize=(15, 12))
+    
+    # Plot 1: Price and predictions
+    axes[0].plot(test_data.index, test_data['close'], label='Close Price', alpha=0.7)
+    
+    # Mark buy/sell signals
+    buy_signals = [i for i, pred in enumerate(predictions) if pred == 2]
+    sell_signals = [i for i, pred in enumerate(predictions) if pred == 0]
+    
+    if buy_signals:
+        axes[0].scatter(test_data.index[buy_signals], test_data['close'].iloc[buy_signals], 
+                       color='green', marker='^', s=100, label='Buy Signal', alpha=0.8)
+    if sell_signals:
+        axes[0].scatter(test_data.index[sell_signals], test_data['close'].iloc[sell_signals], 
+                       color='red', marker='v', s=100, label='Sell Signal', alpha=0.8)
+    
+    axes[0].set_title('Price Action and Trading Signals')
+    axes[0].set_ylabel('Price')
+    axes[0].legend()
+    axes[0].grid(True, alpha=0.3)
+    
+    # Plot 2: Technical indicators
+    axes[1].plot(test_data.index, test_data['close'], label='Close Price', alpha=0.7)
+    axes[1].plot(test_data.index, test_data['sma_20'], label='SMA 20', alpha=0.7)
+    axes[1].plot(test_data.index, test_data['sma_50'], label='SMA 50', alpha=0.7)
+    axes[1].plot(test_data.index, test_data['bb_upper'], label='BB Upper', alpha=0.5)
+    axes[1].plot(test_data.index, test_data['bb_lower'], label='BB Lower', alpha=0.5)
+    
+    axes[1].set_title('Technical Indicators')
+    axes[1].set_ylabel('Price')
+    axes[1].legend()
+    axes[1].grid(True, alpha=0.3)
+    
+    # Plot 3: RSI
+    axes[2].plot(test_data.index, test_data['rsi'], label='RSI', color='purple')
+    axes[2].axhline(y=70, color='r', linestyle='--', alpha=0.5, label='Overbought')
+    axes[2].axhline(y=30, color='g', linestyle='--', alpha=0.5, label='Oversold')
+    axes[2].set_title('RSI Indicator')
+    axes[2].set_ylabel('RSI')
+    axes[2].set_xlabel('Time')
+    axes[2].legend()
+    axes[2].grid(True, alpha=0.3)
+    
+    plt.tight_layout()
+    
+    # Save plot
+    os.makedirs('plots', exist_ok=True)
+    plt.savefig('plots/finrl_trading_results.png', dpi=300, bbox_inches='tight')
+    plt.show()
+    
+    logger.info("Plots saved to plots/finrl_trading_results.png")
+
+
+def print_summary(evaluation_results: dict, predictions: list):
+    """Print trading summary"""
+    print("\n" + "="*60)
+    print("FINRL TRADING SYSTEM SUMMARY")
+    print("="*60)
+    
+    print(f"Algorithm: {evaluation_results.get('algorithm', 'Unknown')}")
+    print(f"Total Return: {evaluation_results['total_return']:.2%}")
+    print(f"Final Portfolio Value: ${evaluation_results['final_portfolio_value']:,.2f}")
+    print(f"Total Reward: {evaluation_results['total_reward']:.4f}")
+    print(f"Sharpe Ratio: {evaluation_results['sharpe_ratio']:.4f}")
+    print(f"Number of Trading Steps: {evaluation_results['steps']}")
+    
+    # Trading statistics
+    buy_signals = sum(1 for pred in predictions if pred == 2)
+    sell_signals = sum(1 for pred in predictions if pred == 0)
+    hold_signals = sum(1 for pred in predictions if pred == 1)
+    
+    print(f"\nTrading Signals:")
+    print(f"  Buy signals: {buy_signals}")
+    print(f"  Sell signals: {sell_signals}")
+    print(f"  Hold signals: {hold_signals}")
+    print(f"  Total signals: {len(predictions)}")
+    
+    print("\n" + "="*60)
+
+
+def main():
+    """Main function to run the FinRL demo"""
+    logger.info("Starting FinRL Demo")
+    
+    try:
+        # Load configuration
+        config = load_config()
+        
+        # Generate data
+        train_data = generate_training_data(config)
+        test_data = generate_test_data(config)
+        
+        # Train FinRL agent
+        agent = train_finrl_agent(config, train_data, test_data)
+        
+        # Evaluate agent
+        evaluation_results = evaluate_agent(agent, test_data)
+        
+        # Generate predictions
+        predictions = generate_predictions(agent, test_data)
+        
+        # Create visualizations
+        plot_results(test_data, predictions, evaluation_results)
+        
+        # Print summary
+        print_summary(evaluation_results, predictions)
+        
+        logger.info("FinRL Demo completed successfully")
+        
+    except Exception as e:
+        logger.error(f"Error in FinRL demo: {str(e)}")
+        raise
+
+
+if __name__ == "__main__":
+    main() 

requirements.txt

@@ -7,3 +7,8 @@ pytest
 pytest-cov
 python-dateutil
 scipy
+finrl
+stable-baselines3
+gymnasium
+tensorboard
+torch

tests/test_finrl_agent.py

@@ -0,0 +1,373 @@
+"""
+Tests for FinRL Agent
+
+This module contains comprehensive tests for the FinRL agent functionality.
+"""
+
+import pytest
+import pandas as pd
+import numpy as np
+import yaml
+import tempfile
+import os
+from unittest.mock import Mock, patch
+
+# Add the project root to the path
+import sys
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from agentic_ai_system.finrl_agent import (
+    FinRLAgent, 
+    FinRLConfig, 
+    TradingEnvironment,
+    create_finrl_agent_from_config
+)
+
+
+class TestFinRLConfig:
+    """Test FinRL configuration"""
+    
+    def test_default_config(self):
+        """Test default configuration values"""
+        config = FinRLConfig()
+        
+        assert config.algorithm == "PPO"
+        assert config.learning_rate == 0.0003
+        assert config.batch_size == 64
+        assert config.gamma == 0.99
+    
+    def test_custom_config(self):
+        """Test custom configuration values"""
+        config = FinRLConfig(
+            algorithm="A2C",
+            learning_rate=0.001,
+            batch_size=128
+        )
+        
+        assert config.algorithm == "A2C"
+        assert config.learning_rate == 0.001
+        assert config.batch_size == 128
+
+
+class TestTradingEnvironment:
+    """Test trading environment"""
+    
+    @pytest.fixture
+    def sample_data(self):
+        """Create sample market data"""
+        dates = pd.date_range('2024-01-01', periods=100, freq='1H')
+        data = pd.DataFrame({
+            'open': np.random.uniform(100, 200, 100),
+            'high': np.random.uniform(100, 200, 100),
+            'low': np.random.uniform(100, 200, 100),
+            'close': np.random.uniform(100, 200, 100),
+            'volume': np.random.uniform(1000, 10000, 100),
+            'sma_20': np.random.uniform(100, 200, 100),
+            'sma_50': np.random.uniform(100, 200, 100),
+            'rsi': np.random.uniform(0, 100, 100),
+            'bb_upper': np.random.uniform(100, 200, 100),
+            'bb_lower': np.random.uniform(100, 200, 100),
+            'macd': np.random.uniform(-10, 10, 100)
+        }, index=dates)
+        return data
+    
+    def test_environment_initialization(self, sample_data):
+        """Test environment initialization"""
+        env = TradingEnvironment(sample_data)
+        
+        assert env.initial_balance == 100000
+        assert env.transaction_fee == 0.001
+        assert env.max_position == 100
+        assert env.action_space.n == 3
+        assert len(env.observation_space.shape) == 1
+    
+    def test_environment_reset(self, sample_data):
+        """Test environment reset"""
+        env = TradingEnvironment(sample_data)
+        obs, info = env.reset()
+        
+        assert env.current_step == 0
+        assert env.balance == env.initial_balance
+        assert env.position == 0
+        assert env.portfolio_value == env.initial_balance
+        assert isinstance(obs, np.ndarray)
+        assert isinstance(info, dict)
+    
+    def test_environment_step(self, sample_data):
+        """Test environment step"""
+        env = TradingEnvironment(sample_data)
+        obs, info = env.reset()
+        
+        # Test hold action
+        obs, reward, done, truncated, info = env.step(1)
+        
+        assert isinstance(obs, np.ndarray)
+        assert isinstance(reward, float)
+        assert isinstance(done, bool)
+        assert isinstance(truncated, bool)
+        assert isinstance(info, dict)
+        assert env.current_step == 1
+    
+    def test_buy_action(self, sample_data):
+        """Test buy action"""
+        env = TradingEnvironment(sample_data, initial_balance=10000)
+        obs, info = env.reset()
+        
+        initial_balance = env.balance
+        initial_position = env.position
+        
+        # Buy action
+        obs, reward, done, truncated, info = env.step(2)
+        
+        assert env.position > initial_position
+        assert env.balance < initial_balance
+    
+    def test_sell_action(self, sample_data):
+        """Test sell action"""
+        env = TradingEnvironment(sample_data, initial_balance=10000)
+        obs, info = env.reset()
+        
+        # First buy some shares
+        obs, reward, done, truncated, info = env.step(2)
+        initial_position = env.position
+        initial_balance = env.balance
+        
+        # Then sell
+        obs, reward, done, truncated, info = env.step(0)
+        
+        assert env.position < initial_position
+        assert env.balance > initial_balance
+    
+    def test_portfolio_value_calculation(self, sample_data):
+        """Test portfolio value calculation"""
+        env = TradingEnvironment(sample_data)
+        obs, info = env.reset()
+        
+        # Buy some shares
+        obs, reward, done, truncated, info = env.step(2)
+        
+        expected_value = env.balance + (env.position * sample_data.iloc[env.current_step]['close'])
+        assert abs(env.portfolio_value - expected_value) < 1e-6
+
+
+class TestFinRLAgent:
+    """Test FinRL agent"""
+    
+    @pytest.fixture
+    def sample_data(self):
+        """Create sample market data"""
+        dates = pd.date_range('2024-01-01', periods=100, freq='1H')
+        data = pd.DataFrame({
+            'open': np.random.uniform(100, 200, 100),
+            'high': np.random.uniform(100, 200, 100),
+            'low': np.random.uniform(100, 200, 100),
+            'close': np.random.uniform(100, 200, 100),
+            'volume': np.random.uniform(1000, 10000, 100)
+        }, index=dates)
+        return data
+    
+    @pytest.fixture
+    def finrl_config(self):
+        """Create FinRL configuration"""
+        return FinRLConfig(
+            algorithm="PPO",
+            learning_rate=0.0003,
+            batch_size=32,
+            total_timesteps=1000
+        )
+    
+    def test_agent_initialization(self, finrl_config):
+        """Test agent initialization"""
+        agent = FinRLAgent(finrl_config)
+        
+        assert agent.config == finrl_config
+        assert agent.model is None
+        assert agent.env is None
+    
+    def test_prepare_data(self, finrl_config, sample_data):
+        """Test data preparation"""
+        agent = FinRLAgent(finrl_config)
+        prepared_data = agent.prepare_data(sample_data)
+        
+        # Check that technical indicators were added
+        assert 'sma_20' in prepared_data.columns
+        assert 'sma_50' in prepared_data.columns
+        assert 'rsi' in prepared_data.columns
+        assert 'bb_upper' in prepared_data.columns
+        assert 'bb_lower' in prepared_data.columns
+        assert 'macd' in prepared_data.columns
+        
+        # Check that no NaN values remain
+        assert not prepared_data.isnull().any().any()
+    
+    def test_create_environment(self, finrl_config, sample_data):
+        """Test environment creation"""
+        agent = FinRLAgent(finrl_config)
+        env = agent.create_environment(sample_data)
+        
+        assert isinstance(env, TradingEnvironment)
+        assert env.data.equals(sample_data)
+    
+    def test_technical_indicators_calculation(self, finrl_config):
+        """Test technical indicators calculation"""
+        agent = FinRLAgent(finrl_config)
+        
+        # Test RSI calculation
+        prices = pd.Series([100, 101, 99, 102, 98, 103, 97, 104, 96, 105])
+        rsi = agent._calculate_rsi(prices, period=3)
+        assert len(rsi) == len(prices)
+        assert not rsi.isnull().all()
+        
+        # Test Bollinger Bands calculation
+        bb_upper, bb_lower = agent._calculate_bollinger_bands(prices, period=3)
+        assert len(bb_upper) == len(prices)
+        assert len(bb_lower) == len(prices)
+        assert (bb_upper >= bb_lower).all()
+        
+        # Test MACD calculation
+        macd = agent._calculate_macd(prices)
+        assert len(macd) == len(prices)
+    
+    @patch('agentic_ai_system.finrl_agent.PPO')
+    def test_training_ppo(self, mock_ppo, finrl_config, sample_data):
+        """Test PPO training"""
+        # Mock the PPO model
+        mock_model = Mock()
+        mock_ppo.return_value = mock_model
+        
+        agent = FinRLAgent(finrl_config)
+        result = agent.train(sample_data, total_timesteps=100)
+        
+        assert result['algorithm'] == 'PPO'
+        assert result['total_timesteps'] == 100
+        mock_model.learn.assert_called_once()
+    
+    @patch('agentic_ai_system.finrl_agent.A2C')
+    def test_training_a2c(self, mock_a2c):
+        """Test A2C training"""
+        config = FinRLConfig(algorithm="A2C")
+        mock_model = Mock()
+        mock_a2c.return_value = mock_model
+        
+        agent = FinRLAgent(config)
+        sample_data = pd.DataFrame({
+            'open': [100, 101, 102],
+            'high': [101, 102, 103],
+            'low': [99, 100, 101],
+            'close': [100, 101, 102],
+            'volume': [1000, 1100, 1200]
+        })
+        
+        result = agent.train(sample_data, total_timesteps=100)
+        
+        assert result['algorithm'] == 'A2C'
+        mock_model.learn.assert_called_once()
+    
+    def test_invalid_algorithm(self):
+        """Test invalid algorithm handling"""
+        config = FinRLConfig(algorithm="INVALID")
+        agent = FinRLAgent(config)
+        sample_data = pd.DataFrame({
+            'open': [100, 101, 102],
+            'high': [101, 102, 103],
+            'low': [99, 100, 101],
+            'close': [100, 101, 102],
+            'volume': [1000, 1100, 1200]
+        })
+        
+        with pytest.raises(ValueError, match="Unsupported algorithm"):
+            agent.train(sample_data, total_timesteps=100)
+    
+    def test_predict_without_training(self, finrl_config, sample_data):
+        """Test prediction without training"""
+        agent = FinRLAgent(finrl_config)
+        
+        with pytest.raises(ValueError, match="Model not trained"):
+            agent.predict(sample_data)
+    
+    def test_evaluate_without_training(self, finrl_config, sample_data):
+        """Test evaluation without training"""
+        agent = FinRLAgent(finrl_config)
+        
+        with pytest.raises(ValueError, match="Model not trained"):
+            agent.evaluate(sample_data)
+    
+    @patch('agentic_ai_system.finrl_agent.PPO')
+    def test_save_and_load_model(self, mock_ppo, finrl_config, sample_data):
+        """Test model saving and loading"""
+        # Mock the PPO model
+        mock_model = Mock()
+        mock_ppo.return_value = mock_model
+        mock_ppo.load.return_value = mock_model
+        
+        agent = FinRLAgent(finrl_config)
+        
+        # Train the agent
+        agent.train(sample_data, total_timesteps=100)
+        
+        # Test saving
+        with tempfile.NamedTemporaryFile(suffix='.zip', delete=False) as tmp_file:
+            agent.save_model(tmp_file.name)
+            mock_model.save.assert_called_once_with(tmp_file.name)
+        
+        # Test loading
+        agent.load_model(tmp_file.name)
+        mock_ppo.load.assert_called_once_with(tmp_file.name)
+        
+        # Clean up
+        os.unlink(tmp_file.name)
+
+
+class TestFinRLIntegration:
+    """Test FinRL integration with configuration"""
+    
+    def test_create_agent_from_config(self):
+        """Test creating agent from configuration file"""
+        config_data = {
+            'finrl': {
+                'algorithm': 'PPO',
+                'learning_rate': 0.001,
+                'batch_size': 128,
+                'gamma': 0.95
+            }
+        }
+        
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as tmp_file:
+            yaml.dump(config_data, tmp_file)
+            tmp_file_path = tmp_file.name
+        
+        try:
+            agent = create_finrl_agent_from_config(tmp_file_path)
+            
+            assert agent.config.algorithm == 'PPO'
+            assert agent.config.learning_rate == 0.001
+            assert agent.config.batch_size == 128
+            assert agent.config.gamma == 0.95
+        finally:
+            os.unlink(tmp_file_path)
+    
+    def test_create_agent_from_config_missing_finrl(self):
+        """Test creating agent from config without finrl section"""
+        config_data = {
+            'trading': {
+                'symbol': 'AAPL',
+                'capital': 100000
+            }
+        }
+        
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as tmp_file:
+            yaml.dump(config_data, tmp_file)
+            tmp_file_path = tmp_file.name
+        
+        try:
+            agent = create_finrl_agent_from_config(tmp_file_path)
+            
+            # Should use default values
+            assert agent.config.algorithm == 'PPO'
+            assert agent.config.learning_rate == 0.0003
+        finally:
+            os.unlink(tmp_file_path)
+
+
+if __name__ == "__main__":
+    pytest.main([__file__])