PROJECT_STRUCTURE.md

# Project Structure

This document provides a detailed overview of the CodeLLaMA-Linux-BugFix project structure, explaining the purpose and organization of each component.

## 📁 Root Directory

```

CodeLLaMA-Linux-BugFix/

├── dataset_builder/          # Dataset creation and processing

├── dataset/                  # Generated datasets and data files

├── train/                    # Model training scripts and outputs

├── evaluate/                 # Model evaluation and testing

├── requirements.txt          # Python dependencies

├── README.md                 # Project documentation

└── PROJECT_STRUCTURE.md      # This file

```

## 🔧 Dataset Builder (`dataset_builder/`)



The dataset builder extracts bug-fix data from the Linux kernel Git repository and converts it into training-ready format.



### Files:

- **`extract_linux_bugfixes.py`** - Main dataset extraction script

  - Uses PyDriller to analyze Linux kernel Git history

  - Filters commits using bug-fix keywords

  - Extracts code context around bug locations

  - Generates structured dataset entries



- **`extract_linux_bugfixes_parallel.py`** - Parallelized version of dataset builder

  - Multi-process implementation for faster processing

  - Configurable worker count (default: 16 workers)

  - Test mode with limited commit processing



- **`format_for_training.py`** - Format conversion script

  - Converts structured data to prompt-completion pairs

  - Formats input for supervised fine-tuning

  - Creates training-ready JSONL format



### Key Features:

- **Commit Filtering**: Identifies bug-fix commits using 17 keywords

- **Code Context**: Extracts 10 lines before/after bug location

- **File Filtering**: Focuses on C and header files (`.c`, `.h`)

- **Diff Extraction**: Captures Git diff patches for fixes



## 📊 Dataset (`dataset/`)



Contains the generated datasets used for training and evaluation.



### Files:

- **`training_data_100k.jsonl`** - Main training dataset

  - 100,000 bug-fix samples

  - Structured format with input/output pairs

  - Stored using Git LFS for large file handling



- **`training_data_prompt_completion.jsonl`** - Converted training format

  - Prompt-completion pairs for supervised learning

  - Optimized for transformer model training

  - Stored using Git LFS



### Data Format:

```json

{

  "input": {

    "original code": "C code snippet with bug",

    "instruction": "Bug fix instruction from commit message"

  },

  "output": {

    "diff codes": "Git diff showing the fix"

  }

}

```



## 🚀 Training (`train/`)



Contains all training-related scripts, configurations, and model outputs.



### Files:

- **`train_codellama_qlora_linux_bugfix.py`** - Main training script

  - QLoRA fine-tuning implementation

  - Optimized for H200 GPU with bfloat16

  - Includes Weights & Biases integration

  - Comprehensive training configuration



- **`train_codellama_qlora_simple.py`** - Alternative training script

  - Simplified QLoRA implementation

  - Basic training setup without advanced features

  - Good for testing and development



- **`download_codellama_model.py`** - Model download utility

  - Downloads base CodeLLaMA-7B-Instruct model

  - Ensures model availability before training



### Output Directory (`train/output/`):

- **`qlora-codellama-bugfix/`** - Main model output

  - **`adapter_model.safetensors`** - LoRA adapter weights

  - **`adapter_config.json`** - LoRA configuration

  - **`tokenizer.json`** - Tokenizer files

  - **`chat_template.jinja`** - Conversation template

  - **`checkpoint-500/`** - Training checkpoint at step 500

  - **`checkpoint-1000/`** - Training checkpoint at step 1000

  - **`README.md`** - Model card and documentation



### Training Configuration:

- **Base Model**: `codellama/CodeLLaMA-7b-Instruct-hf`

- **Method**: QLoRA with 4-bit quantization

- **LoRA Config**: r=64, alpha=16, dropout=0.1

- **Training**: 3 epochs, batch size 64, learning rate 2e-4

- **Hardware**: Optimized for H200 GPU



## 📈 Evaluation (`evaluate/`)



Contains evaluation scripts and results for assessing model performance.



### Files:

- **`evaluate_linux_bugfix_model.py`** - Main evaluation script

  - Loads fine-tuned model for inference

  - Generates predictions on test data

  - Computes BLEU and ROUGE metrics

  - Saves results in multiple formats



- **`test_samples.jsonl`** - Evaluation dataset

  - Test samples for model evaluation

  - Stored using Git LFS



### Output Directory (`evaluate/output/`):

- **`eval_results.json`** - Detailed evaluation results

  - Complete predictions and references

  - Stored using Git LFS



- **`eval_results.csv`** - Tabular evaluation results

  - CSV format for easy analysis

  - Stored using Git LFS



### Evaluation Metrics:

- **BLEU Score**: Measures translation quality

- **ROUGE Score**: Evaluates text generation accuracy

- **Human Evaluation**: Qualitative assessment



## 🔧 Dependencies (`requirements.txt`)



Comprehensive list of Python packages required for the project:



### Core ML Libraries:

- `transformers==4.53.1` - Hugging Face transformers

- `torch==2.7.1+cu128` - PyTorch with CUDA support

- `peft==0.16.0` - Parameter-efficient fine-tuning

- `accelerate==1.8.1` - Distributed training

- `bitsandbytes==0.46.1` - Quantization support



### Data Processing:

- `datasets==3.6.0` - Dataset handling

- `pandas==2.3.1` - Data manipulation

- `numpy==2.3.1` - Numerical computing



### Git Analysis:

- `pydriller` - Git repository mining

- `gitpython` - Git operations



### Utilities:

- `tqdm==4.67.1` - Progress bars

- `wandb` - Experiment tracking

- `evaluate==0.4.4` - Evaluation metrics



## 🔄 Workflow



### 1. Dataset Creation

```bash

cd dataset_builder
python extract_linux_bugfixes.py          # Extract bug-fix data
python format_for_training.py  # Convert format
```



### 2. Model Training

```bash

cd train

python train_codellama_qlora_linux_bugfix.py                  # Train with QLoRA

```

### 3. Model Evaluation
```bash

cd evaluate

python evaluate_linux_bugfix_model.py               # Evaluate performance

```

## 🎯 Key Design Principles

### Modularity
- Each component has a specific responsibility
- Clear separation between data, training, and evaluation
- Easy to modify or extend individual components

### Efficiency
- QLoRA for memory-efficient training
- Parallel processing for dataset creation
- Optimized for modern GPU hardware

### Reproducibility
- Version-controlled dependencies
- Structured data formats
- Comprehensive logging and evaluation

### Scalability
- Configurable parameters for different hardware
- Support for distributed training
- Efficient data handling with Git LFS

## 🔍 File Naming Conventions

- **Scripts**: Descriptive names with clear purpose
- **Datasets**: Include size/version information
- **Models**: Include architecture and method
- **Results**: Include timestamp or version
- **Configs**: Use `.json` or `.yaml` format

## 📝 Documentation

- **README.md**: Project overview and quick start
- **PROJECT_STRUCTURE.md**: This detailed structure guide

- **Model README**: Generated model cards in output directories

- **Code Comments**: Inline documentation in all scripts



This structure ensures the project is organized, maintainable, and easy to understand for both users and contributors.