EDITH_DistilBERT_Traing_Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DistilBERT Model Exploration\n",
    "\n",
    "In this section, we shift our focus to exploring the DistilBERT model, a lighter and faster version of the original BERT architecture. While DistilBERT aims to retain much of the knowledge and capabilities of BERT, it does so with fewer parameters and reduced computational requirements. This exploration is particularly relevant in scenarios where speed and efficiency are critical, such as real-time applications or resource-constrained environments.\n",
    "\n",
    "**Model Implementation and Purpose**\n",
    "\n",
    "Our implementation involves fine-tuning DistilBERT on the same dataset we used for the BERT model, aiming to evaluate its performance in the context of question answering. The goal is to see how well DistilBERT can understand and generate answers based on the provided context, even though it's widely recognized that it may not match BERT's performance in many cases.\n",
    "\n",
    "**Key Features of the Code**\n",
    "\n",
    "1. **Data Preparation**: Just like with BERT, we preprocess our data to ensure it's in the right format for DistilBERT. This includes tokenizing the questions and contexts, as well as padding them to ensure uniform input lengths.\n",
    "\n",
    "2. **Model Training**: We set up the training loop, where DistilBERT is fine-tuned on our dataset. The training process involves adjusting the model's weights based on how well it predicts the answers, optimizing its understanding of the context.\n",
    "\n",
    "3. **Evaluation & Testing**: Post-training, we evaluate the model using metrics such as ROUGE, BLEU, and F1 scores. These metrics provide insight into how accurately DistilBERT generates answers compared to the reference answers.\n",
    "\n",
    "4. **Performance Analysis**: After evaluation, we will compare the performance of DistilBERT with that of the BERT model. This analysis will help us understand the trade-offs involved in using a distilled version of BERT, particularly in terms of response accuracy and computational efficiency.\n",
    "\n",
    "**Conclusion and Future Directions**\n",
    "\n",
    "It's important to note that we haven't invested extensive time in optimizing DistilBERT for our use case, given the general consensus that it often yields inferior performance compared to BERT. However, this exploration is valuable for understanding the potential benefits of using lighter models in appropriate contexts. Future work may involve further tuning and experimenting with different configurations to maximize DistilBERT's effectiveness, particularly in situations where computational resources are limited. By examining DistilBERT alongside BERT, we gain insights into the versatility and adaptability of transformer-based models for context-based question answering applications.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2024-10-20T15:42:01.494045Z",
     "iopub.status.busy": "2024-10-20T15:42:01.493704Z",
     "iopub.status.idle": "2024-10-20T15:42:02.681840Z",
     "shell.execute_reply": "2024-10-20T15:42:02.680746Z",
     "shell.execute_reply.started": "2024-10-20T15:42:01.494004Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "# Import Gaussian filter for smoothing\n",
    "from scipy.ndimage import gaussian_filter1d\n",
    "import json\n",
    "import torch\n",
    "from transformers import DistilBertTokenizerFast, DistilBertForQuestionAnswering, AdamW\n",
    "from torch.utils.data import DataLoader, Dataset, random_split, default_collate\n",
    "from torch.cuda.amp import autocast, GradScaler\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:42:22.376215Z",
     "iopub.status.busy": "2024-10-20T15:42:22.375839Z",
     "iopub.status.idle": "2024-10-20T15:42:33.763837Z",
     "shell.execute_reply": "2024-10-20T15:42:33.762508Z",
     "shell.execute_reply.started": "2024-10-20T15:42:22.376183Z"
    }
   },
   "outputs": [],
   "source": [
    "!pip install -q transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_path = 'train-v1.1.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:42:33.766611Z",
     "iopub.status.busy": "2024-10-20T15:42:33.765615Z",
     "iopub.status.idle": "2024-10-20T15:42:33.923066Z",
     "shell.execute_reply": "2024-10-20T15:42:33.922123Z",
     "shell.execute_reply.started": "2024-10-20T15:42:33.766562Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 1: Initialize the tokenizer\n",
    "tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')\n",
    "\n",
    "# Custom dataset class for question-answering\n",
    "class QADataset(Dataset):\n",
    "    def __init__(self, examples):\n",
    "        self.examples = examples\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.examples)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        return self.examples[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:42:33.925160Z",
     "iopub.status.busy": "2024-10-20T15:42:33.924484Z",
     "iopub.status.idle": "2024-10-20T15:42:33.934652Z",
     "shell.execute_reply": "2024-10-20T15:42:33.933588Z",
     "shell.execute_reply.started": "2024-10-20T15:42:33.925113Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 2: Preprocessing function for the dataset\n",
    "def prepare_features(example):\n",
    "    inputs = tokenizer(\n",
    "        example['question'],\n",
    "        example['context'],\n",
    "        max_length=256,  # Reduced sequence length for faster processing\n",
    "        truncation=True,\n",
    "        padding=\"max_length\",\n",
    "        return_tensors=\"pt\",\n",
    "        return_offsets_mapping=True\n",
    "    )\n",
    "\n",
    "    if 'answers' not in example or len(example['answers']['answer_start']) == 0:\n",
    "        return None\n",
    "\n",
    "    start_char = example['answers']['answer_start'][0]\n",
    "    end_char = start_char + len(example['answers']['text'][0])\n",
    "    offset_mapping = inputs['offset_mapping'][0]\n",
    "\n",
    "    try:\n",
    "        start_pos = next(idx for idx, (start, end) in enumerate(offset_mapping) if start <= start_char < end)\n",
    "        end_pos = next(idx for idx, (start, end) in enumerate(offset_mapping) if start < end_char <= end)\n",
    "    except StopIteration:\n",
    "        return None\n",
    "\n",
    "    inputs['start_positions'] = torch.tensor([start_pos])\n",
    "    inputs['end_positions'] = torch.tensor([end_pos])\n",
    "    inputs.pop('offset_mapping')\n",
    "\n",
    "    return {\n",
    "        'input_ids': inputs['input_ids'].squeeze(),\n",
    "        'attention_mask': inputs['attention_mask'].squeeze(),\n",
    "        'start_positions': inputs['start_positions'].squeeze(),\n",
    "        'end_positions': inputs['end_positions'].squeeze(),\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:42:33.936157Z",
     "iopub.status.busy": "2024-10-20T15:42:33.935788Z",
     "iopub.status.idle": "2024-10-20T15:42:33.947810Z",
     "shell.execute_reply": "2024-10-20T15:42:33.946881Z",
     "shell.execute_reply.started": "2024-10-20T15:42:33.936123Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 3: Load and preprocess the dataset\n",
    "def load_dataset(dataset_path):\n",
    "    with open(dataset_path, 'r') as f:\n",
    "        dataset = json.load(f)\n",
    "        examples = []\n",
    "        for article in dataset['data']:\n",
    "            for paragraph in article['paragraphs']:\n",
    "                context = paragraph['context']\n",
    "                for qa in paragraph['qas']:\n",
    "                    question = qa['question']\n",
    "                    answers = {\n",
    "                        'answer_start': [ans['answer_start'] for ans in qa['answers']],\n",
    "                        'text': [ans['text'] for ans in qa['answers']]\n",
    "                    }\n",
    "                    examples.append({\n",
    "                        'context': context,\n",
    "                        'question': question,\n",
    "                        'answers': answers\n",
    "                    })\n",
    "        return examples\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:42:34.052073Z",
     "iopub.status.busy": "2024-10-20T15:42:34.051810Z",
     "iopub.status.idle": "2024-10-20T15:55:50.144298Z",
     "shell.execute_reply": "2024-10-20T15:55:50.143437Z",
     "shell.execute_reply.started": "2024-10-20T15:42:34.052042Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 4: Preprocess the dataset examples for training\n",
    "  \n",
    "dataset_examples = load_dataset(dataset_path)\n",
    "encoded_dataset = [prepare_features(example) for example in dataset_examples if prepare_features(example) is not None]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([  101,  2000,  3183,  2106,  1996,  6261,  2984,  9382,  3711,  1999,\n",
       "          8517,  1999, 10223, 26371,  2605,  1029,   102,  6549,  2135,  1010,\n",
       "          1996,  2082,  2038,  1037,  3234,  2839,  1012, 10234,  1996,  2364,\n",
       "          2311,  1005,  1055,  2751,  8514,  2003,  1037,  3585,  6231,  1997,\n",
       "          1996,  6261,  2984,  1012,  3202,  1999,  2392,  1997,  1996,  2364,\n",
       "          2311,  1998,  5307,  2009,  1010,  2003,  1037,  6967,  6231,  1997,\n",
       "          4828,  2007,  2608,  2039, 14995,  6924,  2007,  1996,  5722,  1000,\n",
       "          2310,  3490,  2618,  4748,  2033, 18168,  5267,  1000,  1012,  2279,\n",
       "          2000,  1996,  2364,  2311,  2003,  1996, 13546,  1997,  1996,  6730,\n",
       "          2540,  1012,  3202,  2369,  1996, 13546,  2003,  1996, 24665, 23052,\n",
       "          1010,  1037, 14042,  2173,  1997,  7083,  1998,  9185,  1012,  2009,\n",
       "          2003,  1037, 15059,  1997,  1996, 24665, 23052,  2012, 10223, 26371,\n",
       "          1010,  2605,  2073,  1996,  6261,  2984, 22353,  2135,  2596,  2000,\n",
       "          3002, 16595,  9648,  4674,  2061, 12083,  9711,  2271,  1999,  8517,\n",
       "          1012,  2012,  1996,  2203,  1997,  1996,  2364,  3298,  1006,  1998,\n",
       "          1999,  1037,  3622,  2240,  2008,  8539,  2083,  1017, 11342,  1998,\n",
       "          1996,  2751,  8514,  1007,  1010,  2003,  1037,  3722,  1010,  2715,\n",
       "          2962,  6231,  1997,  2984,  1012,   102,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
       "             0,     0,     0,     0,     0,     0]),\n",
       " 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n",
       " 'start_positions': tensor(130),\n",
       " 'end_positions': tensor(137)}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:57:26.018871Z",
     "iopub.status.busy": "2024-10-20T15:57:26.018005Z",
     "iopub.status.idle": "2024-10-20T15:57:26.022910Z",
     "shell.execute_reply": "2024-10-20T15:57:26.021938Z",
     "shell.execute_reply.started": "2024-10-20T15:57:26.018829Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 5: Create PyTorch dataset\n",
    "qa_dataset = QADataset(encoded_dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:57:31.125889Z",
     "iopub.status.busy": "2024-10-20T15:57:31.125528Z",
     "iopub.status.idle": "2024-10-20T15:57:31.154920Z",
     "shell.execute_reply": "2024-10-20T15:57:31.154116Z",
     "shell.execute_reply.started": "2024-10-20T15:57:31.125856Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 6: Split into training and validation sets\n",
    "train_size = int(0.9 * len(qa_dataset))\n",
    "val_size = len(qa_dataset) - train_size\n",
    "train_dataset, val_dataset = random_split(qa_dataset, [train_size, val_size])\n",
    "\n",
    "def custom_collate_fn(batch):\n",
    "    return default_collate([item for item in batch if item is not None])\n",
    "\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=16, shuffle=True, collate_fn=custom_collate_fn)\n",
    "val_dataloader = DataLoader(val_dataset, batch_size=8, collate_fn=custom_collate_fn)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explanation of DistilBERT Question-Answering Pipeline\n",
    "\n",
    "In this section, we outline the steps involved in preparing our dataset for fine-tuning the DistilBERT model on a question-answering task. This process is crucial for ensuring that the model receives the properly formatted input necessary for effective learning.\n",
    "\n",
    "1. **Dataset Path Initialization**:\n",
    "   We start by defining the path to the SQuAD1.1 dataset, which is a widely used dataset for training question-answering models. This JSON file contains questions, contexts, and corresponding answers that our model will learn from.\n",
    "\n",
    "2. **Tokenizer Initialization**:\n",
    "   We initialize the tokenizer for the DistilBERT model. This tokenizer converts text into a format that the model can understand, including input IDs and attention masks. By loading a pre-trained tokenizer, we ensure consistency with the DistilBERT vocabulary.\n",
    "\n",
    "3. **Custom Dataset Class**:\n",
    "   A custom dataset class is defined to handle our examples. This class allows us to easily manage the dataset, including determining its length and accessing individual examples based on their index.\n",
    "\n",
    "4. **Preprocessing Function**:\n",
    "   The preprocessing function is responsible for tokenizing each example (question and context) and generating the necessary input format for the model. It sets a maximum sequence length, handles padding and truncation, and calculates the start and end positions of answers within the context. This step is critical for training the model to locate answers in the text.\n",
    "\n",
    "5. **Loading and Preprocessing the Dataset**:\n",
    "   We load the dataset by reading the JSON file and extracting relevant data into a structured format. Each entry consists of a context, a question, and associated answers. This organization is essential for further processing.\n",
    "\n",
    "6. **Encoding the Dataset Examples**:\n",
    "   After loading the dataset examples, we encode each example using the preprocessing function, filtering out any that cannot be processed. This ensures that we retain only valid inputs for training.\n",
    "\n",
    "7. **Creating the PyTorch Dataset**:\n",
    "   We create an instance of our custom dataset class, passing in the encoded examples. This structure facilitates working with the dataset in PyTorch, allowing for straightforward data manipulation during training.\n",
    "\n",
    "8. **Splitting the Dataset**:\n",
    "   The dataset is split into training and validation sets, typically allocating 90% for training and 10% for validation. This division is crucial for evaluating the model's performance on unseen data during the training process.\n",
    "\n",
    "9. **Custom Collate Function**:\n",
    "   We define a custom collate function to handle batches of data, ensuring that only valid items are included in each batch. This prevents potential issues during the training phase.\n",
    "\n",
    "10. **Creating DataLoaders**:\n",
    "    Finally, we create `DataLoader` instances for both the training and validation datasets. These loaders facilitate batch processing, allowing us to efficiently feed data into the model in specified batch sizes.\n",
    "\n",
    "### Summary\n",
    "\n",
    "This code establishes a comprehensive pipeline for preparing data for fine-tuning the DistilBERT model on a question-answering task. By carefully preprocessing the dataset and structuring it for PyTorch, we set the stage for effective model training and evaluation. As we proceed with the training, we will be able to assess the performance of DistilBERT in understanding context and generating answers based on the provided questions and contexts.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Training\n",
    "\n",
    "\n",
    "In this section, we implement the training and evaluation process for a DistilBERT model tailored for question answering tasks.\n",
    "\n",
    "1. **Model Initialization**: We load a pre-trained DistilBERT model, which provides a robust foundation for understanding language, and set up an optimizer to adjust the model's parameters during training.\n",
    "\n",
    "2. **Device Setup**: We determine whether to use a GPU or CPU for training to maximize efficiency, enabling faster computations.\n",
    "\n",
    "3. **Mixed Precision Training**: By employing mixed precision, we enhance training speed and reduce memory usage, allowing us to work with larger models or batch sizes.\n",
    "\n",
    "4. **Training and Validation Functions**: We define functions for training and validating the model. The training function computes the loss and updates the model's weights, while the validation function evaluates model performance on a separate dataset to monitor its accuracy.\n",
    "\n",
    "5. **Training Loop**: A loop executes the training and validation processes for a specified number of epochs, tracking and printing performance metrics to observe improvements over time.\n",
    "\n",
    "6. **Model Saving**: After training, we save the trained model and tokenizer to a designated directory for future use.\n",
    "\n",
    "This structured approach facilitates efficient training and evaluation of the DistilBERT model, ensuring it is well-prepared for real-world question-answering applications.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:57:36.950445Z",
     "iopub.status.busy": "2024-10-20T15:57:36.950023Z",
     "iopub.status.idle": "2024-10-20T15:57:39.550098Z",
     "shell.execute_reply": "2024-10-20T15:57:39.549285Z",
     "shell.execute_reply.started": "2024-10-20T15:57:36.950405Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "df9ea53ab3344dea8253eac2d673ff4a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "/opt/conda/lib/python3.10/site-packages/transformers/optimization.py:591: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Step 7: Initialize the model for Question Answering\n",
    "model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased')\n",
    "optimizer = AdamW(model.parameters(), lr=3e-5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DistilBertForQuestionAnswering(\n",
       "  (distilbert): DistilBertModel(\n",
       "    (embeddings): Embeddings(\n",
       "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (transformer): Transformer(\n",
       "      (layer): ModuleList(\n",
       "        (0-5): 6 x TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (activation): GELUActivation()\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (qa_outputs): Linear(in_features=768, out_features=2, bias=True)\n",
       "  (dropout): Dropout(p=0.1, inplace=False)\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 8: Set up the device for training\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T16:09:07.905385Z",
     "iopub.status.busy": "2024-10-20T16:09:07.904589Z",
     "iopub.status.idle": "2024-10-20T16:09:07.985212Z",
     "shell.execute_reply": "2024-10-20T16:09:07.984286Z",
     "shell.execute_reply.started": "2024-10-20T16:09:07.905341Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_30/590130087.py:6: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.\n",
      "  scaler = GradScaler()\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "# Mixed precision scaler\n",
    "scaler = GradScaler()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:57:48.870796Z",
     "iopub.status.busy": "2024-10-20T15:57:48.869932Z",
     "iopub.status.idle": "2024-10-20T15:57:48.884453Z",
     "shell.execute_reply": "2024-10-20T15:57:48.881684Z",
     "shell.execute_reply.started": "2024-10-20T15:57:48.870754Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 9: Define the training and validation functions with mixed precision\n",
    "def train(model, dataloader, optimizer, device, scaler):\n",
    "    model.train()\n",
    "    total_loss = 0\n",
    "    for batch in dataloader:\n",
    "        input_ids = batch['input_ids'].to(device)\n",
    "        attention_mask = batch['attention_mask'].to(device)\n",
    "        start_positions = batch['start_positions'].to(device)\n",
    "        end_positions = batch['end_positions'].to(device)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Use autocast for mixed precision\n",
    "        with autocast():\n",
    "            outputs = model(input_ids, attention_mask=attention_mask, start_positions=start_positions, end_positions=end_positions)\n",
    "            loss = outputs.loss\n",
    "            total_loss += loss.item()\n",
    "\n",
    "        # Scale the loss and backpropagate\n",
    "        scaler.scale(loss).backward()\n",
    "        scaler.step(optimizer)\n",
    "        scaler.update()\n",
    "\n",
    "    return total_loss / len(dataloader)\n",
    "\n",
    "def validate(model, dataloader, device):\n",
    "    model.eval()\n",
    "    total_loss = 0\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    with torch.no_grad():\n",
    "        for batch in dataloader:\n",
    "            input_ids = batch['input_ids'].to(device)\n",
    "            attention_mask = batch['attention_mask'].to(device)\n",
    "            start_positions = batch['start_positions'].to(device)\n",
    "            end_positions = batch['end_positions'].to(device)\n",
    "\n",
    "            outputs = model(input_ids, attention_mask=attention_mask, start_positions=start_positions, end_positions=end_positions)\n",
    "            loss = outputs.loss\n",
    "            total_loss += loss.item()\n",
    "\n",
    "            # Get the predicted start and end positions\n",
    "            start_preds = torch.argmax(outputs.start_logits, dim=1)\n",
    "            end_preds = torch.argmax(outputs.end_logits, dim=1)\n",
    "\n",
    "            # Calculate accuracy: Check if both start and end positions are correct\n",
    "            correct += ((start_preds == start_positions) & (end_preds == end_positions)).sum().item()\n",
    "            total += input_ids.size(0)\n",
    "\n",
    "    avg_loss = total_loss / len(dataloader)\n",
    "    accuracy = correct / total if total > 0 else 0\n",
    "    return avg_loss, accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-18T20:24:54.990665Z",
     "iopub.status.busy": "2024-10-18T20:24:54.990241Z",
     "iopub.status.idle": "2024-10-18T20:59:50.571427Z",
     "shell.execute_reply": "2024-10-18T20:59:50.570475Z",
     "shell.execute_reply.started": "2024-10-18T20:24:54.990623Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_30/3732227169.py:14: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n",
      "  with autocast():\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3, Training Loss: 1.7261, Validation Loss: 1.3148, Validation Accuracy: 0.4825\n",
      "Epoch 2/3, Training Loss: 1.1330, Validation Loss: 1.3120, Validation Accuracy: 0.4916\n",
      "Epoch 3/3, Training Loss: 0.8496, Validation Loss: 1.3419, Validation Accuracy: 0.4988\n",
      "\n",
      "Total Average Validation Accuracy over 3 epochs: 0.4910\n"
     ]
    }
   ],
   "source": [
    "# Step 10: Training Loop with accuracy printing\n",
    "num_epochs = 3  # Reduced number of epochs for faster results\n",
    "total_accuracy = 0  # To track cumulative accuracy across epochs\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    avg_train_loss = train(model, train_dataloader, optimizer, device, scaler)\n",
    "    avg_val_loss, val_accuracy = validate(model, val_dataloader, device)\n",
    "    \n",
    "    # Accumulate total accuracy over all epochs\n",
    "    total_accuracy += val_accuracy\n",
    "    \n",
    "    print(f\"Epoch {epoch+1}/{num_epochs}, Training Loss: {avg_train_loss:.4f}, Validation Loss: {avg_val_loss:.4f}, Validation Accuracy: {val_accuracy:.4f}\")\n",
    "\n",
    "# Print the cumulative accuracy over all epochs\n",
    "avg_total_accuracy = total_accuracy / num_epochs\n",
    "print(f\"\\nTotal Average Validation Accuracy over {num_epochs} epochs: {avg_total_accuracy:.4f}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T15:59:38.631362Z",
     "iopub.status.busy": "2024-10-20T15:59:38.630998Z",
     "iopub.status.idle": "2024-10-20T15:59:41.417489Z",
     "shell.execute_reply": "2024-10-20T15:59:41.416356Z",
     "shell.execute_reply.started": "2024-10-20T15:59:38.631329Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('squad-distilbert-trained/DISTILBERT_model/tokenizer_config.json',\n",
       " 'squad-distilbert-trained/DISTILBERT_model/special_tokens_map.json',\n",
       " 'squad-distilbert-trained/DISTILBERT_model/vocab.txt',\n",
       " 'squad-distilbert-trained/DISTILBERT_model/added_tokens.json',\n",
       " 'squad-distilbert-trained/DISTILBERT_model/tokenizer.json')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the directory where you want to save the model\n",
    "model_save_path = 'DISTILBERT_model'\n",
    "\n",
    "# Create the directory if it doesn't exist\n",
    "if not os.path.exists(model_save_path):\n",
    "    os.makedirs(model_save_path)\n",
    "\n",
    "\n",
    "# Save the trained model and tokenizer\n",
    "model.save_pretrained(model_save_path)\n",
    "tokenizer.save_pretrained(model_save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T16:01:01.362751Z",
     "iopub.status.busy": "2024-10-20T16:01:01.362320Z",
     "iopub.status.idle": "2024-10-20T16:01:16.259581Z",
     "shell.execute_reply": "2024-10-20T16:01:16.258628Z",
     "shell.execute_reply.started": "2024-10-20T16:01:01.362708Z"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "# # Compress the model directory into a zip file\n",
    "# !zip -r distilbertmodel.zip squad-distilbert-trained/DISTILBERT_model\n",
    "\n",
    "# # Download the zip file to your local system\n",
    "# from IPython.display import FileLink\n",
    "\n",
    "# # Display a link to download the file\n",
    "# FileLink(r'distilbertmodel.zip')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Post Code Summary\n",
    "\n",
    "In this section, we perform the initialization, training, and validation of the DistilBERT model specifically designed for question answering tasks using the `DistilBertForQuestionAnswering` class from the Hugging Face Transformers library.\n",
    "\n",
    "1. **Model Initialization**: \n",
    "   - We load the pre-trained DistilBERT model, which has been fine-tuned on the SQuAD dataset, making it well-suited for extracting answers from a given context based on a specific question. The model is initialized with a learning rate of `3e-5` to ensure stable convergence during training.\n",
    "\n",
    "2. **Optimizer Setup**: \n",
    "   - The AdamW optimizer is employed to improve weight updates during training, accounting for weight decay, which helps prevent overfitting.\n",
    "\n",
    "3. **Device Configuration**: \n",
    "   - We check for GPU availability and move the model to the appropriate device (GPU or CPU) for faster training. This is critical for handling larger datasets efficiently.\n",
    "\n",
    "4. **Mixed Precision Training**: \n",
    "   - To optimize performance and reduce memory usage, we implement mixed precision training using PyTorch's GradScaler and autocast functions. This technique allows the model to use half-precision floating-point numbers where applicable, significantly speeding up training without sacrificing much in terms of model accuracy.\n",
    "\n",
    "5. **Training Function**: \n",
    "   - The training function iterates through the training data, calculating the loss for each batch. We use the model’s outputs to compute the loss based on the start and end positions of the answers. The loss is scaled and backpropagated, and the optimizer updates the model weights.\n",
    "\n",
    "6. **Validation Function**: \n",
    "   - The validation function assesses the model's performance on a separate validation dataset. It calculates the loss and accuracy by comparing the predicted start and end positions with the true labels. This step helps gauge the model's ability to generalize beyond the training data.\n",
    "\n",
    "7. **Training Loop**: \n",
    "   - Over three epochs, we track the average training loss, validation loss, and validation accuracy. The training loss decreases over epochs, indicating that the model is learning; however, validation loss fluctuates slightly, suggesting that the model might not be fully converging and could benefit from additional training or tuning.\n",
    "\n",
    "8. **Model Saving**: \n",
    "   - After completing the training process, we save the fine-tuned model and tokenizer to a specified directory (`DISTILBERT_model`). This step ensures that the trained model can be easily loaded for inference or further training without the need to retrain from scratch.\n",
    "\n",
    "Overall, this implementation showcases a structured approach to fine-tuning a DistilBERT model for question answering tasks, leveraging modern techniques like mixed precision training to enhance performance and efficiency.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation & Testing\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer=DistilBertTokenizerFast.from_pretrained('DISTILBERT_model')\n",
    "model = DistilBertForQuestionAnswering.from_pretrained('DISTILBERT_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T16:07:34.892291Z",
     "iopub.status.busy": "2024-10-20T16:07:34.891437Z",
     "iopub.status.idle": "2024-10-20T16:07:34.899630Z",
     "shell.execute_reply": "2024-10-20T16:07:34.898495Z",
     "shell.execute_reply.started": "2024-10-20T16:07:34.892244Z"
    }
   },
   "outputs": [],
   "source": [
    "# Step 12: Prediction function\n",
    "def predict(context, question):\n",
    "    inputs = tokenizer(question, context, max_length=256, truncation=True, padding=\"max_length\", return_tensors=\"pt\").to(device)\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        outputs = model(**inputs)\n",
    "        start_scores = outputs.start_logits\n",
    "        end_scores = outputs.end_logits\n",
    "\n",
    "        start_idx = torch.argmax(start_scores)\n",
    "        end_idx = torch.argmax(end_scores)\n",
    "\n",
    "        input_ids = inputs['input_ids'].squeeze().tolist()\n",
    "        answer_tokens = input_ids[start_idx:end_idx + 1]\n",
    "        answer = tokenizer.decode(answer_tokens)\n",
    "\n",
    "    return answer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T16:09:12.435863Z",
     "iopub.status.busy": "2024-10-20T16:09:12.435005Z",
     "iopub.status.idle": "2024-10-20T16:09:26.610097Z",
     "shell.execute_reply": "2024-10-20T16:09:26.609139Z",
     "shell.execute_reply.started": "2024-10-20T16:09:12.435820Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Answer: new delhi\n"
     ]
    }
   ],
   "source": [
    "context = \"India is a country located in South Asia. Its captial is New Delhi.\"\n",
    "question = \"What is the capital of India?\"\n",
    "\n",
    "predicted_answer = predict(context, question)\n",
    "print(\"Predicted Answer:\", predicted_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-18T21:00:02.159304Z",
     "iopub.status.busy": "2024-10-18T21:00:02.158666Z",
     "iopub.status.idle": "2024-10-18T21:00:02.201810Z",
     "shell.execute_reply": "2024-10-18T21:00:02.200906Z",
     "shell.execute_reply.started": "2024-10-18T21:00:02.159268Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Answer: saint bernadette soubirous\n"
     ]
    }
   ],
   "source": [
    "context = \"Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \\\"Venite Ad Me Omnes\\\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\"\n",
    "question = \"To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\"\n",
    "\n",
    "predicted_answer = predict(context, question)\n",
    "print(\"Predicted Answer:\", predicted_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-20T16:11:39.620287Z",
     "iopub.status.busy": "2024-10-20T16:11:39.619610Z",
     "iopub.status.idle": "2024-10-20T16:11:39.641221Z",
     "shell.execute_reply": "2024-10-20T16:11:39.640332Z",
     "shell.execute_reply.started": "2024-10-20T16:11:39.620243Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Answer: music has the power to inspire, heal, and entertain\n"
     ]
    }
   ],
   "source": [
    "\n",
    "context = '''Music is an art form whose medium is sound and silence. It is a universal language that \n",
    "can evoke emotions, convey messages, and bring people together. Music has been an \n",
    "integral part of human culture for centuries, with various genres and styles emerging \n",
    "across the world. From classical to contemporary, music has the power to inspire, heal, \n",
    "and entertain.'''\n",
    "\n",
    "question = \"What is the difference between Music and Harmony?\"\n",
    "\n",
    "predicted_answer = predict(context, question)\n",
    "print(\"Predicted Answer:\", predicted_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-18T21:00:02.223992Z",
     "iopub.status.busy": "2024-10-18T21:00:02.223682Z",
     "iopub.status.idle": "2024-10-18T21:00:02.243487Z",
     "shell.execute_reply": "2024-10-18T21:00:02.242627Z",
     "shell.execute_reply.started": "2024-10-18T21:00:02.223961Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Answer: california\n"
     ]
    }
   ],
   "source": [
    "context = \"Pappu is from america. He is from california\"\n",
    "question = \"which country is pappu from?\"\n",
    "\n",
    "predicted_answer = predict(context, question)\n",
    "print(\"Predicted Answer:\", predicted_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Answer: music has the power to inspire, heal, and entertain\n",
      "Actual Answer: music is an art form and harmony refers to the combination of sounds.\n",
      "BLEU Score: 0.2850\n",
      "ROUGE-1 F1 Score: 0.3636\n",
      "ROUGE-2 F1 Score: 0.0000\n",
      "ROUGE-L F1 Score: 0.1818\n",
      "F1 Score: 0.3636\n"
     ]
    }
   ],
   "source": [
    "from nltk.translate.bleu_score import sentence_bleu\n",
    "from rouge_score import rouge_scorer\n",
    "\n",
    "# Example inputs\n",
    "context = \"\"\"Music is an art form whose medium is sound and silence. It is a universal language that \n",
    "can evoke emotions, convey messages, and bring people together. Music has been an \n",
    "integral part of human culture for centuries, with various genres and styles emerging \n",
    "across the world. From classical to contemporary, music has the power to inspire, heal, \n",
    "and entertain.\"\"\"\n",
    "\n",
    "question = \"What is the difference between Music and Harmony?\"\n",
    "\n",
    "# Define actual and predicted answers\n",
    "actual_answer = \"Music is an art form and harmony refers to the combination of sounds.\"\n",
    "predicted_answer = \"music has the power to inspire, heal, and entertain\"\n",
    "\n",
    "# Normalize answers (lowercase)\n",
    "actual_answer = actual_answer.lower().strip()\n",
    "predicted_answer = predicted_answer.lower().strip()\n",
    "\n",
    "# Calculate BLEU score\n",
    "bleu_score = sentence_bleu([actual_answer.split()], predicted_answer.split(), weights=(1.0, 0.0, 0.0, 0.0))\n",
    "\n",
    "# Calculate ROUGE scores\n",
    "scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)\n",
    "rouge_scores = scorer.score(actual_answer, predicted_answer)\n",
    "\n",
    "# Calculate F1 score based on ROUGE-1 precision and recall\n",
    "precision = rouge_scores['rouge1'].precision\n",
    "recall = rouge_scores['rouge1'].recall\n",
    "if precision + recall > 0:  # Avoid division by zero\n",
    "    f1_score = 2 * (precision * recall) / (precision + recall)\n",
    "else:\n",
    "    f1_score = 0.0\n",
    "\n",
    "# Print the results\n",
    "print(f\"Predicted Answer: {predicted_answer}\")\n",
    "print(f\"Actual Answer: {actual_answer}\")\n",
    "print(f\"BLEU Score: {bleu_score:.4f}\")\n",
    "print(f\"ROUGE-1 F1 Score: {rouge_scores['rouge1'].fmeasure:.4f}\")\n",
    "print(f\"ROUGE-2 F1 Score: {rouge_scores['rouge2'].fmeasure:.4f}\")\n",
    "print(f\"ROUGE-L F1 Score: {rouge_scores['rougeL'].fmeasure:.4f}\")\n",
    "print(f\"F1 Score: {f1_score:.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results Summary\n",
    "\n",
    "**Predicted Answer**: music has the power to inspire, heal, and entertain  \n",
    "**Actual Answer**: music is an art form and harmony refers to the combination of sounds.  \n",
    "**BLEU Score**: 0.2850  \n",
    "**ROUGE-1 F1 Score**: 0.3636  \n",
    "**ROUGE-2 F1 Score**: 0.0000  \n",
    "**ROUGE-L F1 Score**: 0.1818  \n",
    "**F1 Score**: 0.3636  \n",
    "\n",
    "These results indicate that DistilBERT is not performing well at the current level of fine-tuning. The low scores suggest it may require more extensive training to improve its ability to generate accurate and relevant answers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "nvidiaTeslaT4",
   "dataSources": [
    {
     "datasetId": 5902073,
     "sourceId": 9660462,
     "sourceType": "datasetVersion"
    },
    {
     "isSourceIdPinned": true,
     "modelId": 142964,
     "modelInstanceId": 119726,
     "sourceId": 141334,
     "sourceType": "modelInstanceVersion"
    }
   ],
   "dockerImageVersionId": 30787,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}