DistilBERT-Based Intent Detection Model for Banking Customer Queries

This repository contains a fine-tuned DistilBERT model for intent detection in banking customer support scenarios. It is trained on the BANKING77 dataset and designed to accurately classify user queries into 77 distinct banking-related intents.

Model Details

Model Architecture: DistilBERT Base Uncased
Task: Intent Detection for Banking Queries
Dataset: BANKING77
Fine-tuning Framework: Hugging Face Transformers
Language: English
Number of Labels: 77
Quantization: Not applied (full precision)

Usage

Installation

pip install transformers torch datasets

Loading the Model

from transformers import DistilBertTokenizer, DistilBertForSequenceClassification
import torch

# Load fine-tuned model
model_path = "./banking77-distilbert"  # Adjust path if different
model = DistilBertForSequenceClassification.from_pretrained(model_path)
tokenizer = DistilBertTokenizer.from_pretrained(model_path)
model.eval()

# Sample query
text = "I need to reset my online banking password."

# Tokenize and predict
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
with torch.no_grad():
    outputs = model(**inputs)
predicted_class = torch.argmax(outputs.logits, dim=1).item()

# Example label map (replace with the full BANKING77 map)
label_map = {0: "activate_my_card", 1: "balance", 2: "card_arrival", ..., 76: "why_was_my_card_declined"}
print(f"Predicted Intent: {label_map[predicted_class]}")

Performance Metrics

Accuracy: ~95% (on the BANKING77 test split)
Loss: ~0.13 (after fine-tuning for 4 epochs)

Fine-Tuning Details

Dataset

Name: BANKING77
Size: ~13,000 customer support queries
Intents: 77 unique labeled banking intents

Training

Epochs: 4
Batch Size: 16
Learning Rate: 2e-5
Optimizer: AdamW
Evaluation Strategy: per epoch
Loss Function: CrossEntropyLoss

Hardware

GPU Used: NVIDIA Tesla T4 (via Google Colab)
Training Time: ~15 minutes

Repository Structure

.
├── banking77-distilbert/     # Fine-tuned model directory (saved via trainer.save_model)
│   ├── config.json
│   ├── pytorch_model.bin
│   ├── tokenizer_config.json
│   ├── vocab.txt
├── intent_predictor.py       # Script for predicting intents (with preprocessing)
├── README.md                 # Model documentation

Limitations

The model is trained only on banking-related intents; it may misclassify out-of-domain queries.
Multilingual support is not included — limited to English.
Model does not handle multiple intents per query.

Contributing

Contributions and suggestions are welcome. Please open an issue or pull request for improvements or additional features.