omark807/dsb-biases-model

Flan-T5-Large LoRA Adapter for Disability Q&A and Mitigating Disability Biases

Model Description

This repository contains a LoRA (Low-Rank Adaptation) adapter fine-tuned on the google/flan-t5-large base model. The adapter is specifically trained for improving question-answering capabilities related to disability information and actively reducing harmful biases and stereotypes concerning people with disabilities in generated text.

This model leverages the PEFT (Parameter-Efficient Fine-Tuning) library to efficiently adapt the large Flan-T5 model to this specialized domain without requiring full model retraining, making it more resource-efficient and deployable.

• Developed by: omark807
• Finetuned from model: google/flan-t5-large
• Model type: Adapter (LoRA) for Sequence-to-Sequence Language Model
• Language(s) (NLP): English
• License: GPL

Base Model Details (google/flan-t5-large)

Flan-T5 is an instruction-tuned variant of the T5 text-to-text transformer model. It has been fine-tuned on a collection of datasets expressed as natural language instructions. The "large" version has approximately 770 million parameters. This adapter builds upon its strong instruction-following capabilities.

• Original Model Card: https://huggingface.co/google/flan-t5-large

Uses

Direct Use

This adapter is intended to be loaded alongside the google/flan-t5-large model using the PEFT library. It can then be used for:

• Answering questions related to various aspects of disability, accessibility, disability rights, legislation, and common challenges.
• Generating responses that are more inclusive, respectful, and free from common disability biases and stereotypes.
• Providing information in a neutral and empathetic tone when discussing disability-related topics.

Example Inference for Q&A:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel, PeftConfig
import torch

# Load the base model
model_name = "google/flan-t5-large"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Load your adapter
# Replace "your-huggingface-username/your-repo-name" with your actual model ID
adapter_model_id = "[your-huggingface-username]/[your-repo-name]" 
model = PeftModel.from_pretrained(model, adapter_model_id)
model.eval() # Set model to evaluation mode

# Example inference for Q&A
# Input: "What is the Americans with Disabilities Act (ADA)?"
# Expected Output: A concise explanation of the ADA.
input_text_qa = "question: What is the Americans with Disabilities Act (ADA)?"
input_ids_qa = tokenizer(input_text_qa, return_tensors="pt").input_ids

with torch.no_grad():
    outputs_qa = model.generate(input_ids_qa, max_new_tokens=100, num_beams=5, early_stopping=True)

decoded_output_qa = tokenizer.decode(outputs_qa[0], skip_special_tokens=True)
print(f"Input (Q&A): {input_text_qa}")
print(f"Output (Q&A): {decoded_output_qa}")

# Example inference for Bias Mitigation/Instruction Following
# Input: "Rewrite the following sentence to remove any ableist language: 'He was confined to a wheelchair.'"
# Expected Output: "He used a wheelchair." or similar respectful phrasing.
input_text_bias = "instruction: Rewrite the following sentence to remove any ableist language: 'He was confined to a wheelchair.'"
input_ids_bias = tokenizer(input_text_bias, return_tensors="pt").input_ids

with torch.no_grad():
    outputs_bias = model.generate(input_ids_bias, max_new_tokens=50, num_beams=5, early_stopping=True)

decoded_output_bias = tokenizer.decode(outputs_bias[0], skip_special_tokens=True)
print(f"Input (Bias Mitigation): {input_text_bias}")
print(f"Output (Bias Mitigation): {decoded_output_bias}")