khotveer1/gpt2-medium-lora-finetuned-shakespeare

Model Card for Model ID

Model Details

Model Description

This model is a fine-tuned version of gpt2-medium using LoRA (Low-Rank Adaptation) on a cleaned corpus of Shakespeare's plays. It demonstrates how to efficiently adapt large-scale pretrained models using parameter-efficient techniques like LoRA.

• Developed by: Veer Khot
• Funded by [optional]: Self-funded project
• Shared by [optional]: Veer Khot (@khotveer1)
• Model type: Autoregressive Transformer-based Language Model
• Language(s) (NLP): English 🇬🇧
• License: MIT License
• Finetuned from model [optional]: gpt2-medium

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: N/A
• Demo [optional]: [More Information Needed]

Uses

Direct Use

This model can be used for:

• Generating Shakespearean-style dialogue or monologue
• Educational demos of LoRA fine-tuning on LLMs
• Creative writing and style transfer experiments

Downstream Use [optional]

• Fine-tuning for historical language modeling
• Style-specific chatbot backends
• Integration in educational or theatrical NLP tools

Out-of-Scope Use

• Factual question-answering or real-world information retrieval
• Sensitive or high-stakes decision-making applications

Bias, Risks, and Limitations

This model was trained on a literary corpus from the 16th century. The generated content may reflect historical biases, archaic language, and lack relevance to modern context. It should not be used in contexts requiring factual accuracy or up-to-date language norms.

Recommendations

Users should be made aware of:

• Stylized outputs that mimic Shakespearean writing
• Inapplicability in fact-sensitive domains
• Model can hallucinate or produce verbose content

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForCausalLM
  
tokenizer = AutoTokenizer.from_pretrained("khotveer1/gpt2-medium-lora-finetuned-shakespeare")
model = AutoModelForCausalLM.from_pretrained("khotveer1/gpt2-medium-lora-finetuned-shakespeare")

prompt = "ACT I, SCENE I\nfirst citizen:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=80,
                                    do_sample=True,
                                    top_k=40,
                                    temperature=0.80,
                                    repetition_penalty=1.15,
                            )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

A cleaned dataset of Shakespeare's plays from Project Gutenberg (preprocessed for modern training).

Training Procedure

LoRA-based fine-tuning using Hugging Face PEFT and transformers on gpt2-medium. Adapters merged before uploading to Hugging Face.

Preprocessing [optional]

Cleaned the text (removed stage directions, modernized contractions) Tokenized with GPT-2 tokenizer

Training Hyperparameters

• Training regime:
• Epochs: 3
• Batch Size: 4
• Learning Rate: 2e-5
• LoRA Rank: 8
• LoRA Alpha: 16
• Target Modules: c_attn, c_proj
• Optimizer: AdamW

Speeds, Sizes, Times [optional]

• GPU: A100 (1x)
• Training Time: ~40 minutes
• Model size: ~355M parameters

Evaluation

Testing Data, Factors & Metrics

Testing Data

Held-out subset of Shakespeare plays not seen during training.

Factors:

• Perplexity
• Loss reduction over epochs

Metrics:

• Final training loss: ~0.68
• Validation perplexity: ~15.2

Results

The model reliably generates Shakespeare-style prose and rhyme, showing clear stylistic adaptation.

Technical Specifications [optional]

• transformers==4.40.0
• peft==0.14.0
• datasets==2.19.0
• accelerate==0.30.0

Glossary [optional]

• LoRA: A method for fine-tuning models by injecting trainable low-rank matrices into attention layers.
• GPT-2 Medium: 24-layer, 345M parameter transformer decoder for autoregressive text generation.

Model Card Authors [optional]

Veer Khot

GitHub

Medium

LinkedIn

Framework versions

• PEFT 0.14.0