AventIQ-AI/fill-mask-bert-base-uncased

🧠 FillMask-BERT-FineTuned

A BERT-based masked language model fine-tuned on the wikitext dataset. This model predicts missing words in a sentence using the [MASK] token and provides the most probable replacements with confidence scores. It’s useful for tasks like autocomplete, suggestion engines, or masked word prediction.

---

✨ Model Highlights
📌 Based on bert-base-uncased (by Google)
🔍 Fine-tuned on the wikitext dataset
🧠 Predicts masked tokens using contextual understanding
💾 Available in both full and quantized versions
🚀 Compatible with pipeline('fill-mask') from 🤗 Transformers

---

🧠 Intended Uses

• Autocompletion systems
• Language understanding tasks
• Educational language games or language modeling research

🚫 Limitations

• Trained only on English
• May not handle proper nouns or rare entities well
• Long sentences (>128 tokens) are truncated during training
• Not suitable for generation tasks (e.g., summarization, translation)

---

🏋️‍♂️ Training Details

• Base Model: bert-base-uncased
• Dataset: wikitext-2-raw-v1
• Framework: PyTorch with 🤗 Transformers
• Epochs: 5
• Batch Size: 16
• Max Length: 128 tokens
• Loss: CrossEntropyLoss (Masked LM)
• Optimizer: AdamW
• Device: Trained on NVIDIA CUDA-enabled GPU

---

📊 Evaluation Metrics (manual evaluation)

Metric	Value
Mask Accuracy (Top-1)	~34% on simple examples
Mask Accuracy (Top-5)	~90% on simple examples

This is illustrative. Replace with actual if you log accuracy/f1 during testing.

---

🔤 Tokenizer
The tokenizer is based on the bert-base-uncased vocabulary and saved with the model. Includes:

• tokenizer_config.json
• vocab.txt
• special_tokens_map.json

---

🚀 Usage

from transformers import pipeline

model = "your-username/fill-mask-bert-finetuned"  # replace with actual repo ID
fill_mask = pipeline("fill-mask", model=model)

# Predict the masked word
output = fill_mask("The [MASK] is shining in the sky.")
for o in output:
    print(f"{o['sequence']} | Score: {o['score']:.2f}")

---

⚙️ Quantization

Post-training static quantization was applied using PyTorch to reduce model size and boost inference performance. Quantized version works identically with fill-mask pipeline.

---

📁 Repository Structure

/fill-mask-bert-finetuned/ ├── config.json # Model configuration
├── model.safetensors # Fine-tuned model weights
├── special_tokens_map.json # Token mapping
├── tokenizer_config.json # Tokenizer settings
├── vocab.txt # Tokenizer vocabulary
├── README.md # This model card

---

🙏 Contributing Contributions are welcome! If you have suggestions, improvements, or issues, feel free to open an issue or submit a pull request.