README.md

---
license: mit
language: en
tags:
  - multiple-choice
  - quantization
  - W8A8
  - LLMCompressor
  - bf16
  - int8
model_type: causal-lm
base_model: hssawhney/mnlp-model
pipeline_tag: text-generation
---

# Quantized MCQA Model – W8A8

## Model Summary
This model is a quantized version of our MCQA model. It was quantized using post-training quantization (PTQ), targeting both weights and activations (W8A8) using the [LLMCompressor](https://github.com/vllm-project/llm-compressor) framework.

## Technical Details
- **Base model:** [`hssawhney/mnlp-model`](https://huggingface.co/hssawhney/mnlp-model)
- **Quantization method:** SmoothQuant + GPTQ
- **Precision:** BF16 (activations) + INT8 (weights)
- **Calibration data:** 512 samples from [`zay25/quantization-dataset`](https://huggingface.co/datasets/zay25/quantization-dataset)
- **Excluded layers:** `lm_head` (to preserve output logits)
- **Final model size:** ~717 MB

## Evaluation
The quantized model was evaluated on the full MCQA demo dataset using the LightEval framework. Performance dropped with only a **0.02 decrease in accuracy** compared to the full-precision (FP32) version.

## Intended Use
This model is optimized for **efficient inference** in **multiple-choice question answering** tasks, particularly in the context of **STEM tutoring**. It is well-suited for low-resource deployment environments where latency and memory usage are critical.