whisper-large-v3-turbo-encoder-pruned

A pruned and distilled variant of openai/whisper-large-v3-turbo with 6 encoder layers removed (layers 5, 6, 7, 9, 11, 12) and recovered via label-free knowledge distillation.

Paper: Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Code: https://github.com/rasgaard/whisper-encoder-layer-prune

What is this?

The 32-layer Whisper encoder contains redundant middle layers. This model removes the 6 least important layers (identified via leave-one-out ΔWER ranking), then re-aligns the pruned encoder to the original using MSE distillation on unlabelled speech — no transcriptions required.

The result is a drop-in replacement: same architecture family, same tokenizer, same decoder — just a shallower encoder.

Performance

Language	Baseline	Zero-shot pruned	After distillation
Danish	23.9%	32.1% (+8.2 pp)	27.3% (+3.4 pp)
English	15.4%	16.6% (+1.2 pp)	16.1% (+0.7 pp)
German	17.1%	18.3% (+1.2 pp)	18.1% (+1.0 pp)
French	16.3%	20.7% (+4.4 pp)	18.7% (+2.5 pp)
Mean		+3.8 pp	+1.9 pp

Evaluated on FLEURS test splits. Parentheses show absolute change in percentage points.

Efficiency

Metric	Full model	This model
Encoder layers	32	26
Encoder parameters	637M	519M (−18.5%)
Model size (bf16)	1543 MB	1318 MB (−225 MB)
Encoder speedup	1.00×	1.22×

Usage

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "rasgaard/whisper-large-v3-turbo-encoder-pruned",
    torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(
    "rasgaard/whisper-large-v3-turbo-encoder-pruned"
)

Distillation details

• Teacher: openai/whisper-large-v3-turbo (frozen)
• Objective: MSE on final encoder hidden states
• Data: People's Speech validation split (~18k English utterances, unlabelled)
• Training: 2000 steps, lr=1e-5, AdamW, batch=8, ~26 min on A100
• Best checkpoint: step 1500

@misc{aagaard2026layerslessencoderpruning,
      title={Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery}, 
      author={Rasmus Aagaard and Nicki Skafte Detlefsen},
      year={2026},
      eprint={2609.27980},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.27980}, 
}