Whisper encoder pruning removes six layers, recovers accuracy label-free
Researchers describe a method for shrinking the encoder of OpenAI's Whisper speech recognition model, a part of the architecture that has resisted compression even as decoder pruning became common. They note that whisper-large-v3-turbo already cuts its decoder from 32 layers down to 4, and Distill-Whisper goes further, to just 2 decoder layers, both delivering real transcription speedups. The encoder has seen less success: prior attempts required custom inference code to exploit a compressed model, which the authors suggest is why none caught on.
Their approach instead ranks each encoder layer by the leave-one-layer-out change in Word Error Rate (WER), that is, how much accuracy drops when that specific layer is removed. The six layers causing the least change are then deleted outright, removing 18.5% of the encoder stack. Because the result is simply a shallower encoder with fewer layers, it runs with no custom inference code, unlike earlier encoder-compression schemes.
Removing those layers zero-shot pushes mean WER across four languages from an 18.2% baseline up to 21.9%. To recover the loss, the authors distill the pruned model using unlabeled monolingual speech data, meaning no transcription labels are needed for this recovery step. After distillation, mean WER falls back to 20.1%, still above the original baseline but well below the zero-shot pruned figure. The authors have released their code and the pruned whisper-large-v3-turbo-encoder-pruned model publicly on GitHub and Hugging Face.
Key facts
- Encoder layers are ranked by the leave-one-layer-out change in WER, and the six layers causing the least change are pruned, removing 18.5% of the encoder stack
- The pruned model needs no custom inference code, since it is just a shallower encoder, unlike earlier encoder-compression approaches
- Zero-shot pruning raises mean WER across four languages from an 18.2% baseline to 21.9%; label-free distillation on unlabeled monolingual speech recovers it to 20.1%
- For comparison, existing decoder pruning already cuts whisper-large-v3-turbo's decoder from 32 to 4 layers, and Distill-Whisper's to 2 layers
- Code and the pruned whisper-large-v3-turbo-encoder-pruned model are released on GitHub and Hugging Face
Why it matters
Decoder pruning of Whisper is already widely used and delivers real speedups, but encoder pruning has not caught on, reportedly because earlier methods needed custom inference code to benefit from a compressed encoder. This approach produces a plain shallower encoder that drops into standard inference pipelines, removing that adoption barrier while still cutting 18.5% of the encoder stack.
Who it affects
It is aimed at engineers and researchers running Whisper-based transcription who want a smaller, cheaper encoder without rewriting their inference code, and who are willing to accept a WER trade-off relative to the unpruned model.
How to use it
The authors have published the code for the pruning and distillation procedure and the resulting pruned model, whisper-large-v3-turbo-encoder-pruned, as public releases on GitHub and Hugging Face, so it can be used as a drop-in encoder replacement.
How solid is it
The account rests on a single paper-style source that gives concrete before-and-after WER figures (18.2% baseline, 21.9% zero-shot after pruning, 20.1% after distillation) and links to working code and model repositories. It does not name the authors or their affiliation, give a publication date, or specify which four languages were evaluated.
Risks and caveats
Even after label-free distillation, mean WER across the four evaluated languages remains higher than the unpruned baseline (20.1% versus 18.2%), so some accuracy is permanently lost. The source gives no measurement of inference speed or latency gains from the encoder pruning itself, only citing decoder-pruning speedups from other models for context, and it does not specify how much unlabeled speech data the distillation step used.