Cal-OPD strips teacher noise out of on-policy distillation
On-policy distillation trains a reasoning model (the student) by having it learn from the gap between its own token-level predictions and those of a stronger teacher model, while the student is generating its own on-policy samples. The authors argue that this teacher-student gap is not a clean measurement of how much stronger the teacher actually is. Part of it comes from the teacher's own inconsistencies: the teacher itself deviates from its typical behavior in ways that have nothing to do with the student's capability gap, and standard on-policy distillation learns these teacher-side deviations indiscriminately, mixed in with the useful signal. The authors say the problem gets worse under privileged on-policy distillation, a variant where the teacher is given extra privileged information during training: that privileged information causes larger shifts in the teacher's own likelihoods, which in turn pushes the student to absorb even more of the teacher's self-deviation rather than the actual capability gap. To fix this, the authors introduce Calibrated On-Policy Distillation (Cal-OPD). The method estimates the region of the teacher's self-deviation by applying both positive and negative privileged interventions to the teacher, then calibrates the original teacher-student discrepancy by keeping only the part of it that falls outside that estimated self-deviation region. In experiments on mathematical reasoning benchmarks, Cal-OPD retained only about 52 to 65 percent of the original teacher-student discrepancy as its optimization signal, yet it consistently outperformed standard on-policy distillation and its variants across different model scales.
Key facts
- Standard on-policy distillation conflates two things in the teacher-student discrepancy signal: the real capability gap, and deviations that come from the teacher itself.
- Privileged on-policy distillation makes this worse, since privileged information given to the teacher causes larger teacher-side likelihood shifts that the student ends up learning.
- Cal-OPD estimates the teacher's self-deviation region using positive and negative privileged interventions, then removes that component from the training signal.
- On mathematical reasoning benchmarks, Cal-OPD kept only about 52 to 65 percent of the original discrepancy as its optimization signal.
- Despite using a smaller, calibrated signal, Cal-OPD consistently outperformed standard on-policy distillation and its variants across model scales.
Why it matters
On-policy distillation is a common way to train smaller or cheaper reasoning models from a stronger teacher, by having the student learn from the gap between its own predictions and the teacher's. This paper argues that the gap used as a training signal is noisier than it looks: some of it is the teacher's own inconsistency, not a true measure of the student's shortfall, and a training method that cannot tell the two apart ends up teaching the student some of the teacher's noise instead of its strengths. That problem compounds under privileged on-policy distillation, where giving the teacher extra information shifts the teacher's own outputs and feeds even more of that noise into the student.
Who it affects
The result is aimed at researchers and teams that train reasoning models through distillation from a stronger teacher model, particularly those already using on-policy or privileged on-policy distillation for mathematical or other reasoning tasks.
How to use it
Cal-OPD is presented as a calibration step layered on top of on-policy distillation training. It works by running positive and negative privileged interventions on the teacher to map out the region of the teacher's own self-deviation, then filtering the observed teacher-student discrepancy so only the component outside that region drives the optimization signal. The source does not give implementation details, code availability, or licensing terms.
How solid is it
The claims come from the paper's own experiments on mathematical reasoning benchmarks, where Cal-OPD is reported to consistently beat standard on-policy distillation and its variants across multiple model scales, while relying on only about 52 to 65 percent of the original discrepancy signal. The source text does not name the specific benchmarks, datasets, or model families used, nor does it give absolute accuracy numbers alongside the discrepancy-retention figure.
Risks and caveats
The source text does not include author names, institutional affiliations, publication dates, or a description of how the positive and negative privileged interventions are constructed beyond naming them. The reported gains are the authors' own experimental results, with no independent replication mentioned, and the retained-signal figure of 52 to 65 percent is a range across experiments rather than a single fixed number.