Order-Only On-Policy Distillation trains students on teacher token rankings
On-policy distillation (OPD) trains a student model on its own generations by matching the teacher's token probabilities. The paper argues that those probabilities mix two kinds of information: which tokens the teacher prefers, and how confident it is. The confidence reflects the teacher's own capacity and calibration, which a smaller student may be unable to reproduce. There is a second problem: the teacher's temperature changes the probability values without changing the ordering of tokens, so OPD is also sensitive to the temperature choice.
The authors ask whether the teacher's ordering of candidate tokens is enough, and propose Order-Only On-Policy Distillation (O^{3}PD). At each prefix the student generates, O^{3}PD trains the student to reproduce the order of the teacher's top-K tokens, without fitting their probability values. Because the ordering does not depend on temperature, the objective is invariant to the teacher temperature and needs no temperature tuning.
Across six teacher-student pairs, O^{3}PD outperforms OPD on the average over HMMT25, AIME24 and AIME25 in every pair, by up to 2.9 points. On AIME25 alone, the gains reach up to 5.6 points. Both figures are best cases across pairs.
The authors also report that other ranking losses yield similar gains, so in their reading the benefit comes from ordinal supervision itself, not from one particular loss. Their analysis finds that O^{3}PD departs further from the teacher's probabilities than OPD does, yet reproduces the teacher's ranking more faithfully and yields a stronger student. They conclude that the ranking, not the probability values, carries the knowledge that on-policy distillation needs to transfer.
Key facts
- O^{3}PD trains the student to reproduce only the order of the teacher's top-K tokens at each student-generated prefix, with no fitting of probability values.
- The objective is invariant to teacher temperature, so no temperature tuning is needed; standard OPD is sensitive to that choice.
- In all six teacher-student pairs, O^{3}PD beats OPD on the average over HMMT25, AIME24 and AIME25, by up to 2.9 points.
- On AIME25 alone the gain over OPD reaches up to 5.6 points.
- Other ranking losses give similar gains, which the authors take as evidence that ordinal supervision itself is what helps.
Why it matters
Distillation compresses a large teacher model into a smaller student, and OPD does this on the student's own generations. This paper challenges what needs to be copied. Its claim is that the teacher's confidence is something a smaller student may be unable to reproduce, and that the ranking of tokens is the part worth transferring. If that holds, a simpler target removes a tuning knob: the teacher temperature.
Who it affects
Mainly researchers and engineers who distil large reasoning models into smaller ones using on-policy methods, and who currently have to choose a teacher temperature for OPD. The evaluation covers math benchmarks (HMMT25, AIME24 and AIME25).
How to use it
The idea is a change of training objective: at each prefix the student generates, replace probability matching against the teacher with a loss that makes the student reproduce the order of the teacher's top-K tokens. The authors say the objective needs no temperature tuning, and that other ranking losses give similar gains, so the exact choice of ranking loss may matter less than using ordinal supervision. No code release is mentioned in the source.
How solid is it
The claims rest on the authors' own reported results across six teacher-student pairs, and O^{3}PD wins on the three-benchmark average in every pair. The headline numbers are maxima, though: up to 2.9 points on the average and up to 5.6 points on AIME25. Typical or mean gains are not stated, and it is not stated whether the two maxima come from the same pair. The pairs, model sizes and the value of K are not given in the text available here.
Risks and caveats
Evidence is limited to math benchmarks (HMMT25, AIME24 and AIME25); no evaluation outside them is mentioned. Baseline OPD scores and absolute O^{3}PD scores are not given, so the size of the gains relative to overall performance is unclear. The text does not say which other ranking losses were tested. The authors' conclusion that ranking carries the transferable knowledge is their interpretation of their own analysis.
“The ranking, not the probability values, thus carries the knowledge that on-policy distillation needs to transfer.”
— From the paper's abstract