CriticOPD has teachers critique student errors to fix on-policy distillation
On-policy distillation (OPD) trains a student model on its own rollouts, with token-level supervision from a teacher. The method assumes that the teacher can reach a correct answer from any prefix the student writes. The paper calls this property the recoverability of the teacher, and shows that it fails on incorrect rollouts: the teacher follows the student's errors, although it recognizes them.
The proposed fix is CriticOPD, which uses that error recognition to restore recoverability. When a student rollout is incorrect, the teacher critiques it. The critique lets the student rewrite its solution with the error in view. The teacher then scores the resulting output, with the critique included as privileged information. For training, the student uses the revision without the critique.
The authors test CriticOPD on two Qwen3 students and six mathematical reasoning benchmarks. They report that it improves OPD, with the largest gains on competition problems. The abstract gives no figures for the size of the improvement.
Key facts
- On-policy distillation assumes the teacher can reach a correct answer from any prefix the student writes; the paper calls this the recoverability of the teacher.
- The authors show recoverability fails on incorrect rollouts: the teacher follows student errors even though it recognizes them.
- CriticOPD has the teacher critique an incorrect rollout, the student rewrite its solution, and the teacher score the output with the critique as privileged information.
- The student trains on the revision without the critique.
- Evaluated on two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Why it matters
OPD gives a student dense, token-level feedback on its own outputs, which only works if the teacher steers well from wherever the student has wandered. The paper argues this breaks exactly where it counts: on incorrect rollouts, the teacher goes along with the student's mistakes even though it can recognize them. CriticOPD turns that recognition into a usable training signal instead of leaving it unused.
Who it affects
The work is aimed at people training smaller student models from a stronger teacher with on-policy distillation, particularly for mathematical reasoning. The evaluation covers two Qwen3 students and six mathematical reasoning benchmarks, so the evidence applies most directly to that setting.
How to use it
The recipe as described has three steps. The teacher critiques an incorrect rollout, the student rewrites its solution following the error, and the teacher scores the output with the critique as privileged information. The student then trains on the revision without the critique. The abstract mentions no code, model or data release.
How solid is it
This is a preprint abstract with a stated empirical result: CriticOPD improves OPD on two Qwen3 students and six mathematical reasoning benchmarks, with the largest gains on competition problems. No numeric size of the improvement is given, the Qwen3 model sizes and the teacher are not named, and the six benchmarks are not listed. No comparison with methods other than OPD is given.
Risks and caveats
Without accuracy figures, baselines or named benchmarks, the size and robustness of the gains cannot be judged from the abstract alone. The tested setting is mathematical reasoning with Qwen3 students, and the abstract does not say whether the approach carries over to other domains or model families. Training cost and compute are not reported.
“the teacher follows student errors, although it recognizes them.”
— CriticOPD paper abstract