On-Policy Reverse Distillation lets student models outgrow weak teachers

Weak-to-strong generalization studies whether a stronger AI model can learn from a weaker supervisor and end up surpassing it, rather than being pulled down to the supervisor's level. Youngrok Park and co-authors argue this matters most for successive generations of a model line and for consolidating skills learned across different domains, situations where retraining a frontier-scale model from scratch on every update would be prohibitively expensive. The problem, they write, is that conventional distillation treats the weak teacher's own output as the target the student is optimized toward, which can impose the teacher's capacity ceiling on the student instead of letting the student exceed it.

The paper proposes On-Policy Reverse Distillation (OPRD) as an alternative. Instead of training the student to match the teacher's outputs directly, OPRD measures how the teacher's policy shifts relative to its own reference policy, evaluated on rollouts generated by the student itself, then amplifies the part of the student's own verifier-driven policy gradient that points in the same direction as that shift. Only updates already backed by the verifier get rescaled this way, which the authors say lets OPRD preserve the stationary points of policy optimization, the points training would converge to anyway, while still accelerating learning beyond what the teacher alone could reach.

Tested on successive model transfer, where a new model generation learns from the previous one, and on multi-teacher distillation, where a student draws on more than one teacher, OPRD reaches higher performance with fewer student training updates than existing reinforcement learning and distillation approaches, according to the paper. A response-style analysis finds that the resulting OPRD students end up closer in style to models trained with verifier-based reinforcement learning alone than to their own weak teachers, which the authors read as evidence that teacher guidance accelerates the student's own optimization process rather than redirecting it toward the teacher's habits.

The paper also runs the conventional case, a strong teacher guiding a weaker student, and reports that OPRD still combines verifier-driven policy optimization with teacher guidance successfully there too, regardless of which side, teacher or student, has more capacity.

Key facts

On-Policy Reverse Distillation (OPRD) lets a stronger student model learn from a weaker teacher without inheriting the teacher's own performance ceiling.
OPRD evaluates the teacher's policy shift relative to its reference policy on the student's own rollouts, then amplifies the aligned part of the student's verifier-driven policy gradient, rescaling only verifier-supported updates.
In successive model transfer and multi-teacher distillation, OPRD reaches higher performance with fewer student updates than existing RL and distillation approaches, according to the paper.
A response-style analysis shows OPRD students end up closer to models trained with verifier-based RL alone than to their weak teachers, suggesting the teacher speeds up rather than redirects the student's own optimization.
OPRD also works in the conventional strong-to-weak distillation setup, combining verifier-driven optimization with teacher guidance regardless of which side has more capacity.

Why it matters

Post-training a frontier-scale model from scratch for every new generation, or to fold in skills learned by separate specialist models, is expensive enough that repeating it often is impractical. Conventional distillation does not solve this cleanly: it trains the student to match the weak teacher's own outputs, which can drag the student down to the teacher's capacity ceiling instead of letting it improve on the teacher. OPRD keeps the speed advantage of learning from a teacher, fewer training updates than training the student from scratch, without inheriting the teacher's own limits, because it uses the teacher's policy shift only as a signal to amplify the student's own verifier-driven learning, rather than as a target to copy.

Who it affects

The method targets teams building successive generations of a model line, where a new model is meant to build on the previous one without repeating full post-training from scratch, and teams doing multi-teacher distillation, where one student draws on more than one existing model. It also applies more broadly to anyone running verifier-based reinforcement learning or RL-style post-training, since OPRD works by reweighting the student's own verifier-driven policy gradient rather than replacing it with the teacher's outputs. The text names no institutions, affiliations or specific model families, so this reads as a general method contribution rather than a result reported for one lab's particular models.

How to use it

OPRD is described as a training-time technique for post-training pipelines that already use a verifier, the same kind of component verifier-based RL setups already rely on, rather than a hosted product or API. It only rescales updates the verifier already supports, so it plugs into an existing verifier-driven RL loop instead of replacing it. The paper does not mention code or a model release, so at this point it reads as a method described in the paper rather than something available to run directly.

How solid is it

The paper reports that OPRD outperforms existing RL and distillation approaches on two setups, successive model transfer and multi-teacher distillation, reaching higher performance with fewer student updates, and that it also works in the reverse, conventional strong-to-weak distillation case. The text gives no numeric results, no percentage or point improvements, and no named benchmarks, datasets or baseline methods, so the size of the advantage and exactly what it was measured against are not verifiable from this text alone. The response-style analysis, showing OPRD students resemble RL-only trained models more than their weak teachers, is a qualitative comparison rather than a quantified one.

Risks and caveats

The paper does not define what threshold separates a weak model from a strong one, so how far apart teacher and student need to be for OPRD to help is unclear from this text. No publication or submission date is given, so it is hard to tell from this text alone how recent the work is. With no code or model release mentioned, the method cannot yet be reproduced or tried directly, so it is worth treating this as an early report of a training technique rather than a ready-to-use tool.

“OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers”

— the paper