Glonce

RetireOPD teaches agentic RL models to fire their own teacher

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact62Novelty72Relevance98Surprise48Players58Credibility78
RetireOPD teaches agentic RL models to fire their own teacher

Researchers describe RetireOPD (Self-Retiring On-Policy Distillation), a training recipe for multi-turn AI agents that fixes a weakness in an existing technique called self on-policy distillation (OPD). Agents trained with plain reinforcement learning (RL) get only a single scalar reward at the end of a whole trajectory, which is a weak training signal. OPD tries to fix this by having a self-teacher, a version of the model with privileged access to task skills, hand the student model dense, token-by-token guidance instead. The authors find two problems with using OPD this way on agentic tasks: giving a teacher privileged information does not automatically make it a reliable source of guidance, and the value of that guidance changes depending on the stage of training. RetireOPD addresses both. It first trains a separate, skill-conditioned teacher model using environment rewards, then trains a skill-free student jointly with both RL and OPD. Instead of following a fixed schedule for how long to use the teacher, the method uses what the authors call Adaptive Retirement: the student itself decides to stop learning from the teacher once two conditions are met, the gap between student and teacher performance stops shrinking, and the student has reached a set fraction of the teacher's success rate. After that point, training continues with RL alone. Tested on Qwen2.5 models ranging from 1.5B to 7B parameters, RetireOPD raised ALFWorld task success rate over the plain RL baseline by 14.1 to 18.8 percentage points, and WebShop accuracy by 11.8 to 19.0 percentage points, depending on model size. The authors also report that the trained student ends up outperforming its own skill-conditioned teacher in every tested setting.

Key facts

  • RetireOPD trains a skill-free student agent jointly with RL and on-policy distillation from a separate, skill-conditioned teacher model.
  • The student decides on its own when to stop learning from the teacher (Adaptive Retirement), once the student-teacher performance gap stops shrinking and the student hits a target fraction of the teacher's success rate.
  • On Qwen2.5 models from 1.5B to 7B parameters, RetireOPD improves ALFWorld success rate over an RL-only baseline by 14.1 to 18.8 percentage points.
  • The same method improves WebShop accuracy by 11.8 to 19.0 percentage points over the RL-only baseline.
  • The trained student model surpasses its own skill-conditioned teacher in every tested setting.

Why it matters

Multi-turn agentic RL is notoriously hard to train because the only feedback an agent gets is one reward number at the very end of a long sequence of actions, a signal too sparse to say which specific step went wrong. Distilling knowledge from a more capable teacher is one way to densify that signal, but the paper's own finding, that a privileged teacher is not automatically a reliable one and that its usefulness changes over the course of training, is itself a useful caution for anyone building agent training pipelines this way. RetireOPD's answer, having the student decide for itself when the teacher has stopped helping, is a fix aimed squarely at that instability.

Who it affects

The work is aimed at researchers and engineers building reinforcement-learning agents for multi-step, tool-using tasks, the kind represented here by ALFWorld (embodied household tasks) and WebShop (web shopping tasks). It is also relevant to anyone using teacher-student distillation setups for agents built on open Qwen2.5 models, since that is the model family the results are reported on.

How to use it

The source is a research paper describing a training method and its results, not a released tool, product or paid service; the source text names no license, code release or pricing terms, so none should be assumed.

How solid is it

The reported gains are consistent across a fairly wide model-size range, Qwen2.5 from 1.5B to 7B parameters, on two different benchmark environments (ALFWorld and WebShop), and the method is also reported to beat its own teacher model in every tested configuration. The source text does not specify what numerical fraction of the teacher's success rate triggers retirement, nor does it compare RetireOPD against other distillation methods beyond the RL-only baseline and its own skill-conditioned teacher, which limits how the results can be judged against the broader field.

Risks and caveats

The source text gives no author names, institutional affiliations, publication date, training compute or dataset size, so independent verification of the claims is limited to what is stated in the abstract itself. The reported improvements are measured only against an RL-only baseline and the paper's own teacher model, not against other published distillation techniques, so it is unclear how RetireOPD compares to the wider landscape of agentic RL training methods.