Glonce

AgentOPSD sharpens credit assignment in agentic RL

researchThe Glonce editorsHugging Face🔥55Glonce rating55Impact60Novelty75Relevance95Surprise50Players30Credibility82
AgentOPSD sharpens credit assignment in agentic RL

The authors propose AgentOPSD, a critic-free method for turn-level credit assignment in agentic reinforcement learning. Standard reinforcement learning with verifiable rewards builds advantage estimates at the level of the whole trajectory, but in long, multi-turn agentic tasks this often fails to credit the specific decisions that actually determined the outcome. Recent work introduced privileged self-distillation to provide denser, per-step supervision, but how that local signal should represent credit across a sequence of turns has remained unclear.

AgentOPSD addresses this by aggregating token-level log-probability gaps between a teacher and a student model into turn-level evidence, then recursively updating a Bayesian belief state in log-odds space. The result is a reweighting scheme that converts sparse, outcome-only supervision into turn-level credit signals, and identifies the pivotal turns in a trajectory by tracking how much each one shifts the belief state relative to the one before it. The method plugs directly into standard policy optimization: it needs neither an added critic network nor extra rollouts beyond what standard training already performs.

The authors test AgentOPSD on three agentic benchmarks, ALFWorld, WebShop, and Search-QA, using Qwen2.5 models at two parameter scales, 3B and 7B. AgentOPSD outperforms both GRPO and strong self-distillation baselines, reaching 89.1% success on ALFWorld with the 7B model. Ablation studies trace the gains to two specific design choices: aggregating evidence at the turn level rather than the token level, and using history-dependent recursive belief updates rather than treating each turn independently.

Key facts

  • AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.
  • It aggregates token-level teacher-student log-probability gaps into turn-level evidence via a recursive Bayesian belief update in log-odds space.
  • It requires no additional critic network and no extra rollouts beyond standard policy optimization.
  • Tested with Qwen2.5 models at 3B and 7B scales on ALFWorld, WebShop, and Search-QA, it reaches 89.1% success on ALFWorld with the 7B model, beating GRPO and self-distillation baselines.
  • Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

Why it matters

Reinforcement learning with verifiable rewards typically scores an entire trajectory at once, so in long, multi-turn agentic tasks it struggles to identify which specific turns actually decided the outcome. A single early misstep can sink an otherwise good trajectory, yet the training signal does not say where the mistake happened. AgentOPSD converts that sparse, whole-trajectory reward into turn-level credit without adding a critic network or extra rollouts, which is the kind of denser, more targeted supervision that long-horizon agent training has been missing.

Who it affects

The method targets researchers and engineers who train agentic systems with reinforcement learning, particularly those using policy optimization approaches like GRPO in multi-turn settings such as embodied tasks, web navigation, or tool use. The evaluation uses Qwen2.5 models, so it is directly relevant to teams already building on that model family.

How to use it

AgentOPSD is built to be fully compatible with standard policy optimization, so it functions as an addition to an existing agentic RL pipeline rather than a separate training paradigm, and it does not require extra rollouts or a new critic component.

How solid is it

The method is evaluated on three established agentic benchmarks, ALFWorld, WebShop, and Search-QA, using Qwen2.5 models at two scales, and is compared against GRPO and strong self-distillation baselines, with ablation studies isolating the contribution of turn-level aggregation and recursive belief updates. The abstract does not state a publication venue, conference, or timeframe, and it reports a numeric success rate only for ALFWorld; no results are given for WebShop or Search-QA.

Risks and caveats

The abstract names no authors or institutions and gives no comparison numbers, such as GRPO's own success rate on ALFWorld, only that AgentOPSD outperforms it. Without baseline figures or results on the other two benchmarks, the size of the improvement cannot be independently checked from the abstract alone.