Glonce

ActObs improves RL training for Qwen3 coding agents by also predicting observations

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact60Novelty70Relevance100Surprise50Players25Credibility82
ActObs improves RL training for Qwen3 coding agents by also predicting observations

A paper titled "Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL" questions a default in how agents are trained. Standard supervised fine-tuning (SFT) computes loss only on the tokens an agent generates for its actions; the environment's observations sit in the trajectory as context but are never a prediction target. The authors introduce ActObs, which also supervises those observation tokens. A deployed agent still never generates observations, so nothing changes at inference time, but learning to predict them during training pushes the policy to model the consequences of its actions, at no extra cost in data, parameters, sequence length or forward passes.

After plain SFT, ActObs and the action-only baseline perform similarly; the difference appears once GRPO reinforcement learning is applied on top. On Qwen3-4B, GRPO started from ActObs beats the action-only counterpart's pass@k at every sampling budget tested on Terminal-Bench 2.0, though the paper does not give a single numeric gap for this comparison. On Qwen3-8B the pattern shifts: ActObs gives up some pass@1 reliability in exchange for a pass@k gain of 3.4 percentage points at pass@16, and it solves more distinct tasks overall than the action-only version.

The advantage carries over to a benchmark the models were never trained on. On aider-polyglot, a cross-domain code-editing task set unseen during both SFT and RL, the 4B ActObs model gains 4.2 percentage points at pass@1 over the baseline.

The authors trace the mechanism to what happens during SFT itself. Under action-only training, gradients from action tokens and gradients from observation tokens rapidly become orthogonal, leaving a large residual observation gradient that actually degrades the model's ability to predict the environment below what the base model could already do. Supervising both token types together prevents that one-sided specialization: it preserves the model's ability to predict consequences and keeps more entropy during RL, so the final ActObs policy needs less movement from its SFT starting point to reach a given level of performance.

Key facts

  • ActObs extends standard SFT by also applying the training loss to environment-observation tokens already present in agent trajectories, not only to action tokens.
  • On Qwen3-8B after GRPO, ActObs trades some pass@1 reliability for a pass@k gain of 3.4 percentage points at pass@16 on Terminal-Bench 2.0, and solves more distinct tasks than the action-only baseline.
  • On Qwen3-4B, GRPO from ActObs beats the action-only baseline's pass@k at every evaluated sampling budget on Terminal-Bench 2.0.
  • On aider-polyglot, a cross-domain code-editing benchmark unseen during SFT and RL, the 4B ActObs model gains 4.2 percentage points at pass@1.
  • The authors attribute the effect to SFT: action-only training makes action and observation gradients become orthogonal, leaving a residual observation gradient that degrades environment prediction; joint supervision on both token types prevents this.

Why it matters

Most agent training pipelines throw away a training signal that is already sitting in the data: the environment's own responses to what the agent did. This paper argues that ignoring those observation tokens during SFT leaves the policy worse prepared for the reinforcement learning stage that typically follows, and shows that supervising them changes how the resulting policy explores once GRPO is applied, without adding any data, parameters, tokens or compute passes.

Who it affects

Researchers and teams building RL-trained tool-use or coding agents on an SFT-then-GRPO pipeline, in this case on Qwen3-4B and Qwen3-8B. The result speaks most directly to anyone tuning agents for terminal or code-editing tasks, where Terminal-Bench 2.0 and aider-polyglot are used as evaluation benchmarks.

How to use it

ActObs is a change to the training objective, not a product or a released model. It requires no new data collection: the observation tokens it supervises already exist in any recorded agent trajectory, so the change is to the loss computation during SFT, at no added cost in parameters, sequence length or forward passes before the usual GRPO stage runs.

How solid is it

The method is tested on two model sizes (Qwen3-4B and Qwen3-8B) and two benchmarks, including one, aider-polyglot, that is genuinely out of domain for both the SFT and RL data. The authors back the result with a mechanistic analysis of gradient orthogonality and entropy retention during RL, rather than resting on benchmark numbers alone. The reported gains themselves are modest: single-digit percentage points, and on the 8B model the improvement in pass@k comes with a drop in pass@1 reliability rather than a clean win.

Risks and caveats

The paper does not report a specific numeric gap for the Qwen3-4B Terminal-Bench 2.0 comparison, only that pass@k is higher at every sampling budget tested. It gives no training compute cost or wall-clock figures, and no detail on the ActObs loss formulation beyond supervising the observation tokens already in each trajectory. The abstract names no individual authors or institutions and carries no publication date. As a single paper posted with modest engagement (16 points, 2 comments), it has not been independently replicated.