Glonce

Agent-G2 draws RL guidance depth from a Gaussian instead of a fixed number

researchThe Glonce editorsHugging Face🔥69Glonce rating69Impact60Novelty72Relevance92Surprise55Players25Credibility82
Agent-G2 draws RL guidance depth from a Gaussian instead of a fixed number

Hint-based reinforcement learning is a way to fight reward sparsity in long-horizon agentic tasks: before each rollout, the policy is given a prefix of an expert trajectory so it starts exploring from a state closer to success. How well this works depends heavily on the guidance depth, meaning how much of that expert trajectory to keep. Researchers behind a new paper argue that every existing approach treats this depth as a single fixed number per setup. Scheduled methods use one shared value across all training samples and ignore that some tasks need more help than others; per-sample probing estimates a separate depth for each sample, but that costs extra rollouts just to find the right value. The authors report a different finding: the depths that actually help a given task are not clustered at one ideal point, but form a band, and how useful each depth in that band is follows a shape that is approximately Gaussian around the center of the band. Based on this, they propose Agent-G2, a framework that draws the guidance depth for each task from a Gaussian distribution whose center and spread are estimated online, directly from the rollouts already being collected for policy optimization, with no separate probe rollouts and no learned model that predicts depth. The center is set from a global baseline adjusted by how difficult a task's cluster tends to be, and the spread tracks how much variance exists within that cluster. The authors test Agent-G2 on two long-horizon agentic benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B and Qwen2.5-7B-Instruct as the underlying policy models. On ALFWorld, Agent-G2 beats the strongest hint-based, hint-free, and auxiliary-RL baselines by margins of 2.3, 3.9, and 7.4 points respectively, while using under one-third the rollout budget that per-sample probing would need. The text does not give WebShop results as numbers, and it does not spell out what the 'points' metric measures or what scale it is on.

Key facts

  • Hint-based RL keeps a prefix of an expert trajectory before each rollout so a long-horizon agent starts closer to success; existing methods fix the length of that prefix, either as one value shared by all samples or, in per-sample probing, at the cost of extra rollouts to estimate it per sample.
  • The authors find that the depths that help a task form a band whose usefulness is approximately Gaussian around a center point, rather than peaking at one single optimal depth.
  • Agent-G2 samples guidance depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization; the center blends a global baseline with per-cluster difficulty, the spread tracks within-cluster variance, and no extra probe rollouts or learned depth predictor are needed.
  • Tested on ALFWorld and WebShop with Qwen2.5-1.5B and 7B-Instruct backbones, Agent-G2 outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3, 3.9, and 7.4 points respectively.
  • Agent-G2 reaches those ALFWorld gains at under one-third the rollout cost of per-sample probing; the source gives no numeric WebShop results and does not define the 'points' metric or its scale.

Why it matters

Long-horizon agentic tasks suffer from reward sparsity: an agent can wander for many steps without any signal of progress, which makes reinforcement learning slow or unstable. Hint-based RL is one fix, but how much of an expert trajectory to hand the policy has so far been a single number tuned by hand or by expensive per-sample search. Agent-G2's contribution is treating that number as a distribution rather than a point, estimated for free from data the training run is already producing, which removes a manual tuning knob without adding rollout overhead.

Who it affects

Researchers and engineers building reinforcement learning pipelines for long-horizon agentic tasks, such as household-instruction agents (ALFWorld) or web-shopping agents (WebShop), and anyone using hint-based or curriculum-style RL to train agents on top of models like Qwen2.5.

How to use it

The source is a research paper describing a method, not a released product; it gives no code repository, license, or pricing information. Practically, the appeal is that the center and spread of the guidance-depth Gaussian are estimated online from rollouts already gathered for policy optimization, so adopting the approach would not require extra probe rollouts or a separately trained depth predictor.

How solid is it

The evidence comes from one paper's own benchmarks, ALFWorld and WebShop, with two backbone sizes, Qwen2.5-1.5B and 7B-Instruct. The reported gains, 2.3, 3.9, and 7.4 points over the strongest hint-based, hint-free, and Aux-RL baselines, are stated only as margins on ALFWorld; the source text gives no absolute scores and no numeric results for WebShop despite naming it as an evaluation environment, and it does not define what the 'points' metric measures or what scale it runs on. No author names, affiliations, publication venue, or date appear in the text.

Risks and caveats

The claimed Gaussian shape of guidance-depth usefulness and the per-cluster difficulty estimate are specific to the two benchmarks and two model sizes tested here, and the text gives no indication of how the method behaves on other agentic tasks or larger models. Because WebShop results are not given numerically and the 'points' metric is undefined, readers cannot independently judge the size or robustness of the improvement beyond the ALFWorld figures reported.