Glonce

DeltaTTG trains multi-turn agents with a Token-to-Go progress meter, beating GRPO

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact42Novelty58Relevance100Surprise28Players8Credibility74

Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents usually relies on sparse terminal rewards, which say little about individual steps. The paper behind DeltaTTG points to the drawbacks of existing fixes. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes. Direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics.

DeltaTTG instead learns a reusable progress signal through a shared progress meter. The meter is supervised cheaply, using recorded trajectories from diverse policies, and needs no task-specific progress rules. Its training target is Token-to-Go (TTG): the number of tokens remaining to be generated until the task is complete, which the authors treat as an observable proxy for the remaining effort.

During actor training, the meter is frozen. Changes in its estimates from one step to the next redistribute the outcome-derived advantages across the steps of a trajectory. The reward itself is not altered, and each trajectory keeps its advantage sign and its token-average scale.

On ALFWorld, PaperSearchQA and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 points with Qwen3-4B and 7.2 points with Llama-3.1-8B-Instruct. It outperforms all compared RL baselines on average, and actor-training GPU-hours stay close to GRPO. Further experiments show improved sample efficiency and out-of-domain performance, and support the meter's ability to distinguish step quality and to guide test-time action selection across actor sizes. The code is available on GitHub at Xuyan923r/DeltaTTG.

Key facts

  • DeltaTTG trains a shared progress meter to predict Token-to-Go, the number of tokens remaining until a task is complete, from recorded trajectories of diverse policies.
  • During actor training the frozen meter's estimate changes redistribute outcome-derived advantages across steps, without altering the reward or each trajectory's advantage sign and token-average scale.
  • Average score over GRPO improves by 5.7 points with Qwen3-4B and 7.2 points with Llama-3.1-8B-Instruct across ALFWorld, PaperSearchQA and BFCLv3.
  • It outperforms all compared RL baselines on average while actor-training GPU-hours stay close to GRPO.
  • Code is available on GitHub.

Why it matters

Multi-turn agents get a single outcome reward at the end of a long sequence of steps, so it is hard to tell which steps helped. DeltaTTG tries to solve this credit assignment problem without per-step human labels, external evaluators or task-specific rules. It does so by turning remaining token count into a progress signal that can be learned from trajectories that already exist.

Who it affects

Researchers and engineers who train tool-using or interactive agents with RLVR and GRPO-style methods. The reported setups use Qwen3-4B and Llama-3.1-8B-Instruct as actors, and evaluation covers ALFWorld, PaperSearchQA and BFCLv3.

How to use it

The code is published on GitHub at Xuyan923r/DeltaTTG. The method has two stages as described: first train the progress meter on recorded trajectories from diverse policies, then freeze it and use its estimate changes to reweight advantages while training the actor.

How solid is it

The evidence is the paper's own abstract-level results: gains of 5.7 and 7.2 points in average score over GRPO on three benchmarks and two actor models, plus better results than all compared RL baselines on average. The 5.7 and 7.2 figures are improvements in average score in points, not percentages. Further experiments on sample efficiency, out-of-domain performance and test-time action selection are reported as supportive. The compared baselines are not named in the source text.

Risks and caveats

The headline numbers are averages across three benchmarks, so individual benchmark gains may differ. The tested actors are small to mid-sized models (Qwen3-4B and Llama-3.1-8B-Instruct), and the results say nothing here about larger models beyond the claim of guiding action selection across actor sizes. Training cost is described only as GPU-hours close to GRPO, without a figure. The method also needs a separate meter-training stage on recorded trajectories.