Glonce

Tracking-Value PPO (TPPO) reports 7.0 and 8.4 point math gains over PPO on Qwen3

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact40Novelty55Relevance100Surprise25Players8Credibility70
Tracking-Value PPO (TPPO) reports 7.0 and 8.4 point math gains over PPO on Qwen3

A paper introduces Tracking-Value PPO, or TPPO, a variant of Proximal Policy Optimization aimed at reinforcement learning with verifiable rewards for language-model reasoning. PPO is widely used for this kind of training, and the authors start from a specific weakness: the way a trajectory's value evolves from token to token characterizes relative token-level credit, yet standard critics often fail to track that evolution when the only reward is a sparse one at the end of the answer.

To model the evolution, the authors analyze the relation between successive prefix values and show that these values admit a Bayesian decomposition with additive updates in log-odds space. They use that additive structure to build TPPO, a cumulative value parameterization: the critic emits one scalar per position, and these scalars are summed into value logits. The position-wise outputs are learned through terminal-outcome supervision on the reconstructed values, so the only training signal is still the final result.

On results, the authors report that TPPO improves aggregate mathematical reasoning scores over PPO by 7.0 points on Qwen3-4B-Base and 8.4 points on Qwen3-8B-Base, while also improving out-of-distribution generalization. They say the gains extend to Qwen3-4B, Qwen2.5-Math-1.5B and Qwen3.5-9B, and to countdown and scientific coding tasks. Further analysis, they write, shows more accurate tracking of value evolution and more reliable advantage estimates. The code is publicly available at github.com/LINs-lab/TPPO.

Key facts

  • TPPO changes the PPO critic into a cumulative value parameterization that sums position-wise scalar outputs into value logits.
  • The design rests on a Bayesian decomposition of successive prefix values, which update additively in log-odds space.
  • Reported gain over PPO in aggregate mathematical reasoning: 7.0 points on Qwen3-4B-Base and 8.4 points on Qwen3-8B-Base.
  • The authors also report better out-of-distribution generalization and gains on Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B, countdown and scientific coding tasks.
  • Code is public at github.com/LINs-lab/TPPO.

Why it matters

Training reasoning models with verifiable rewards usually gives one signal at the end of a long answer, and the critic has to work out which earlier tokens deserved credit. The authors argue that standard critics often fail to track how value evolves along the trajectory under such sparse terminal rewards. TPPO targets exactly that step by giving the critic a structure that follows the additive log-odds updates they derive. If the reported 7.0 and 8.4 point gains over PPO hold up, it is a change to the critic alone, not a new training pipeline.

Who it affects

Researchers and engineers who train language models with PPO on verifiable-reward tasks such as math reasoning, countdown and scientific coding. The reported experiments use Qwen-family models: Qwen3-4B-Base, Qwen3-8B-Base, Qwen3-4B, Qwen2.5-Math-1.5B and Qwen3.5-9B.

How to use it

The code is publicly available at https://github.com/LINs-lab/TPPO, hosted under the LINs-lab GitHub organization. Practitioners already running PPO for reasoning can look there for the cumulative value parameterization, in which per-position scalar outputs are summed into value logits and trained from terminal outcomes alone.

How solid is it

This rests on a paper abstract in which the authors report their own results. The abstract names no authors or institutions (only the GitHub organization LINs-lab appears, in the URL). No absolute benchmark scores or baseline PPO scores are given, only the point differences over PPO. The benchmarks that make up the aggregate mathematical reasoning scores are not named. The size of the gains on the other models, on countdown and on scientific coding, and of the out-of-distribution improvement, is not given either.

Risks and caveats

The headline figures are point differences over PPO, not relative percentages, and without the baseline scores their weight is hard to judge. The abstract states no comparison with methods other than PPO and no compute, training cost or speed comparison with PPO. The claims about more accurate value tracking and more reliable advantage estimates come from the authors' own further analysis.