Paper finds dense Monte-Carlo rewards lose to plain GRPO at matched compute

RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit. Dense process rewards were proposed to credit intermediate steps instead, for example a Monte-Carlo reward that scores each step by sampling continuations from it. The paper's complaint is that such rewards are usually judged per training update, or vetted with a gradient test on a single batch, and both leave out what the reward costs. In the authors' setting, the continuations alone take 91% of all generated tokens.
To fix that, the authors introduce a matched-compute protocol. It gives every training intervention the same budget, measured in generated tokens and training GPU-hours, and tests it in preregistered studies with controls for side effects. Across 1,668 runs on six models, they report that matching compute reverses the ranking.
The headline example is Qwen2.5-1.5B. There the Monte-Carlo reward leads standard GRPO per update, yet it ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time. None of four dense-reward variants outperforms GRPO, even when given as much training time. A per-token gradient test orders these rewards the same way training does, but it overestimates each by one to two points, which is as much as the gains the test is meant to detect.
The protocol also confirms one idea. Training on the prompts whose predicted success is nearest 1/2 improves accuracy at matched compute: by 1.7 points on Qwen3-0.6B and 0.8 points on Qwen2.5-1.5B. These gains replicate on fresh seeds and unseen problems. A simple rule, that the candidate prompts are predicted to fail more often than not, predicted where the gains appear in five of six preregistered tests. Both gains come early in training, and standard GRPO with twice the initial learning rate comes within half a point of them.
The authors conclude that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.
Key facts
- A new matched-compute protocol gives every RLVR training intervention the same budget in generated tokens and training GPU-hours, tested in preregistered studies; the paper reports 1,668 runs on six models.
- On Qwen2.5-1.5B, the Monte-Carlo step reward leads GRPO per update but ends 14.3 percentage points lower at equal tokens and 7.8 points lower at equal training time.
- In the authors' setting, the sampled continuations alone take 91% of all generated tokens, and none of four dense-reward variants outperforms GRPO.
- Training on prompts with predicted success nearest 1/2 improves matched-compute accuracy by 1.7 points on Qwen3-0.6B and 0.8 on Qwen2.5-1.5B, but GRPO with twice the initial learning rate comes within half a point.
Why it matters
Dense process rewards are a popular answer to GRPO's habit of giving every token in a solution the same credit. This paper argues that the usual way of judging them, per training update, hides their cost. When the Monte-Carlo reward is charged for the tokens it spends sampling continuations (91% of all generated tokens in the authors' setting), its per-update lead on Qwen2.5-1.5B turns into a deficit of 14.3 percentage points at equal tokens. The wider point is methodological: the ranking of two methods can depend on which budget is held equal.
Who it affects
People who train reasoning models with verifiable rewards and are weighing GRPO against step-level reward schemes, and researchers who report gains for new RLVR tricks. The results are on small models: Qwen2.5-1.5B and Qwen3-0.6B are the ones named, out of six models in total.
How to use it
The practical takeaway is the protocol itself: when comparing a training change with a baseline, give both the same budget in generated tokens and in training GPU-hours, rather than the same number of updates. The paper also suggests a cheap candidate to try, training on prompts whose predicted success is nearest 1/2, with the rule that candidate prompts predicted to fail more often than not are where gains appear. A baseline worth running alongside is standard GRPO with twice the initial learning rate.
How solid is it
This account rests on the abstract alone. The scale is large, with 1,668 runs on six models, and the studies were preregistered with controls for side effects. The prompt-selection gains replicate on fresh seeds and unseen problems, and the simple failure-rate rule predicted where gains appear in five of six preregistered tests. The dense-reward gaps of 14.3 and 7.8 points are stated only for Qwen2.5-1.5B. The abstract does not give absolute accuracy levels of any run, and it does not name the benchmark.
Risks and caveats
The prompt-selection gains are small, 1.7 and 0.8 points, and both gains come early in training. Standard GRPO with twice the initial learning rate comes within half a point of them, so the gain may be reachable by simpler tuning. The 91% continuation share is stated for the authors' setting, not as a general property of Monte-Carlo rewards. The abstract does not say which four dense-reward variants were tested beyond the Monte-Carlo reward, nor which models make up the six. No authors or institutions are named in the source text, and no code, data release or compute totals are mentioned.
“We argue that training interventions should be validated at matched compute, which overturns some apparent gains and confirms others.”
— Paper abstract, Hugging Face papers page