Glonce

Study of 20 RL-trained models: gains come from finishing and reliability, rarely from reach

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact45Novelty60Relevance95Surprise40Players10Credibility75
Study of 20 RL-trained models: gains come from finishing and reliability, rarely from reach

Earlier work (Yue et al., 2025) found that outcome RL raises pass@1 but not pass@k at large k. This paper starts from an observation about that result: every pass@1 is scored under a token budget, and every training recipe also trains under one. So the authors ask whether a pass@1 gain reflects changed behaviour or new knowledge.

On physics and math problems at fixed token budgets, they compare the pass@1 of 20 released RL models (1.5B to 32B parameters) with that of the thinking models each one started from. Each change in pass@1 is divided into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach measures the problems RL solves that its base answered wrongly in all eight tries, minus the problems the base solves that RL answered wrongly in all eight.

The authors report four findings. First, where the base often hits the token budget before answering, most of the change, gain or loss, is finishing: 82% at 4k and 8k budgets. Given more tokens, the base closes most of the finishing gap itself. Second, where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both models finish.

Third, by the authors' definition, RL rarely solves what its base cannot. Reach adds at most 2.1 percentage points on any of their four physics and math sets. They also checked 144 physics problems that only the RL model solved; of these, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples.

Fourth, in a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.

Key facts

  • The paper compares 20 released RL models (1.5B to 32B) with the thinking models they started from, on physics and math problems at fixed token budgets.
  • Each pass@1 change is split into finishing, reliability and reach; where the base often runs out of tokens, finishing accounts for 82% of the change at 4k and 8k budgets.
  • Where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both finish.
  • Reach adds at most 2.1 percentage points on any of the four sets; of 144 physics problems only the RL model solved, 3.5% hold up when resampled or read.
  • In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat.

Why it matters

Outcome RL is widely credited with making reasoning models better. Prior work (Yue et al., 2025) already showed that it lifts pass@1 without lifting pass@k at large k. This paper adds a finer split of where the pass@1 gain comes from. Its answer is that much of it is behaviour under a token budget: answering before running out of tokens, and being right more often on problems the base can already finish. New problems solved are a small part.

Who it affects

People who train or evaluate RL reasoning models, and anyone who reads pass@1 gains as proof of new capability. The paper's setting is physics and math problems with models from 1.5B to 32B parameters.

How to use it

The paper offers a way to read a pass@1 gain: ask how much is finishing, how much is reliability and how much is reach, and note the token budget the score was taken under. The authors also report that giving the base more tokens closes most of the finishing gap by itself, so budget matters when comparing a base with its RL descendant.

How solid is it

The authors compare 20 released models against their own bases and back the reach finding with several checks: a 2.1 percentage point ceiling across four sets, a review of 144 physics problems that only the RL model solved (3.5% hold up), and a human-confirmed sample of new RL steps, nine of ten of which also appear in the base's samples. This account rests on the paper's abstract. The abstract names no authors or institutions, none of the 20 models, and none of the four sets.

Risks and caveats

The reach finding holds "by our definition" and is limited to the tested physics and math sets, so it is not a claim that RL can never produce new knowledge. The 82% finishing share is given only for 4k and 8k budgets. No numeric reliability gain is stated for the case where the base has enough tokens. The 8B run is small-scale, and the abstract does not say which RL recipe it used. The result on supplying each problem's physical setup is "not shown to improve" on plain RL, which is not the same as shown to fail.

“By our definition, RL rarely solves what its base cannot.”

— The paper's authors