Glonce

'Never Give Up' targets RL training's bias toward easy problems

researchThe Glonce editorsHacker News🔥69Glonce rating69Impact65Novelty78Relevance98Surprise55Players45Credibility82

A blog post walking through a new paper on reinforcement learning (RL) post-training for large language models argues that the standard way of measuring progress hides an important failure. The author's eval is an average over 30 AIME 2025 math questions, run during RL training of Olmo 3.1 RL-Zero Math. Split into three difficulty subsets by how the pre-RL model performed (a question is labeled "hard" if the starting model scores 0 on pass@32, and the rest are split evenly into "medium" and "easy"), the three subsets start at pass@1 averages of 0%, 3.8% and 22.7%. Averaging over all 30 questions hides that most of the improvement during training comes from the easiest subset going from partly solved to mostly solved; the hardest problems barely move.

The same pattern shows up outside math. Using Deepcoder and DeepSWE, two open-source projects that released their own trained models and training logs, the author builds difficulty buckets for code and agentic RL: one benchmark (LCBv6) is split using an initial model's own performance (Deepseek-R1-Distilled-Qwen-14B), another (SWEBench) using its existing task-difficulty labels. Across these domains, the gains from RL scale with how easy a problem already was. The post names this the "Matthew Effect" in RL for LLMs, borrowing the "rich get richer" term from network science and economics (Merton, 1968): RL improves performance on a task in proportion to a model's initial competence, so easy tasks get easier while hard tasks often stay hard.

To explain the cause, the author first tests whether it is simply a sampling problem: with GRPO, a prompt yields no training gradient at all if none of its k sampled completions are correct, a failure the post calls "signal loss" (following Xiong et al., 2025), and the obvious fix is to sample more completions per prompt, a larger k. Training Qwen 2.5 0.5B Instruct with GRPO on GSM8k platinum, split by initial pass@1 into easy (25%), medium (10%), hard (5%) and extra-hard (0%) buckets, and comparing k values of 4, 8, 16 and 32 at a fixed batch size, the surprising result is that the smaller k=4 wins overall. The reason is batch composition: because GRPO filters out any prompt whose completions are all correct or all wrong, a larger k does find more rare correct solutions to hard problems, but it also finds more rare incorrect solutions to already-easy ones, and filtering those out takes far more compute (a problem is filtered once solved in 4 of 4 completions at k=4, but needs 32 of 32 at k=32). Early in training, the larger k=32 does better at catching rare hard-problem solutions; after an inflection point around step 200, the smaller k=4 pulls ahead by spending far less compute re-confirming easy problems. The author frames this as a matter of "signal efficiency," not just signal loss: the deeper issue is not undersampling hard problems but overspending compute on easy ones. Because the training setup is asynchronous, the compute saved by quickly filtering easy problems is redirected straight into training on harder ones.

The proposed fix, Never Give Up (NGU), tries to get the best of both: small k for easy problems, large k for hard ones, chosen adaptively rather than preset. Sampling starts with a small k. If a prompt is solved within those first k completions, it goes straight to training; if it is fully solved, meaning all k completions are correct, it is filtered out quickly. If every completion in the first batch is wrong, then with probability p the method never gives up: it sends the prompt back to the generator for k more completions, keeps the earlier ones, and once the prompt is eventually solved it trains on the whole accumulated batch of k times however many rounds it took. This produces a geometric distribution of sample counts; a prompt that never gets solved is expected to consume k/(1-p) samples. Unlike curriculum learning, which fixes a problem's difficulty in advance, NGU adapts online as problems get easier or harder over the course of training. On GSM8k, k=4 with NGU at p=0.9 beats every tested value of plain GRPO, with the largest gains on the hardest subset, by combining the effective large-k behavior of early training with the effective small-k behavior of late training.

Resampling in rounds introduces a side effect: some of the completions used to eventually train on a hard prompt were generated earlier in training and are stale by the time they are used, and stale negative completions are known to hurt RL training of LLMs, citing Le Roux et al. (2025). The post filters out completions older than an age threshold, finding T=4 rounds works best, but that raises a bookkeeping problem for the GRPO baseline: in a worked example with 4 stale negative completions, 3 fresh negatives and 1 fresh positive, the baseline is set to 1/8 over all eight completions, and training on only the four fresh ones leaves the group's total reward at 3/8 instead of the zero it should be. The post lays out three options: ignore the filtered completions in the baseline, leave the non-zero baseline as is, or anchor the positive and rescale the negative advantages, by a factor of 7/3 in this example, to bring the total back to zero. Its own conclusion: use every sample gathered for the baseline, even the ones excluded from training.

Scaled up to DeepScaler with a Qwen 3 4B base model, NGU still improves on a strong GRPO k=16 baseline, again concentrated on the hardest subsets of an AIME and BRUMO 2025 evaluation; the Matthew Effect persists even with NGU, but is reduced, and the harder questions improve without the easier ones getting worse. Code RL behaves differently, since a single coding problem can contain both easy and hard tests rather than being simply right or wrong. On the Manufactoria benchmark, following the setup of Sun et al. (2025), standard GRPO improves at first and then stagnates, spending most of its training signal revisiting partially solved medium-difficulty tests. Applied here, NGU only accepts a new batch of completions once it clears the previous best: if an earlier batch passed 7 of 12 tests, the next accepted batch must pass more than 7 of 12, pushing the model to keep improving where plain GRPO stalls out.

The author also checks a competing explanation: that the Matthew Effect is really a version of the "primacy bias" documented in deep RL (Nikishin, Schwarzer, D'Oro et al., 2022), where bad early samples can derail a training run, in part because of plasticity issues in the network. Resuming from a Manufactoria checkpoint at step 6,000 that had already been stagnant for at least 3,000 steps, training continued either with a single reward for passing every test, the "all-tests" approach Sun et al. originally used to break the stagnation, or with the original per-test reward plus NGU. Both approaches recovered strong performance, which the author reads as evidence that plasticity loss is not the main driver and that these models can generally recover from a long stretch of bad training data. The stated limits of NGU: it is unlikely to help when a task is dominated by very hard problems, and its sample-wait-resample cycle takes longer per batch than sampling k/(1-p) completions upfront, which makes early samples more off-policy and can slow learning, unless a reasonable k for the data is already known. In conclusion, the author argues that a single averaged eval score can conceal exactly this kind of imbalance, and that Never Give Up offers a simple way to reallocate training compute from easy problems to hard ones, with future work aimed at more complex, multi-step agentic settings.

Key facts

  • The post argues that standard RL post-training such as GRPO mostly improves LLM performance on problems a model can already partly solve, while the hardest problems barely move, a pattern it names the "Matthew Effect" in RL for LLMs after the "rich get richer" concept from network science and economics (Merton, 1968).
  • On an Olmo 3.1 RL-Zero Math run evaluated on 30 AIME 2025 questions, the three difficulty subsets (hard, medium, easy) start at pass@1 averages of 0%, 3.8% and 22.7%, and almost all of the eval's overall improvement comes from the easiest subset.
  • In a GRPO ablation on Qwen 2.5 0.5B Instruct with GSM8k platinum testing k values of 4, 8, 16 and 32, the smaller k=4 wins after an inflection point around step 200, because larger k wastes compute finding rare incorrect solutions to already-easy problems.
  • The proposed fix, Never Give Up (NGU), resamples an unsolved prompt with probability p instead of dropping it; at k=4 with p=0.9 it beats every tested standard-GRPO k on GSM8k, with the biggest gains on the hardest subset.
  • NGU also improves a larger DeepScaler math setup on a Qwen 3 4B base model (on top of a GRPO k=16 baseline) on the hardest AIME and BRUMO 2025 subsets, and on the Manufactoria code benchmark it keeps pushing past prior best test-pass thresholds where standard GRPO stalls.

Why it matters

Teams training reasoning models with RL typically watch a single averaged eval curve go up and read that as progress. This post shows that curve can hide a specific failure: the gains concentrate on problems the model could already mostly solve, while performance on the hardest problems, often the ones a reasoning model is built for, barely moves. That matters because a rising average score can mask stalled progress on exactly the cases that most need it. The proposed fix, Never Give Up (NGU), is not a new model or a new loss function; it is a change to how training prompts are resampled inside an existing asynchronous GRPO pipeline, so it is a comparatively cheap lever for redirecting training compute toward the hard problems that plain GRPO tends to neglect.

Who it affects

Researchers and engineers who post-train LLMs with RL, particularly GRPO-style methods for math, code and agentic reasoning, and who already run asynchronous RL training infrastructure that NGU is designed to slot into. It is less directly relevant to people who only use finished chat or coding assistants; the post is about how such models get trained, not about a product feature end users would see.

How to use it

NGU is described as a sampling policy layered on top of standard GRPO, not a separate tool or product: start with a small k, train immediately on any prompt solved within the first k completions, and filter it out once every completion is correct. If all k completions on a prompt are wrong, resample it with probability p for k more completions, keep the earlier ones, and once it is eventually solved, train on the whole accumulated batch. An age threshold discards completions that go too stale from repeated resampling (the tests here use T=4 rounds), and the GRPO baseline needs a matching adjustment for whichever completions get filtered out; the post lays out three options for that adjustment, ignore the filtered completions, leave the baseline unrescaled, or anchor the positive and rescale the negative advantages to keep the group's total reward at zero; its own conclusion is to use every sample gathered for the baseline calculation, even the ones excluded from training. The post points to a paper on arXiv and code on GitHub for the full method, though it does not give either link directly.

How solid is it

The claim is tested across four settings of increasing scale and difficulty: a full Olmo 3.1 RL-Zero Math AIME run, a controlled ablation on Qwen 2.5 0.5B Instruct with GSM8k platinum that isolates the effect of k, a larger DeepScaler setup on a Qwen 3 4B base model, and the Manufactoria code-RL benchmark, plus a dedicated experiment that argues against a competing "primacy bias" explanation by showing training can recover after a long stagnant stretch. That is a reasonably broad spread across math and code, with a consistent GRPO baseline for comparison in each case. It remains a first-party account: this is the paper's own author summarizing the paper's own results in blog form, not an independent replication, and the full paper, with its complete author list, institution and arXiv identifier, is referenced but not linked in the retrieved text.

Risks and caveats

The author states directly that NGU is unlikely to help when a task is dominated by very hard problems, since its benefit comes from quickly freeing up compute on easy ones. The resample loop is also slower per training batch than sampling all completions upfront, which makes early samples more off-policy and can weaken the learning signal unless a reasonable k for the data is already known. Even where NGU works, the Matthew Effect is described as mitigated, not eliminated: on the larger DeepScaler run it still persists, just less severely. Details needed to independently verify the work, such as the paper's full author list, arXiv identifier, institution and publication date, are not present in the retrieved text.

“This demonstrates how simple scalar values may not be sufficient for accurate evaluations of LLMs.”

— the author, in the post's conclusion