Glonce

Evolution strategies beat GRPO on reasoning coverage, study finds

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact62Novelty72Relevance98Surprise52Players35Credibility78
Evolution strategies beat GRPO on reasoning coverage, study finds

Evolution Strategies (ES) have recently emerged as a memory-efficient way to post-train large language models for reasoning, but researchers say the method's optimization behavior has stayed understudied, making it hard to know when it actually beats the mainstream approach, Group Relative Policy Optimization (GRPO). A new paper sets out to fix that by systematically studying ES dynamics and comparing them directly against GRPO.

The central finding is that ES gives broader reasoning coverage than GRPO, letting it better exploit the reasoning ability already present in a pretrained model. Theoretically, the authors show that verifier-projected Jensen-Shannon diversity across the ES population helps produce higher Pass@K scores. Empirically, GRPO exhibits entropy collapse during training, while ES both improves Pass@1 and reaches higher Pass@K than GRPO. Building on this, the authors develop a sequential GRPO-then-ES training strategy meant to combine GRPO's strength on Pass@1 with ES's gains on Pass@K.

A second finding concerns what actually changes inside the model. Even though ES causes substantial drift across the whole model's parameters, the paper reports that the task-performance gains trace back to only a sparse subset of the larger-magnitude updates. The authors call this functional sparsity, arguing that large parameter movement does not have to mean widespread functional change; held-out evaluations back this up by showing the drift does not necessarily cause catastrophic forgetting.

Finally, the paper looks at how hyperparameter choices affect ES, finding that a larger LLM needs a smaller ES population size to train effectively. Taken together, the authors conclude that ES should be treated as a distinct reasoning post-training paradigm in its own right, rather than a weaker, merely memory-efficient stand-in for GRPO.

Key facts

  • ES reaches higher Pass@K than GRPO while also improving Pass@1, and does so without the entropy collapse GRPO shows during training.
  • Theoretically, verifier-projected Jensen-Shannon diversity across the ES population is shown to help drive higher Pass@K results.
  • The authors propose a sequential GRPO-then-ES training strategy to combine GRPO's Pass@1 strength with ES's Pass@K gains.
  • Despite substantial whole-model parameter drift under ES, the performance gains trace to only a sparse subset of larger-magnitude updates, a pattern the authors call functional sparsity.
  • Held-out evaluations show this parameter drift does not necessarily cause catastrophic forgetting, and a larger LLM needs a smaller ES population size to train well.

Why it matters

ES has been treated mainly as a cheaper, memory-efficient fallback to GRPO for reasoning post-training, with an implicit assumption that it trades away performance for that efficiency. This study challenges that framing directly: by showing ES gives broader reasoning coverage and avoids GRPO's entropy collapse, it argues ES is a genuinely different optimization paradigm with its own advantages, not just a lighter-weight substitute.

Who it affects

The findings speak to researchers and engineers who post-train language models for reasoning and have to choose between RL-style methods like GRPO and population-based methods like ES, particularly teams working under memory constraints where GRPO's costs are hard to sustain. It also matters to anyone designing hybrid training pipelines, since the paper's sequential GRPO-then-ES strategy is aimed squarely at that audience.

How to use it

There is no product or release here, only a training-method finding. The practical takeaway is the sequential recipe: run GRPO first to capture its Pass@1 strength, then follow with ES to pick up its Pass@K gains. The hyperparameter result also matters in practice: population size for ES should shrink as the underlying LLM gets larger, rather than staying fixed.

How solid is it

The paper backs its main claim with both a theoretical argument, based on verifier-projected Jensen-Shannon diversity, and empirical comparisons against GRPO, including a check for catastrophic forgetting via held-out evaluation. That said, the available text does not name the benchmarks, datasets, model families or model sizes used, and gives no numeric values for Pass@1 or Pass@K gains, only qualitative comparisons of higher versus lower performance.

Risks and caveats

Because the source text omits concrete numbers, benchmark names and model sizes, the actual magnitude of ES's advantage over GRPO cannot be judged from what is available here. The text also does not identify the paper's authors, their institutions, a publication venue or a timeline, so the work's provenance and review status are unclear from this material alone.

“These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.”

— the paper's authors