AhaBench benchmark finds Claude Opus 4.6 leads at learning from experience

AhaBench is a new benchmark suite built around a question most agent evaluations skip: when a fixed language model is given useful experience, does its later behavior actually improve once the obvious support is removed, changed, or delayed? Typical evaluations reset the agent after each prompt or score only the outcome of a single trajectory, so they cannot show whether an agent learned anything that survives a changed setup.

The suite has three components. Aha-Puzzle presents no-hint exploration tasks after the agent has already solved related hidden-state puzzles. Aha-Euler turns Project-Euler-style mathematical problems into generated taught and held-out tasks with exact validators. Aha-Vending, an open-source implementation inspired by Vending-Bench, tests whether a simulated vending agent stays profitable while handling delayed feedback and operational incidents.

AhaBench scores each model on three numbers: an Initial Score for starting competence, a Post-Experience Score for the later empirical outcome, and a Learning Lift, the difference between the two. The authors say this decomposition is the paper's main empirical message: models that use visible support well, models that reach high post-experience scores, and models that improve the most during a run are not always the same model.

On a common eight-model panel, Claude Opus 4.6 leads with an aggregate Post-Experience Score of 64.3 and an aggregate Learning Lift of plus 25.8. Gemini 3.1 Pro follows close behind on Post-Experience Score at 63.4. The other six models in the panel are not named in the released text. The component results explain part of the gap: puzzle traces raise scores when support is visible but often fail to turn into no-hint exploration behavior afterward. On Aha-Euler, full teaching reaches 78.6 to 100.0 percent across models, while answer-only transfer, without the worked steps, ranges from just 0.0 to 73.9 percent. On Aha-Vending, the benchmark separates agents that handle incidents profitably from those that go bankrupt or fail to place orders at all.

The team releases the benchmark tasks, rubrics, validators, simulator code, and interfaces so other researchers can run the same three-part scorecard against new agents.

Key facts

AhaBench has three components, Aha-Puzzle, Aha-Euler, and Aha-Vending, each testing a different form of long-horizon learning from prior experience.
On a common eight-model panel, Claude Opus 4.6 leads with an aggregate Post-Experience Score of 64.3 and an aggregate Learning Lift of plus 25.8; Gemini 3.1 Pro follows close behind at 63.4 on Post-Experience Score.
On Aha-Euler, full teaching scores range from 78.6 to 100.0 percent across models, while answer-only transfer without worked steps ranges from only 0.0 to 73.9 percent.
Puzzle traces raise scores when the support is visible but often fail to become no-hint exploration behavior once that support is removed.
The team releases the benchmark tasks, rubrics, validators, simulator code, and interfaces for evaluating other agents against the same suite.

Why it matters

Most agent benchmarks reset the model after each prompt or judge only the outcome of a single trajectory, so they cannot show whether an agent actually carries experience forward. AhaBench targets that gap directly by comparing a model's score before and after it receives useful experience, under a version of the task where the original support has been removed, changed, or delayed. The paper's central claim, that supported performance, final performance, and learning gain produce three different rankings rather than one, is a challenge to any benchmark that reports a single number as if it captured agent learning on its own.

Who it affects

Teams building or evaluating long-horizon language agents, and anyone choosing between models such as Claude and Gemini for agent products that must retain and reuse experience across a session rather than start fresh each time, gain a way to check Learning Lift directly instead of relying on a single end-of-run score.

How to use it

The authors release the benchmark tasks, rubrics, validators, simulator code, and interfaces, so the suite can be run against any agent to obtain the same Initial Score, Post-Experience Score, and Learning Lift figures reported here. The text gives no price, license terms, or repository link, so those details are not confirmed here.

How solid is it

The results come from a single source text, the paper's own abstract-level description, with no independent replication mentioned. Only two of the eight models in the panel, Claude Opus 4.6 and Gemini 3.1 Pro, are identified with actual scores; the other six are not named. The text also gives no authors, no institutions, and no publication or release date, so the claims here are bounded to what the released description itself states.

Risks and caveats

Because only two of eight panel models are named with scores, the lead reported here reflects a partially disclosed comparison; the ranking among the other six models is unknown from this text. The source also does not describe what the operational incidents in Aha-Vending actually consist of, nor does it name or link a code or data repository beyond stating that the components are released, so readers cannot yet verify reproducibility from the text alone.