Elo-per-token analysis finds AI agents plateau while top humans keep improving

Researchers propose a method called Elo-per-token analysis to measure how AI agents' performance scales as they spend more compute at test time, revising answers, using tools, exploring alternatives and deciding when to stop. The method tracks the best solution an agent has found at each token budget and uses a Bradley-Terry model to turn task-by-task rankings into Elo ratings that can be compared across tasks with different scoring scales. The team applied it to four general-purpose AI agents on four open-ended benchmarks, running sessions of up to 100 million tokens, and separately to three feedback-driven LLM optimization harnesses in controlled single-task tests. They compare agent performance against a theoretical reference of independent sampling, for which Elo is expected to grow linearly with the log of compute spent. Against that reference, the agents studied initially convert tokens into Elo faster than independent sampling would, but their marginal gains shrink over time and eventually drop below the reference line. By contrast, the strongest historical human players in the AtCoder Heuristic Contest, working on the same shared tasks, keep improving superlinearly over the course of a contest, which the researchers read as evidence that humans keep learning and that there is still substantial room for agents to improve once their own gains slow down. The paper defines a "scaling inflection point": the per-session token budget at which an agent's marginal Elo gain matches what independent sampling would achieve. Using that inflection point as a guide, the researchers split a fixed 100-million-token budget across multiple parallel sessions on a benchmark task called FrontierCS Polyomino Packing, and found this strategy gained 264 Elo points over running one long single session with the same total budget, and 355 Elo points over splitting the budget into ten shorter sessions.

Key facts

Elo-per-token analysis tracks an agent's best solution at each token budget and uses a Bradley-Terry model to convert within-task rankings into comparable Elo ratings across tasks with different scoring scales.
The method was applied to four general-purpose agents on four open-ended benchmarks, with sessions of up to 100 million tokens, plus three feedback-driven LLM optimization harnesses tested separately.
Compared to a theoretical independent-sampling reference (where Elo grows linearly with log of compute), agents initially gain Elo faster but their marginal gains diminish and eventually fall below the reference.
The strongest historical human players in the AtCoder Heuristic Contest improve superlinearly over contest time on the same shared tasks, which the authors take as evidence agents still have headroom after they slow down.
Splitting a 100-million-token budget across parallel sessions at the measured scaling inflection point, on the FrontierCS Polyomino Packing benchmark, gained 264 Elo over one long session and 355 Elo over ten short sessions.

Why it matters

How well an AI agent scales with more test-time compute is hard to measure directly, because agents spend that compute adaptively: they revise answers, call tools, branch into alternatives and decide on their own when to stop. Elo-per-token analysis addresses this by scoring the best intermediate solution at every point in a long agent session and turning that into an Elo rating comparable across very different tasks. That gives researchers a common yardstick for a question that previously had no clean answer: whether letting an agent keep working actually keeps paying off, and at what point it stops.

Who it affects

The method targets researchers and engineers who build or evaluate AI agents and optimization harnesses, and who need to decide how much test-time compute to allocate to a task and how to structure a session. It also speaks to benchmark designers working with open-ended tasks that produce continuous scores for intermediate submissions rather than a single pass or fail result.

How to use it

The practical finding is about session structure rather than a product: once an agent passes its scaling inflection point, the point where its marginal Elo gain per token drops to what plain independent sampling would achieve, further gains from one long session slow sharply. Splitting the same total token budget into several parallel sessions, sized around that inflection point, outperformed both a single long session and ten short sessions in the study's own test on the FrontierCS Polyomino Packing benchmark, gaining 264 and 355 Elo respectively.

How solid is it

The method was tested across four general-purpose agents on four open-ended benchmarks with sessions running up to 100 million tokens, plus three separate feedback-driven optimization harnesses in controlled single-task interventions, and it is compared against a theoretically derived independent-sampling baseline rather than an arbitrary one. The parallel-session result quantified with exact Elo gains (264 and 355) comes from one specific benchmark, FrontierCS Polyomino Packing, rather than being demonstrated across the full set of benchmarks.

Risks and caveats

The available text does not name the four general-purpose agents, the four benchmarks or the three optimization harnesses used in the study, nor does it state a publication venue or date, or mention any code or data release. It also does not say whether the parallel-session Elo gains seen on FrontierCS Polyomino Packing generalize to the other benchmarks in the study.