FM-Bench tests AI agents as 20-year football club managers

FM-Bench, short for Football Management Benchmark, has a language model agent run a football club for 20 in-game years, using 26 tools across roughly 340 to 400 decision stops per run. The agent drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth development, sets lineups, and answers to a board that can fire it. A deterministic engine, not an LLM judge or a human rater, accumulates the outcome of every year into one final score. The authors built it to measure a capability they say is largely unmeasured: whether an agent can sustain effective decision-making once its actions carry cumulative consequences and the environment reacts to its choices, rather than just complete a bounded, one-off task.

The benchmark runs in two modes. The solo track pits each of 15 frontier models individually against a frozen scripted world. The Arena puts the same 15 models, plus a scripted anchor, into one shared 20-year world together, which the authors call, to their knowledge, the first head-to-head evaluation of language models at this scale. Across three seeds per model, all 15 complete every 20-year horizon, while the blind scripted baselines die out in most of theirs. claude-fable-5 tops both the solo leaderboard on mean score and the Arena's final standings. Within the Arena itself, though, the lead rotates among ten different models before the order finally settles late in the horizon. Neither a model's scale, its price, nor its vendor predicts where it lands. Even the best-performing human, playing the club for the first time, finishes at the bottom of the model rankings.

The paper attributes the spread in scores to managerial behavior rather than raw computation. Models that score higher cut slow-payoff investment as the 20-year run nears its end, keep cash actively invested rather than idle, and open contract renewals well before their deadlines. Token spend predicts nothing. Two weaknesses show up across the whole field: no model learns the market's hidden prices even after hundreds of rejected bids, and every self-managed memory strategy fails in one of two opposite ways, either growing into an archive that is never pruned or being rewritten from scratch every season instead of building on what came before. The paper measures six such behavioral capabilities behind the final score, though it does not detail the individual capabilities in the available text. Code for FM-Bench is published on GitHub under the Analogy-AI organization.

Key facts

FM-Bench runs a language model agent as a football club manager for 20 in-game years, using 26 tools across roughly 340 to 400 decision stops per run, with a deterministic scoring engine and no LLM judge or human rater.
15 frontier models are tested solo against a frozen scripted world, then together with a scripted anchor in a shared 20-year Arena, which the authors call, to their knowledge, the first head-to-head evaluation at this scale.
claude-fable-5 tops both the solo leaderboard on mean score and the Arena's final standings, though within the Arena the lead rotates among ten different models before the order settles late in the horizon.
Across three seeds, all 15 models complete every 20-year horizon while the blind scripted baselines die out in most of theirs; neither a model's scale, price, nor vendor predicts the final order, and the best-performing first-play human finishes at the bottom of the model rankings.
Higher-scoring models cut slow-payoff investment near the end, keep cash invested rather than idle, and open contract renewals well before deadlines, while token spend predicts nothing; every self-managed memory strategy fails, either by growing into an unpruned archive or by being rewritten from scratch each season.

Why it matters

Most agent benchmarks test whether a model can complete a bounded task. FM-Bench targets a different, largely unmeasured question: can an agent sustain good decisions once its actions compound and the environment reacts, across a 20-year horizon rather than a single episode. The answer the paper reports cuts against a common assumption: neither a model's scale, its price, nor its vendor predicts how well it manages the club, and the final ranking among the 15 models only becomes stable late in the run. What actually separates strong performers from weak ones is managerial behavior, such as when they invest, whether they keep cash idle, and how early they renew contracts, not raw computation. A deterministic scoring engine, with no LLM judge and no human rater in the loop, removes a major source of noise and bias that affects many agent evaluations.

Who it affects

The benchmark is aimed at anyone building or evaluating language model agents meant to operate over a long horizon with cumulative, compounding consequences, rather than agents that just execute a single bounded task and stop. That includes the labs behind the 15 frontier models tested, though only claude-fable-5 is named individually in the available text; the other 14 are not identified. It also speaks to researchers designing agent benchmarks generally, since FM-Bench's deterministic, multi-year, competitive-and-solo format is itself a methodological proposal. The finding that even the best first-play human finishes below every model is relevant to anyone comparing AI agents against human performance on tasks requiring sustained, long-term planning.

How to use it

FM-Bench's code is published on GitHub under the Analogy-AI organization, letting researchers reproduce the solo track, the Arena, or both, and run additional models through the same 20-year club-management setup: the same 26 tools, budget constraints, and board that can fire an underperforming agent. The source text gives no pricing, license terms, or setup instructions beyond the repository link itself, so those details are not covered here.

How solid is it

The design has real methodological strengths: a deterministic engine removes the LLM-judge or human-rater variability common to agent evaluations, each model is run across three seeds, and the solo track is checked against frozen scripted baselines that mostly fail to survive the full 20 years, giving the passing models a real floor to clear. The Arena adds a second, competitive layer using a scripted anchor as a shared reference point. That said, the authors' description of it as the first evaluation of its kind is a claim about their own novelty, not an independently verified one. The available text also does not enumerate the six behavioral capabilities the paper claims to measure, does not give numeric scores or margins for claude-fable-5 or any other model, and names only one of the 15 models tested, which limits how far specific rankings can be checked from the source alone.

Risks and caveats

The source text names no individual authors, no institution, and no publication date, so the work cannot yet be tied to a specific team from what is available here. Fourteen of the 15 models tested are not identified, and no score or margin is given for claude-fable-5's lead, so the size of its advantage is not verifiable from the text. The paper's own results double as a limitation of current agents: none of the 15 models learns the market's hidden prices even after hundreds of rejected bids, and every self-managed memory approach fails, either by growing into an archive that is never pruned or by being rewritten from scratch every season rather than building on prior seasons. Those read as findings about present-day agent limits, not as flaws in the benchmark itself, but they mean the benchmark documents a genuine ceiling on current long-horizon planning, not just a leaderboard.

“What separates the models is managerial behavior rather than computation.”

— the authors