Codex Astra tops Brood War Bench, beating Grok and Claude Fable at StarCraft

A developer who goes by benswerd built Brood War Bench, a benchmark that has AI models play the real time strategy game StarCraft: Brood War against each other, controlling the game entirely through agents, and published a report on how each model performed. The project grew out of a side experiment: the author had built a version of Brood War that could only be played through agents, tried it with friends who had barely played StarCraft before, and watched them do surprisingly well simply by telling their agent to attack while it built a small army and carried out the assault on its own. That made the author want to see how far the agents could go without human direction, which became the benchmark.

The setup was a round robin matrix: every model and effort-level configuration played every other one, with matches run in parallel on Freestyle VMs while the harness logged both the game engine data and each agent's own reasoning and command logs. Effort settings for a given model are pooled together in the results, and the report includes time series charts of unit and building counts, a win rate versus cost chart (Codex and Claude costs are estimated from token usage), and a head to head win/loss/tie matrix.

Codex Astra was the clear leader, beating the other models consistently. Its strongest pattern was disruption rather than economy: in Protoss games it often sent a single Probe across the map to hit enemy workers or buildings, which worked well because opposing agents would spend dozens of seconds deciding how to respond to a lone Probe instead of doing anything else. Codex was weaker at sustained production, tending to delay its tech, trickle one or two basic units at a time into defended bases, and throw workers into hopeless last stands. The author also noticed Codex often spun up separate subagents to run the economy, army production, and army control, and these subagents did not coordinate well: the army-control subagent would frequently send each freshly produced unit straight into an attack on its own, unaware of the larger force the other subagents were building toward, a classic beginner mistake of attacking piecemeal instead of massing for a planned strike. In games where the author stepped in to direct Codex, it coordinated its subagents and timed its attacks better. Codex also showed persistence: in game G009, after Codex 5.6 Terra (medium effort) lost its army and main base, it lifted its last Command Center and flew it toward the opposite corner of the map, surviving another six minutes before finally being eliminated.

Grok models, by contrast, were judged not yet capable enough to play Brood War. Grok 4.6 tended to spend long stretches reasoning while issuing very few actual commands. In game G043, the xhigh effort setting logged 11,138 reasoning tokens but issued only six command batches over 43 minutes and never produced a single combat unit. In G003, Grok at xhigh effort built three Marines and never reached the enemy base; in G002, Grok at medium effort built two Zealots and likewise never crossed the map. The author reads these less as bad strategic choices than as a failure to keep observing the game state and acting on it.

Claude Fable was the model the author found themself rooting for. It usually tried to build a real economy and climb the tech tree rather than stopping at the first available unit, seeming more genuinely interested in playing the game than the other models. In game G007 it reached a Lair, a Spire, and Mutalisks and won. In G027 it went further, adding a Robotics Facility, Citadel of Adun, Observatory, and Templar Archives before winning that match too. Ambition did not always translate into results: in G036, Fable had reached a Factory and an Academy but was overrun by Opus 5.

The overall verdict, in the author's words, was that no agent played beyond a beginner level. Even the two strongest performers, Codex Astra and Claude Fable, could not build complex armies, defend against simple attacks, or execute concrete strategies, and the author judges that a human beginner doing a basic photon rush would beat every model in every one of these games. Despite that, the author describes coming away more excited than they have been in a while, and considers the benchmark far from exhausted, with plenty more for the agents, and the benchmark itself, to grow into.

Key facts

• Brood War Bench, built by an independent developer, has AI models play StarCraft: Brood War against each other purely through agents, in a round robin matrix run in parallel on Freestyle VMs.
• Codex Astra was the clear overall leader, winning largely through disruption: sending a lone Probe to harass workers or buildings while opponents wasted time deciding how to react, though it was weak at sustained production.
• Codex's subagents for economy, production, and army control often failed to coordinate, causing it to send units into attacks one at a time instead of massing a force.
• Grok 4.6 was judged not capable of playing Brood War yet: in one game it logged 11,138 reasoning tokens but issued only six command batches in 43 minutes and never fielded a combat unit.
• Claude Fable tried hardest to build a real economy and tech tree and won multiple games, but the report concludes no model, including Fable and Codex Astra, played beyond a beginner level.

Why it matters

Most public AI evaluations test models on math, coding, or short question answering. Brood War Bench instead tests sustained, real time, adversarial decision making: an agent has to keep observing a changing game state, plan production and army composition over many minutes, and react to an opponent, all through a chain of individual commands rather than a single response. That makes it a useful stress test for the kind of long horizon, tool using autonomy that agent builders are trying to sell, independent of any benchmark the model makers themselves publish.

Who it affects

The models named, Codex Astra, Grok 4.6, Claude Fable, and Opus 5, are benchmarked by a third party with no apparent involvement from OpenAI, xAI, or Anthropic. Anyone evaluating how far current AI agents can be trusted with multi step, real time tasks gets a concrete data point here, as does anyone deciding which model to route to an agentic task where sustained planning matters more than a single correct answer.

How to use it

The full report, including time series charts, a win rate versus cost comparison, and a head to head results matrix, is published at bw.swerdlow.dev/report. It is a benchmark report rather than a product: there is no pricing, access tier, or service to sign up for, and the interesting part is reading the per game breakdowns of how each model actually played.

How solid is it

The benchmark used a systematic round robin design, with every model and effort configuration playing every other one, and the harness preserved both game engine data and each agent's own logs, which is what lets the author describe specific plays such as the G009 Command Center retreat or Grok's token counts in G043. It remains a one person, independently run project: the text does not state how many total games were played, does not give an overall numeric win rate table beyond naming Codex Astra as leader, and does not date when the games were run or the report published.

Risks and caveats

The report itself is explicit that none of the models played above a beginner level, so the results describe how far current agents are from competent real time strategy play, not a ranking of finished skill. The account of 'Freestyle VMs' as the platform running the matches is not explained further in the text, and the author's read of what happened in particular games, such as calling Grok's low activity a failure to keep acting rather than a deliberate cautious strategy, is the author's own interpretation rather than something the models themselves stated.

“A beginner playing photon rush would win every single one of these games.”

— Brood War Bench report