Glonce

Gemini 4 Argon places 3rd as GPT-6 Astra leads StarCraft bot benchmark

researchThe Glonce editorsHacker News🔥64Glonce rating64Impact40Novelty55Relevance95Surprise50Players35Credibility35

StarSkirmish is a benchmark where LLMs write bots for StarCraft: Brood War. Each model gets one hour of wall clock time to write a Protoss bot in C++, and the bot is then scored by its win rate against a roster of competitive human-written bots and demo bots. The page's own pitch is that this tests long horizon reasoning and agentic coding, because the model has to fold lessons from practice games back into its bot. The page names no authors or institutions behind the benchmark.

Version 0.2 adds flagship models from OpenAI, Anthropic, Google and Meta, each running in its own lab's first-party CLI harness. Google's Gemini 4 Argon and Meta's Muse Wonder Full replace v0.1's Gemini 3.8 Flash and Muse Spark 1.3. Gemini 4 Argon places 3rd and Muse Wonder Full 7th. GPT-6 Astra and Claude Opus 5.5 were already covered in v0.1.

The results: GPT-6 Astra is the top scorer, a clear step ahead of GPT-6.1 Sol in second. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are described as functionally tied behind them, all scoring between 37 and 39. The page does not give exact scores for the top two. Scores are the average win rate (0 to 100) of each model's 5 runs. One caveat on the comparison: Gemini 4 Argon runs at high reasoning effort, the highest available to it, while every other model runs at max.

On cost, GPT-6.1 Sol is the only model in the most attractive score-versus-cost quadrant, scoring second at just over a tenth of GPT-6 Astra's cost per run. The page gives no dollar figures in its text. Muse Wonder Full's cost is based on Muse Wonder's API pricing.

The benchmark shows no sign of saturating. The strongest human-written bots are still well ahead of every model: against Stardust, PurpleWave and BananaBrain, the best model, GPT-6 Astra, wins only 7% of its games. The authors add that the one-hour reasoning limit is a factor, but also that the matches are played on three 2 player maps and real human-written bot tournament conditions are significantly harder.

The strongest predictor of a model's score is how much it practiced against the 7 strongest bots on the roster. GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon start playing these bots early in the hour, while their own bots still lose to them. Muse Wonder Full, Claude Haiku 5.5 and GPT-6 Luna rarely or never do. Claude Fable 5.1 is the outlier: it scores 37 while practicing almost entirely against weaker bots.

Results also vary between runs of the same model. Gemini 4 Argon varies far more than the other labs' flagships: its best and worst runs are 18 points apart, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.

Setup details. Bots are written against BWAPI 4.4.0 and played on OpenBW, Protoss versus Protoss, on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side's buildings are all destroyed. Games that reach the 60 minute cap (86,400 frames) are decided as on the BASIL ladder: the higher kills + razings score wins. The model starts from a skeleton bot that only mines, plus the source of three demo bots and two libraries, FAP (a combat simulator) and BWEM (a map analyzer). Whatever is in the bot folder when the hour ends is the final bot, so it must keep compiling. The model has tools to compile, play practice games, and read each game's summary, transcript and its bot's own log. Practice opponents come in tiers: D (the three demo bots), C and B (three mid-ranked human-written bots each), A (BananaBrain, Locutus) and S (Stardust, PurpleWave). The CLIs' web search and web fetch tools are off and the agent has no other internet access. There is no submit tool; the harness collects the bot when time runs out. Every v0.2 harness compacts the model's context at about 250k tokens. CLI versions: Claude Code 2.1.296, Codex 0.162.1, Antigravity CLI 1.3.5 and Muse Code 1.4.7.

In the tournament, each model's 5 final bots play each of the 13 roster bots (10 competitive human-written bots and 3 demo bots) 6 times on each map, on fresh seeds. Unlike v0.1, LLM bots no longer play each other, and v0.2 replaces v0.1's Elo ratings with plain win rates. A model's score is the average of its 5 bots' scores, from 0 (every game lost) to 100 (every game won). Because the roster is fixed, a new model can be scored without replaying anyone else, and scores across model generations are more directly comparable than fitted Elo ratings. v0.1 used one neutral third-party harness at high reasoning; v0.2 uses each lab's own CLI at the highest setting it offers.

On correlations, StarSkirmish no longer tracks the most widely reported evals. It barely moves with the Artificial Analysis Intelligence Index (Spearman ρ 0.26), where GPT-6 Astra only ties for 4th of these 9 models. It has drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1: Claude Sonnet 5.5 and Muse Wonder Full both rank near the top of Terminal-Bench 4.0 but near the bottom here. It still tracks Vals IOI (olympiad programming problems solved in C++) and WeirdML, though WeirdML v3 covers only 7 of the 9 models, with Gemini 4 Argon and Muse Wonder Full not yet included. One reading offered is that the most heavily benchmarked evals have become targets that models are tuned for, while less targeted tests of long horizon, difficult coding tasks still move together.

Key facts

  • Each LLM gets one hour to write a Protoss StarCraft: Brood War bot in C++, run in its lab's own CLI harness; the score is the average win rate of 5 runs against 13 roster bots.
  • GPT-6 Astra is first, GPT-6.1 Sol second; Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind them, all between 37 and 39. Gemini 4 Argon places 3rd.
  • Even the best model, GPT-6 Astra, wins only 7% of games against the top human bots Stardust, PurpleWave and BananaBrain.
  • GPT-6.1 Sol scores second at just over a tenth of GPT-6 Astra's cost per run.
  • Gemini 4 Argon is the most erratic flagship: its best and worst runs are 18 points apart, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.

Why it matters

Most coding benchmarks hand a model a fixed task. This one asks it to build something, test it against opponents for an hour and keep improving it, which is closer to long agentic work. The result is also a reminder that rankings can shuffle: the page reports a correlation of only ρ 0.26 with the Artificial Analysis Intelligence Index, and says it has drifted from Terminal-Bench 4.0. The best model still wins just 7% of games against the top human bots, so the benchmark is far from saturated.

Who it affects

Teams choosing a model for agentic coding will find a different ordering here than on general leaderboards: GPT-6 Astra on top, GPT-6.1 Sol second at a fraction of the cost, and Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 in a near-tie. Benchmark builders may take note of the shift to fixed rosters and plain win rates. StarCraft bot developers see how far LLM-written bots are from the strongest human-written ones.

How to use it

The page presents the results as charts: score by model, score against cost per run, win rate against the top human bots, practice games against strong bots, and run-to-run variance. For a practical reading, compare cost and score together: GPT-6.1 Sol is the only model in the best quadrant. For a closer look at agent behaviour, the page's trace analysis shows that early, frequent practice against the strongest opponents tracks with higher scores. The page does not state licence or pricing terms for the benchmark.

How solid is it

This is an independent benchmark published on its own site, and the page names no authors or institutions. Scores are averages of 5 runs per model, each final bot playing 13 roster bots 6 times on each of three maps on fresh seeds. The three models in the 37 to 39 band are described as functionally tied, and the page itself shows large run-to-run spread for Gemini 4 Argon (18 points between best and worst run). The page gives no exact scores for the top two models. Model names and results come from the page alone.

Risks and caveats

Gemini 4 Argon runs at high reasoning effort, the highest available to it, while every other model runs at max, so the comparison is not like for like. Each model uses its own lab's CLI harness, so harness differences are mixed into the scores. The authors say the one-hour limit and the three 2 player maps make these matches easier than real human-written bot tournaments. WeirdML v3 results cover only 7 of the 9 models, leaving out Gemini 4 Argon and Muse Wonder Full. The claim that heavily benchmarked evals have become tuning targets is offered as one possible reading, not an established finding.

“The strongest human written bots are still well ahead of every model.”

— StarSkirmish Bench v0.2 page