Codex Astra Goes 18-0 on Brood War Bench, Where No Agent Beat Beginner Level

TL;DR

Brood War Bench is a round-robin tournament in which 19 model and effort configurations play StarCraft: Brood War against each other through tool calls. Codex Astra at xhigh effort finished 18-0 with a 100% win rate, averaging 12.6 actions per minute at $10.54 a game. The author, Ben Swerdlow, does not spin it: "None of the models played beyond a beginner level," and "a beginner playing photon rush would win every single one of these games." The ranking is the least interesting part. What real-time play exposes about agent harnesses is the part worth your time.

What was actually measured

Swerdlow built a version of Brood War you can only play through an agent, originally so he could play it with friends. When those friends did better than their StarCraft experience justified, entirely by telling their agent to attack, he turned it into a benchmark. Games ran on parallel VMs from Freestyle, recording game-engine data alongside agent logs.

The setup matters more than usual here, and Swerdlow spelled it out on Hacker News. The agent harnesses were Claude Code, Codex, and Grok Build, chosen because he had subscription tokens rather than an API budget. The game harness was minimal: BWAPI issue-command and get-observation calls exposed as tools, nothing else. No vision, no curated state summary, no scaffolding that would do the strategic work for the model.

Thinking is a move, and the clock does not stop

Every eval you already trust is turn-based. SWE-bench waits for you. Terminal-Bench waits for you. An RTS does not, and that single change reorders the leaderboard.

In a turn-based benchmark, reasoning is billed in dollars and seconds. In Brood War it is billed in territory. It is the difference between taking your time on a crossword and taking your time on a fire drill.

The clearest case is game G043. Grok 4.6 at xhigh effort logged 11,138 reasoning tokens and issued six command batches across 43 minutes, never fielding a single combat unit. It thought about Brood War more carefully than most people have since 1998 and never built a soldier.

More effort only helps the model that can afford it

Raise the effort dial on the strongest system and it climbs cleanly: Codex Astra scored 100.0% at xhigh, 88.9% at medium, 77.8% at low. Do the same on a weaker sibling and it goes backwards. Codex 5.6 Sol landed at 72.2% on medium, 66.7% on low, and 61.1% on xhigh, its worst result at its most expensive setting.

Swerdlow's reading is that older models "tended to play the RTS as a turn-based game, leading them to get destroyed while they were thinking." If your agent cannot convert deliberation into a better action quickly enough, the deliberation is a liability, and that holds well outside of video games.

12.6 actions per minute

The winning configuration issued 12.6 actions per minute. It is worth putting that number next to the last time a machine took StarCraft seriously.

AlphaStar averaged roughly 280 APM with a deliberate 350ms observation-to-action delay, and beat professionals MaNa (390 APM) and TLO (678 APM) on the way past 7000 MMR, above Grandmaster. That work was published in Nature in 2019.

Be precise about what that comparison does and does not say. AlphaStar was reinforcement learning trained on the game for the game. These are general models handed two tool calls and asked to improvise. Nobody spent a training run on Brood War here, and the benchmark is not a replication of anything.

APM alone does not order the board either. The highest action rate on the leaderboard belongs to Codex 5.6 Terra at low effort, 48.3 APM, and it finished 8-10. Speed without a plan is just more clicking.

Cheese beat macro, and the subagents did not talk

Codex's most reliable idea was harassment. In Protoss games it repeatedly sent a Probe across the map to attack workers and buildings, which worked embarrassingly well because opposing agents "spent dozens of seconds thinking about what to do about a probe instead of doing anything else." One worker, sent early, effectively froze an opponent.

What the same systems could not do was build. They delayed tech, trickled one or two basic units into defended bases, and threw workers into last stands.

The most transferable finding is structural. Swerdlow observed that Codex often spun up separate subagents for economy, army production, and army control, and that they barely communicated. The army agent therefore sent each new unit straight into an attack, unaware of the larger force the production agent was assembling. That is not a StarCraft bug. That is every unsupervised multi-agent decomposition you have ever shipped, rendered as pixels dying one at a time.

Claude Fable went the other way and actually played the game, reaching Lair, Spire, and Mutalisks to win G007, and stacking a Robotics Facility, Citadel of Adun, Observatory, and Templar Archives before winning G027. Ambition did not always convert: in G036 it reached a Factory and Academy and Claude Opus 5 ran it over.

And in G009, after losing its army and its main base, Codex 5.6 Terra at medium effort lifted its last Command Center and flew it toward the opposite corner, surviving another six minutes. Not a strategy, but you have to respect the commitment.

What this does not show

Read the caveats before you quote the leaderboard anywhere.

• It is one person's self-published benchmark. The report is thorough and unusually candid, but there is no third-party replication and no released results dataset at the time of writing.
• The cells are not uniform. Top configurations played 18 games; several at the bottom show 16 or 17 decisions, so records are not all drawn from an identical schedule.
• Cost per game is indicative, not a price list. Swerdlow ran subscription harnesses specifically to avoid metered API pricing, so treat $0.16 for Codex 5.6 Luna and $21.07 for Codex Astra at low effort as relative signals.
• The ceiling is low and the author says so first. His own summary is that even Astra and Fable could not build complex armies, defend simple attacks, or execute a concrete strategy.

None of that makes the exercise less useful. A benchmark whose author leads with "none of the models played beyond a beginner level" is doing something most leaderboards refuse to do.

Key Takeaways

• Codex Astra at xhigh effort won Brood War Bench 18-0 with a 100% win rate, 12.6 APM, and $10.54 per game; Claude Fable took third at 83.3%.
• Real-time play converts reasoning latency into a loss condition. Grok 4.6 at xhigh burned 11,138 reasoning tokens for six command batches in 43 minutes and never fielded a combat unit.
• Higher effort settings helped the strongest model monotonically and hurt weaker ones. Codex 5.6 Sol scored its worst result, 61.1%, at its most expensive setting.
• Uncoordinated subagents were a visible failure mode: separate economy, production, and army agents that did not communicate fed units into attacks one at a time.
• The best agent issues about 12.6 actions per minute against AlphaStar's roughly 280 in 2019, a reminder that general tool-calling agents and purpose-trained RL are not the same category of thing.
• Every configuration remained below beginner level by the author's own assessment, so the headline is the gap, not the ranking.

Sources: Brood War Bench report, Ben Swerdlow, Hacker News discussion including the author's harness notes, Freestyle, BWAPI on GitHub, AlphaStar, Google DeepMind, Vinyals et al., Nature 575 (2019)