Open-Source warcraftsim Beats Warcraft III's Insane AI in 89% of Games on One RTX 3090

TL;DR

Robin Kahlow published warcraftsim, an MIT-licensed harness that runs the real, unmodified Warcraft III (The Frozen Throne, patch 1.29) headless under Wine and steps it from Python. On top of it he trained a small self-play agent on one desktop (an RTX 3090 and 16 CPU cores). In about three weeks it went from losing to the easy AI to winning 89% of its test games against the built-in AI on insane, the hardest setting, on the ladder map Echo Isles. The write-up and the experiment log record every run, including the many that failed.

What he built

Warcraft III has no bot API, no headless mode and no way to run faster than real time. It is a 32-bit Windows program from 2002, and reimplementing its pathfinding, spells and AI scripts faithfully would take longer than the RL work. So Kahlow runs the original game and bolts three pieces onto it:

• A harness written in JASS, the game's own map-scripting language, injected into the map. Every half second of game time it reports what the player can see and issues the agent's orders through the game's native functions.
• A shim, w3shim.dll, loaded into the game process. It swaps the game's clocks for a virtual one, so the game runs as fast as the CPU allows and freezes at every step until Python answers.
• Config files that start a local game with no menus: map, players and AI difficulty.

Because the games are deterministic, a replay reproduces a game step for step, which is how every video in the post was made. The repo ships no Blizzard files, so you need your own copy of The Frozen Throne on patch 1.29 to run it. Reforged and the current 3.0 release sit behind Battle.net, and Kahlow did not attempt them.

Making a 2002 game run at datacenter pace

Most of the speed came from deleting waste on the same hardware. Each step in the chart below adds one optimization on top of the ones before it, measured in agent steps per second (one decision by one player).

Today the desktop plays about 50 games at once, each roughly 15 times faster than real time. Kahlow puts that at about 730 hours of Warcraft III per wall-clock hour. Over the whole project the agent played about 11 years of the game across 840,000 games. The README adds one footnote: a second machine, a laptop under WSL2, plays the evaluation games so verdicts come back in minutes.

The agent is tiny

One network plays one side, seeing only what that player sees through the fog of war. Up to 160 units in view become tokens, plus a player token for gold, lumber, supply and upgrades, and everything goes through a 5-layer transformer. A minGRU memory carries state between steps. For each of its own units, an order head picks one of 399 orders (or none), and target heads point at a unit in view or a map point, picked as x and then y on a 352-cell grid.

The whole thing is about 3.2 million weights. One decision for 60 units with 120 in view takes about 13 ms on four CPU cores, against a budget of half a second. Your laptop could run the finished policy while also running Slack, which is a higher bar than most laptops clear.

How it learned

Random orders almost never win, so the agent starts by copying. Kahlow recorded thousands of games of the built-in AI playing itself and trained the agent to imitate its orders (behavioral cloning). Then it switches to self-play against a league of itself, its past snapshots and the built-in AI, using policy gradients plus a KL penalty that pulls it back toward an earlier "anchor" version of itself.

The hard part is credit assignment. A 25-minute game at two decisions a second is about 3,000 decisions per side before the single win-or-loss signal shows up. Even the cloned agent lost all 40 of its first test games on the real map. So training ran as a curriculum: seconds-long micro fights (91% against a scripted opponent), then 2-minute games on a small duel map with rules sped up 7x (79% against the built-in AI), then 6-minute games (80% against insane), then Echo Isles with fast rules, and finally Echo Isles under the real rules.

Two tricks from the early stages are worth stealing. The first is a tax on the built-in AI: it loses a share of the gold and lumber it gathers, the share rises after each agent loss and falls after each win, and test games never carry the tax. Over training the tax fell from about 75% to almost zero. The second is on-policy distillation. In half the training games the built-in AI "shadows" the agent and records what it would have ordered in each state the agent reached, and training nudges the agent toward those orders. Think of a driving instructor in the passenger seat who never touches the wheel but grumbles at every wrong turn. That took the agent from 1 win in 98 test games to 13 in 50 within fifty minutes. Once the student passed the teacher (about 60% against insane), switching the coaching off made it stronger and faster to train.

Six changes then carried the real-rules win rate from 49% to 89%, each measured on a few hundred held-out games against the untaxed insane AI:

• Look further ahead. At a discount of 0.999 per half-second step, a win eight minutes out counted for about a third of a win now. Raising it to 0.9997 stretched the horizon to about 28 minutes, and half the test games now ended within 13 minutes instead of 18.
• Find room to build. Spots learned on the flat duel map landed in Echo Isles' trees and cliffs, and the game refused three of four undead buildings. The harness now searches three times farther (768 units instead of 256) for a free spot.
• Halve the shaping reward. The agent also earns potential-based rewards for changes in its material lead, so training a footman and losing it nets zero. Once the longer horizon made the win itself count early, halving that weight helped most for the weakest races.
• Loosen the anchor. Kahlow measured the per-update reward push at about 1% signal and 99% noise, while the KL pull was clean. Cutting the pull to a quarter did nothing for ninety minutes, then pulled ahead.
• Give it a plan. Borrowing from AlphaStar, half of training games come with a target build order drawn from 8,072 real ones (from the built-in AI's games and human replays), fed in as an input with a small reward for following it.

The credit-assignment fix from the six-minute stage, UPGO, also comes from DeepMind's AlphaStar. The micro-fight trainers use PufferLib 5.0 alongside Kahlow's own torch trainer.

The compute gap

AlphaStar trained a league of 12 agents for 44 days on 32 TPUs each, according to DeepMind's 2019 Nature work. OpenAI Five trained for 10 months. Both had the game makers' cooperation and beat top human players. warcraftsim had one consumer GPU from 2020, an unmodified retail binary and about three weeks, from 27 September to 18 October.

The comparison flatters the solo project if you stop reading there, and Kahlow does not stop there. His opponent is the game's built-in script, and pros beat it reliably. That is a much lower bar, which goes a long way toward explaining why one desktop was enough.

The caveats, straight

• It has one strategy. It wins with an early rush of cheap units, half of its wins inside ten minutes. Across the best agent's 183 long losses (27 minutes on average) it fielded 35 workers to the AI's 16 but only two or three unit types to the AI's nine, and three upgrades to the AI's seventeen.
• Heroes and macro are weak. It raises about one hero a game against the built-in AI's 1.5 or more, rarely creeps or expands, and never buys items or hires mercenaries. Even with a build order enforced, it makes only about a third of the target's buildings, units and upgrades.
• Insane wins on income, not smarts. After a Reddit comment, Kahlow measured that insane gathers gold and lumber twice as fast as a normal player, and added an update on 18 October. Every test game was against that double-income AI, so the agent is out-playing a richer opponent that still follows a script.
• Humans are untested. Good human players reliably beat insane. Kahlow expects a human would see the rush coming and hold it off, and the long game after that is where the agent is weakest.
• One map, one patch. Only Echo Isles on 1.29 so far.

Why builders should care

Reinforcement learning on real software usually stalls on the environment. You cannot run the target fast enough, so you end up reimplementing it badly. warcraftsim shows the other route. Leave the binary alone, intercept its clock, script its I/O from the inside, and spend your time on throughput. The same shape (a virtual-clock shim plus an in-process scripting bridge) applies to plenty of closed legacy programs that will never ship an API.

The other lesson is in the log. Most of the gain came from small, measured, one-change experiments against a fixed held-out opponent, and most experiments failed. Kahlow lists them anyway: hero-level rewards, a doubled learning rate, a bigger batch, an exploiter agent, a moving anchor and more.

Next on his list are more varied strategies, exploiter agents to stop the league from forgetting, more maps (patch 1.29 ships Lost Temple, Turtle Rock and others) and, eventually, good human players.

Key Takeaways

• warcraftsim (MIT) runs the unmodified Warcraft III 1.29 under Wine, about 50 games at once, each roughly 15x real time, driven step by step from Python through a virtual-clock DLL and a JASS harness.
• A 3.2M-weight transformer agent, trained with behavioral cloning, league self-play and a curriculum on one RTX 3090 and 16 CPU cores, wins 89% of test games against the insane built-in AI on Echo Isles.
• The jump from 49% to 89% under real rules took three days and six single-change experiments: a longer discount, a wider build search, a halved shaping reward, a looser KL anchor and target build orders.
• Throughput rose from 560 to 2,280 agent steps per second through software fixes alone, and the agent played about 11 years of Warcraft III across 840,000 games.
• The opponent is a script with double income, the agent plays one rush strategy, and it has not faced humans. Treat it as a strong environment and a training recipe, not a superhuman player.

Sources: Robin Kahlow: Beating Warcraft III's insane AI with self-play reinforcement learning on one desktop, GitHub: RobinKa/warcraftsim, warcraftsim experiment log, warcraftsim experiments, Google DeepMind: AlphaStar, OpenAI: Dota 2 with Large Scale Deep Reinforcement Learning, Were RNNs All We Needed? (minGRU), PufferLib