AI-TechScienceGovernmentBusinessSportsEntertainment
AI-Tech

Language models struggle to survive the real-time demands of StarCraft

Leading artificial intelligence models failed to progress past beginner play in StarCraft: Brood War because protracted reasoning cycles left them vulnerable to continuous real-time pressure.

This article has been updated · 3 Nov see what changed
Language models struggle to survive the real-time demands of StarCraft
Source: Primary generated image. © 2026
Published3 Nov 2026, 13:10 Last updated3 Nov 2026, 13:13 Sources
Source mode Marks each sentence drawn from a source or a contributor

A real-time strategy game gives no quarter to deliberation. When opposing forces advance across an open terrain, the clock does not pause to let a competitor ponder alternative responses. Success requires an unbroken loop of continuous action: workers must gather minerals, production structures must produce fighting units without interruption, technology trees must be unlocked in sequence, and scouts must track enemy movements. Any hesitation leaves an army outnumbered, because every second spent deliberating without issuing commands is a second conceded to an active adversary.

Artificial intelligence systems trained on vast text corpora have demonstrated impressive proficiency in turn-based reasoning, where they can ponder a puzzle for as long as their computational budget permits. Real-world tasks, however, frequently demand fast decisions inside environments where the ground shifts while the system is still calculating. Testing whether large language models can act as autonomous agents in complex, time-pressured environments reveals how fragile their practical coordination remains when an opponent refuses to wait.

A public benchmark evaluation called Brood War Bench, released on September 19, 2026, by developer Ben Swerdlow, pitted leading language models against one another in StarCraft: Brood War.1 Swerdlow reported that none of the tested configurations played beyond a beginner level.1 Even the strongest models failed to sustain production economies, coordinate multi-unit assaults, or execute fundamental defenses against simple aggression.1

Why does real-time strategy expose the weaknesses of reasoning models?

Real-time strategy games expose reasoning weaknesses because thinking carries an immediate physical cost in lost game time. In a match of StarCraft, a player must first harvest minerals and vespene gas with worker units. Next, the player must reinvest those resources into production facilities, such as Barracks or Gateways. Those facilities must continuously manufacture combat units to assemble a critical mass of troops before initiating an attack. Simultaneously, the player must respond to enemy harassment without pausing base construction. If an automated system enters a protracted reasoning cycle before dispatching orders, its workers sit idle and its base remains undefended.

Swerdlow observed that older and more deliberate models regularly treated the real-time environment as though it were turn-based, leading them to be dismantled while generating reasoning tokens.1 Explaining this computational bottleneck, Ben told Primary that "Grok would regularly notice it was under attack and spend minutes deciding what to do without acting. By the time it decided to act the original units it wanted to move were often dead and it spent more minutes thinking about what to do instead." Systems running under the Grok 4.6 family suffered severely from this paralysis.1 In game G043 of the benchmark, the Grok 4.6 extra-high effort configuration generated 11,138 reasoning tokens over 43 minutes of match play, yet issued only six command batches and failed to train a single combat unit.1 Ben noted in response to questions from Primary that the model stalled from an "Analysis Paralysis" where it became "caught up in considering actions rather than taking them." In game G003, the same extra-high configuration produced three Marines but never sent them across the map toward the opposing base.1 In game G002, Grok 4.6 running on medium effort trained two Zealots and similarly never crossed the map to engage its opponent.1

Across the complete tournament, Grok 4.6 on extra-high effort recorded an action rate of 2.8 actions per minute across 18 matches, finishing with two wins, 15 losses, and one match that reached the time limit, representing a win rate of 11.1 percent at an average cost of $0.66 per game.1 The medium-effort configuration of Grok 4.6 managed 3.2 actions per minute across 18 matches, ending with one win, 16 losses, and one time-limit draw for a 5.6 percent win rate at $0.79 per game.1 Grok 4.6 on low effort finished with zero wins, 16 losses, and two time-limit draws out of 18 games, logging 4.2 actions per minute and an average cost of $1.26 per game.1

How did different language models handle Brood War?

Different language models diverged sharply in their tactical approaches, with some favoring immediate disruption and others attempting to scale their economies. Swerdlow reported that Codex Astra established itself as the tournament leader by finding disruptive tactics before mastering economic expansion.1 Playing as the Protoss faction, Codex Astra frequently dispatched an early Probe, a basic worker unit, directly across the map to harass enemy workers and structures.1 This early disruption repeatedly crippled opposing language models, which spent dozens of seconds deciding how to handle the intruder rather than building their own bases.1

Language models struggle to survive the real-time demands of StarCraft
Source: GitHub

Codex Astra running on the extra-high effort setting captured first place in the benchmark, recording 18 wins and zero losses across 18 matches for a 100.0 percent win rate, maintaining 12.6 actions per minute at an average cost of $10.54 per game.1 The medium-effort setting of Codex Astra secured second place, finishing with 16 wins and two losses across 18 matches for an 88.9 percent win rate, averaging 17.2 actions per minute at a cost of $15.11 per game.1 Operating on low effort, Codex Astra achieved 14 wins and four losses across 18 games to take fourth place, posting a 77.8 percent win rate, 25.7 actions per minute, and an average cost of $21.07 per game.1 Ben noted to Primary that smarter models cost less because their ability to plan ahead minimally allowed them to "spend less time during the game considering everything."

Claude Fable adopted a noticeably different operational style, which Swerdlow described as an earnest attempt to climb the technology tree, the hierarchical progression of unlocking advanced structures and units.1 In game G007, Fable constructed a Lair and a Spire to field flying Mutalisk units, securing a victory.1 In game G027, playing as Protoss, Fable erected advanced structures including a Robotics Facility, a Citadel of Adun, an Observatory, and a Templar Archives before closing out the win.1 This ambition did not always succeed; in game G036, Fable developed a Terran Factory and Academy, but an assault from Claude Opus 5 overran the base before those assets could mature.1 Overall, Claude Fable finished third in the tournament with 15 wins and three losses across 18 matches, registering an 83.3 percent win rate, 12.6 actions per minute, and an average game cost of $12.24.1

What prevented even the best systems from playing beyond a beginner level?

Basic coordination failures between specialized sub-components prevented even the leading models from mounting sophisticated military campaigns. Swerdlow observed that Codex Astra frequently partitioned its gameplay among distinct subagents, specialized internal software processes assigned separately to manage resource gathering, unit production, and combat movement.1 Because these subagents communicated poorly with one another, the military subagent habitually directed individual combat units to charge straight at the enemy base the moment they completed training.1 Asked about this breakdown, Ben said "The Codex army subagent always assumed the army it had was what it was going to get." This behavioral disconnect repeated a classic beginner mistake known as unit trickling, feeding single units into established enemy defenses rather than grouping them into a decisive strike force.

The benchmark tournament operated as a round-robin matrix where 19 model and effort configurations played every other configuration in parallel virtual machines hosted on Freestyle infrastructure.1 Explaining the environment setup, Ben told Primary that "every single game ran 4 VMs" including dedicated machines for each agent, the game, and a data visualizer connected over a private network. Match duration statistics reflected the pace of play: games involving Codex Astra reached a median duration of 8 minutes and 10 seconds across 51 recorded player-runs, while matches featuring Claude Fable showed a median duration of 10 minutes and 37 seconds across 18 player-runs. Grok 4.6 recorded a median match duration of 9 minutes and 15 seconds across 51 player-runs, with 7.8 percent of its games stretching beyond 40 minutes.

These experimental trials provide a specific measurement of autonomous agent performance under strict interface constraints, rather than an absolute assessment of machine intelligence. The benchmark evaluated models interacting through an agent harness, operating without human intervention, and the costs reported for Codex and Claude Sonnet models represent token-based estimates rather than universal billing standards.1 Furthermore, the round-robin format tested agents solely against one another rather than against human ladders or dedicated neural networks such as AlphaStar. Swerdlow noted that a novice human player relying on a elementary photon cannon rush, an aggressive early-game static defense tactic, would comfortably defeat every agent tested on the platform.1 When asked why current models would fail against basic human pressure, Ben said they were "Too slow, and none are smart enough to build complex strategies today."

What would demonstrate genuine strategic competence in future models?

Future benchmarks will require automated players to process dynamic reconnaissance and adjust tactics based on enemy actions. Current language models operate largely through open-loop planning, producing predetermined build orders rather than reacting to tactical choices made across the map. When asked what milestone would prove that language models can truly handle real-time strategy, Ben told Primary that "the real sign for me would be response to feedback." Ben observed that the closest example occurred when Fable detected an absence of anti-air defenses and produced flying units, noting that seeing agents deduce an adversary strategy from partial scouting "and responding accordingly that would convince me of a real shift."

Bridging the divide between abstract deliberation and real-time execution remains an active challenge for artificial intelligence developers. While frontier models can plan multi-step sequences in isolation, integrating those plans into continuous, adversarial environments demands tighter internal communication and a constant awareness of time as an expiring resource. Until automated agents learn to coordinate their internal processes without freezing under pressure, real-time strategy will continue to present a barrier that pure linguistic reasoning cannot easily breach.

Update

3 Nov 2026, 13:13 UTC — This article was updated to reflect new contributions and sources.

The record

36 sentences trace to 1 source and 1 contributor.

  1. 1 BS Ben Swerdlow Document · bw.swerdlow.dev · 19 Sep 2026 Brood War Bench See the source
  2. 2 BS Ben Swerdlow Contributor · interview 27 Sep 2026 · in the article since 3 Nov 2026 Contribution — Ben Swerdlow 17 statements added to this article

Article history

  1. Update 3 Nov 2026, 13:13

    This article was updated to reflect new contributions and sources.

  2. Published 3 Nov 2026, 13:10
    Assembled by the Primary desk from 1 source · 1 contributor · 36 cited sentences
  3. Ben Swerdlow 17 statements added 38 days before publication
    1. Go to the sentence: Even the strongest models failed to sustain production economies, coordinate multi-unit assaults, or execute fundamental defenses against simple aggression.
    2. Go to the sentence: Swerdlow observed that older and more deliberate models regularly treated the real-time environment as though it were turn-based, leading them to be dismantled while generating reasoning tokens.
    3. Go to the sentence: Explaining this computational bottleneck, Ben told Primary that "Grok would regularly notice it was under attack and spend minutes deciding what to do without acting. By the time it decided to act the original units it wanted to move were often dead and it spent more minutes thinking about what to do instead."
    4. Go to the sentence: Systems running under the Grok 4.6 family suffered severely from this paralysis.
    5. Go to the sentence: In game G043 of the benchmark, the Grok 4.6 extra-high effort configuration generated 11,138 reasoning tokens over 43 minutes of match play, yet issued only six command batches and failed to train a single combat unit.
    6. Go to the sentence: Ben noted in response to questions from Primary that the model stalled from an "Analysis Paralysis" where it became "caught up in considering actions rather than taking them."
    7. Go to the sentence: Operating on low effort, Codex Astra achieved 14 wins and four losses across 18 games to take fourth place, posting a 77.8 percent win rate, 25.7 actions per minute, and an average cost of $21.07 per game.
    8. Go to the sentence: Ben noted to Primary that smarter models cost less because their ability to plan ahead minimally allowed them to "spend less time during the game considering everything."
    9. Go to the sentence: Because these subagents communicated poorly with one another, the military subagent habitually directed individual combat units to charge straight at the enemy base the moment they completed training.
    10. Go to the sentence: Asked about this breakdown, Ben said "The Codex army subagent always assumed the army it had was what it was going to get."
    11. Go to the sentence: The benchmark tournament operated as a round-robin matrix where 19 model and effort configurations played every other configuration in parallel virtual machines hosted on Freestyle infrastructure.
    12. Go to the sentence: Explaining the environment setup, Ben told Primary that "every single game ran 4 VMs" including dedicated machines for each agent, the game, and a data visualizer connected over a private network.
    13. Go to the sentence: The benchmark evaluated models interacting through an agent harness, operating without human intervention, and the costs reported for Codex and Claude Sonnet models represent token-based estimates rather than universal billing standards.
    14. Go to the sentence: Swerdlow noted that a novice human player relying on a elementary photon cannon rush, an aggressive early-game static defense tactic, would comfortably defeat every agent tested on the platform.
    15. Go to the sentence: When asked why current models would fail against basic human pressure, Ben said they were "Too slow, and none are smart enough to build complex strategies today."
    16. Go to the sentence: When asked what milestone would prove that language models can truly handle real-time strategy, Ben told Primary that "the real sign for me would be response to feedback."
    17. Go to the sentence: Ben observed that the closest example occurred when Fable detected an absence of anti-air defenses and produced flying units, noting that seeing agents deduce an adversary strategy from partial scouting "and responding accordingly that would convince me of a real shift."