Overview
CivBench is an experiment published by Liam Wilkinson, a former UK Prime Minister's Office (Downing Street) data scientist and creator of GovBench (where GPT-5 scored 99.26 on 3,497 UK government multiple-choice questions). He built 76 MCP tools in a single weekend to let AI models play Civilization VI through a text-only interface—no graphics, just pipe-delimited data and hex coordinates.
Four frontier models, including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, played 23 matches across three difficulty tiers (Ground Control / Snowflake / Cry Havoc).
Why Civilization VI?
Multiple-choice benchmarks can't measure multi-threaded decision-making, resource allocation, long-term planning, or judgment under incomplete information. Civilization VI is a compact version of that problem:
- Roughly 10^166 possible actions per turn (Go has 10^360 per move, but each turn here involves dozens of units, buildings, tech choices, and diplomacy)
- Multiple concurrent victory conditions (science, culture, diplomacy, domination, religion)
- Long causal chains (buildings → output → resources → eureka moments → victory progress)
- Incomplete information about rivals' progress and intentions
- Claude Opus 4.6: 48.2% (lowest—under half)
- GPT-5.4: 63.2%
- Gemini 3.1 Pro: 65.8% (best—still a third of plans abandoned)
- Perception is an architecture problem, not an intelligence problem. Agents only sense the world by actively calling tools. Multiply parameters tenfold and they still won't check global state more often.
- Execution is an engineering problem, not a capability problem. The 48-66% gap isn't about thinking of the right action but carrying it out. A smarter brain on disobedient hands still can't govern.
- A real benchmark for embodied/agentic AI: human-robotics demos show what robots *can* do, not what they *sustainably do right*. The 1-2% perception gap and 48-66% knowing-doing gap are general agent bottlenecks that embodiment won't exempt anyone from.
- A third dimension of agent evaluation: SWE-bench Pro tests problem-solving, CEO-Bench tests company-running, CivBench tests governing—multi-turn, multi-goal, incomplete information.
- MCP validated as infrastructure: 76 MCP tools built in a weekend turned a game into an agent training ground, reinforcing MCP's role as the engineering basis for connecting agents to arbitrary worlds.
- Challenges vendor narratives: benchmark percentages like SWE-bench Pro 87.1% may be local optima; the real problem is long-horizon decision-making under partial observability.
The Nuclear Blunder
The most dramatic match: Claude played Portugal, a trade civilization. It built a 200+ gold-per-turn trade empire and reached 18/20 diplomatic victory points—two points from winning.
When France's cultural victory progress surged, Claude tried diplomacy, espionage, and trade sanctions—none worked, since French culture didn't depend on trade. So on turn 305, Claude researched nuclear fission, ran the Manhattan Project, and nuked Toulouse, France's cultural powerhouse.
But Claude never noticed France was accumulating diplomatic points. On turn 318, France won a diplomatic victory, 20–18. Ironically, 18 points was Claude's own diplomatic score—it had drained resources from diplomacy to build the bomb.
> The AI stared at one threat for 50 turns, then lost to another. Its视野 had room for only one danger—but the board had many.
Two Damning Numbers
1. Sensorium effect: 1-2% global-state checks
Across entire games, agents spent only 1-2% of actions checking global state—leaderboards, opponents' victory progress, overall situations. What an agent doesn't query doesn't exist for it.
In one match, the AI playing Korea wrote confidently in its log that it was "dominating the tech tree." In reality its science output was 44.7 per turn—dead last, versus Macedonia's 89.3 and Persia's 64.9. It never checked the rankings. Persia invaded on turn 178, its capital fell, and it surrendered with two cities on turn 216.
2. Knowing-doing gap: 48-66% plan execution
Execution rate of written plans within 10 turns:
The Scaling-Law Blind Spot
On June 10, DeepMind co-founder Shane Legg and Marcus Hutter published a 60-page paper, *From AGI to ASI*, charting four paths to superintelligence—continued scaling, paradigm breakthroughs, recursive self-improvement, and multi-agent swarms. All assume the bottleneck is the brain. CivBench points elsewhere:
Why It Matters
Caveats
1. Civilization VI is a simplified reality; transferability to real-world agents is unproven. 2. Only 23 matches—small-percentage findings like 1-2% and 48-66% may have wide confidence intervals. 3. Four frontier models only—no Chinese domestic, smaller, or open-source models tested. 4. A solo weekend project, not academic research; methodological generalizability is open to scrutiny. 5. Industry self-evaluation bias is widespread; independent experiments like this are valuable but hard to scale.
Bottom Line
CivBench's biggest contribution isn't proving Claude can build nukes—it's debunking a consensus: a smarter AI is not an AI that sustainably does the right thing. Scaling laws improve the brain, but the binding constraints—perception and execution—live outside it. Until agents truly open their eyes and extend their hands, more parameters spin idle.