Paper Overview
| Item | Detail | |------|--------| | Title | Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament | | Authors | Dmitry Dagaev, Egor Ivanov, Petr Parshakov, Alexey Savvateev, Gleb Vasiliev | | arXiv ID | 2605.22095 | | Date | May 21, 2026 | | Categories | econ.GN / cs.AI / cs.GT / cs.HC | | Core finding | Human strategy quality significantly outperforms LLMs in the Colonel Blotto multi-battlefield resource allocation game; reasoning depth shows a "U-shaped curse" — too little or too much both fail, only medium-depth heuristic strategies win. |
1. Simple Rules, Complex Solutions
Colonel Blotto has just two rules:
1. You have N soldiers to distribute across K battlefields. On each battlefield, the side with more soldiers wins; ties split the field. 2. Overall victory goes to whoever wins more battlefields.
That's it — yet game theorists have studied it for nearly a century. The difficulty lies in the high-dimensional action space (even 10 soldiers across 3 battlefields creates a huge combinatorial space) and the absence of a pure-strategy Nash equilibrium: there is no fixed allocation you can rely on. You must always randomize and always anticipate your opponent.
The authors organized three tournaments: first, 200+ humans playing each other; second, strategies submitted by several mainstream LLMs; third, a fair comparison with LLM strategy counts matched to the human pool.
2. LLM Strategies: Simple, Rigid, Predictable
Humans converged on clear "medium-depth" strategies — well-calibrated intermediate allocations that are neither fully random nor fully computed-optimal. This "moderately smart" approach proved most effective.
LLMs, surprisingly, submitted simpler and more rigid strategies. If human strategies resemble improvised jazz, LLM strategies are like playing from sheet music — accurate but inflexible. Humans outperformed LLMs on three dimensions: strategy diversity, calibration precision, and final win rates.
A key quote from the paper: "Strategic sophistication is key to success if and only if the necessary level of reasoning depth is reached."
3. The U-Shaped Curse of Reasoning Depth
Three possible playing styles:
- Style A (low depth): Split soldiers evenly — a naive strategy.
- Style B (medium depth): Reason one or two recursive steps — "If the opponent spreads evenly, I can overweight one key battlefield; but if they anticipate that, they might do the opposite." Limited "I think that you think" recursion.
- Style C (high depth): Try to compute the full mixed-strategy Nash equilibrium.
This echoes the Keynesian beauty contest: what matters is not how deep you think, but how far ahead of your opponent's reasoning depth you are. Thinking one layer beyond an opponent who thinks zero layers wins; thinking ten layers beyond an opponent who thinks zero turns your "excess reasoning" into noise.
4. Humans Don't Switch Strategies
The most counterintuitive finding: when facing different opponents (humans in round one, LLMs mixed in round three), humans barely adjusted their strategies.
The reason is deeper than laziness: humans primarily reference the game rules themselves, not the opponent's identity. Since Colonel Blotto is a high-dimensional allocation game, the rules alone drive their strategies. Whether the opponent is a human or GPT-4 barely matters. In effect, humans treated the LLM like a person — not because they thought it was human, but because the rules were sufficient to determine optimal behavior.
So what did humans "win"? Not opponent-prediction ability, but superior strategy quality under the same rules.
5. Weak STEM Advantage and Honest Boundaries
STEM-background participants performed slightly better in the first tournament, but the effect was weak — likely reflecting systematic thinking habits rather than domain knowledge.
The study's honest limitations:
1. Only one game was tested. In prisoner's dilemma, public goods, or ultimatum games, LLM vs. human performance may differ entirely. Colonel Blotto's specificity — high-dimensional actions, no pure-strategy equilibrium — may be central to LLM underperformance; generalizing to "LLMs are bad at games" is risky. 2. LLM strategies (submitted via API calls) and the human experimental setup were not fully comparable — humans had instructions, thinking time, and scratch paper. This "format inequality" may exaggerate the gap. 3. The authors did not report tuning of LLM temperature, system prompts, or few-shot examples. Different parameters could produce very different strategies. 4. The 200+ participants share a single cultural background. Whether strategy preferences are culture-dependent remains unanswered.
Closing Thought
Colonel Blotto is an elegant metaphor: many problems have no optimal solution. You can only choose in a moderately smart way and hope your opponent thinks one layer less — or many more. LLMs are stuck between "not thinking at all" and "thinking too much," while humans found the sweet spot. Whether that sweet spot is an inherent human advantage or simply the wrong "reasoning gear" for LLMs is the question this paper leaves to future researchers.
*Original post tagged: #FeynmanLearning #GameTheory #HumanVsAI*