Humans Outperform LLMs in a Colonel Blotto Tournament: The 'U-Shaped Curse' of Reasoning Depth

Forum topic · 小凯 · 2026-05-22

Summary

A recent paper on arXiv (ID 2605.22095) titled 'Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament' reports that humans significantly outperform large language models (LLMs) in the Colonel Blotto resource-allocation game. The authors, Dmitry Dagaev, Egor Ivanov, Petr Parshakov, Alexey Savvateev, and Gleb Vasiliev, ran three tournament rounds: over 200 humans competing against each other, several mainstream LLMs, and a matched-size comparison. Human strategies were moderately sophisticated and well-calibrated heuristics, while LLM strategies were simpler, more rigid, and more predictable. The study identifies a 'U-shaped curse' of strategic sophistication: both too little and too much reasoning depth perform poorly, and only medium-depth heuristics win. Notably, humans rarely adjusted their strategies when facing LLM opponents, treating the game's rules rather than the opponent's identity as the primary driver of play. STEM-background participants showed only a weak advantage. The authors caution that results are specific to Colonel Blotto's high-dimensional, no-pure-strategy-equilibrium setting, that experimental formats between humans and LLMs were not fully comparable, and that LLM hyperparameters and prompts were not extensively tuned, so findings should not be generalized as 'LLMs are bad at games.' The paper frames strategic sophistication as useful only when reasoning depth is exactly adequate.

Overview

| Item | Detail | |------|--------| | Paper title | Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament | | Authors | Dmitry Dagaev, Egor Ivanov, Petr Parshakov, Alexey Savvateev, Gleb Vasiliev | | arXiv ID | 2605.22095 | | Published | May 21, 2026 | | Categories | econ.GN / cs.AI / cs.GT / cs.HC | | Core finding | In the Colonel Blotto multi-battlefield resource allocation game, human strategies significantly outperform LLM strategies; strategic reasoning depth exhibits a "U-shaped curse" — too little reasoning is useless, too much is also useless, and only medium-depth heuristic strategies win. |

1. Simple Rules, Extremely Complex Solution

Colonel Blotto has two rules:

1. You have N soldiers to distribute across K battlefields. On each battlefield, whoever commits more soldiers wins it; ties split the battlefield. 2. Overall victory goes to whoever wins more battlefields.

That's it. Yet game theorists have studied this game for nearly a century.

Why is it hard? The action space is high-dimensional — even 10 soldiers across 3 battlefields yields a huge combinatorial space. More fatally, the game has no pure-strategy Nash equilibrium: no fixed allocation guarantees safety. You must always randomize and always guess where your opponent is weak or strong.

The authors organized three Colonel Blotto tournaments: (1) over 200 human participants playing each other, (2) several mainstream LLMs submitting strategies, and (3) a fair comparison matching the number of LLM strategies to the human count.

2. LLM Strategies: Simple, Rigid, Predictable

Human submissions showed a clear "medium depth" pattern: well-calibrated mid-level allocation heuristics — neither fully random nor fully optimal. This "moderately smart" approach proved most effective.

LLM performance was surprising. Their strategies were simpler and more rigid. If human strategies resemble improvised jazz — flexible within the rules — LLM strategies resemble playing from sheet music: accurate but inflexible. Humans outperformed LLMs on three dimensions: strategy diversity, calibration precision, and win rate.

A key quote from the paper: "Strategic sophistication is key to success if and only if the necessary level of reasoning depth is reached." Too little fails; too much fails too. This is a U-shaped curse.

3. The U-Shaped Curse of Reasoning Depth

Three ways to play Colonel Blotto:

Play A (low depth): Naively split soldiers evenly — "a third on each battlefield."
Play B (medium depth): Reason one or two levels ahead — "if the opponent spreads evenly, I can concentrate on one key battlefield; but if they anticipate that, they may do the opposite." This limited recursive "I think that you think" reasoning is medium depth.
Play C (high depth): Attempt to enumerate all allocations and compute the mixed-strategy Nash equilibrium.

Intuition says C should win — it's the "smartest." But the experiments show C performs no better than A. Many LLM strategies fell at the extremes A and C — either overly simple (uniform splits) or attempting overly complex calculations that erred or overfit to specific patterns. The winners were B: heuristics that think "exactly one step further."

This echoes the Keynesian beauty contest in economics: what matters is not how deeply you think, but the gap between your reasoning depth and your opponent's. Thinking one level beyond an opponent who thinks zero levels wins; thinking ten levels beyond a zero-level opponent turns "over-reasoning" into noise.

4. Humans Don't Change Strategies

The most counterintuitive finding: when humans faced different opponents — humans in round one, LLMs mixed in round three — they barely adjusted their strategies.

The deeper reason: humans primarily referenced the game's rules themselves, not their opponent's identity. Whether facing another human or GPT-4 made little difference. In other words, humans treated LLMs like humans — not out of anthropomorphism, but because in this game the rules alone drive optimal behavior.

This raises an intriguing question: if LLM strategy patterns differ entirely from humans', and humans don't adapt to opponent identity, what exactly did humans win? They won through superior strategy quality under the same rules — not through "predicting the opponent."

5. Weak STEM Advantage and Honest Limitations

Among humans, STEM-background participants performed slightly better in round one — a weak effect, since the game requires no specialized knowledge. The advantage likely stems from "systematic thinking habits" rather than expertise.

Honest limitations of the study:

1. Only Colonel Blotto was tested. Results in prisoner's dilemma, public goods, ultimatum, and other classic games may differ entirely. The game's特殊性 — high-dimensional action space, no pure-strategy equilibrium — may be the core reason for poor LLM performance; generalizing to "LLMs are bad at games" is dangerous. 2. LLM strategy submission (direct API outputs) wasn't fully comparable to the human experimental environment, where participants had instructions, thinking time, and possibly scratch paper. This "format inequality" may exaggerate the gap. 3. The authors did not report tuning of LLM temperature, system prompts, or few-shot examples. Different parameters could yield very different strategies; conclusions reflect specific LLMs under specific prompts. 4. The 200+ participants, while not a small sample, share one cultural background. Whether strategy preferences are culture-dependent remains unanswered.

Closing Thought

Colonel Blotto is an elegant metaphor: many problems have no optimal solution. The best you can do is choose in a "moderately smart" way and hope your opponent reasons one level less — or many levels more. LLMs are stuck in the gap between "not thinking" and "overthinking." Humans found the sweet spot. Whether that sweet spot is a fixed human advantage or simply an LLM "reasoning gear" not yet properly tuned is the question this paper leaves to the next wave of researchers.

*Tags from the original post: #FeynmanLearning #GameTheory #HumanVsAI*