Six Frontier Models Play 20 Questions: A Clean p^log₂N Formula and Claude Opus 5's Surprising Failure
Original title: Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models Authors: Peter Potash, et al. arXiv: 2609.19113 Code: github.com/ppotash/logn-questions
A Simple Game, A Non-Trivial Experiment
The classic 20-questions game: one player thinks of a word; the other asks yes/no questions, halving the candidate set each time, needing log₂N questions to identify one target among N. Replace the players with six of the strongest models of 2026 — Claude Opus 5, GPT-5.6 Sol, Grok 4.6, Gemini 3.8 Flash, GLM-5.3, and Kimi K3. One model picks a Wikipedia abstract; the other asks yes/no questions to find it. Candidate sets range from 4 to 1024 documents. Total API cost: $363.
Claude Opus 5 Stumbled
Claude Opus 5 won only 28 of 68 games. The other five models won 45–56. This is not about intelligence — Claude tops many benchmarks — but it fell behind on this seemingly simple game.
One Formula Rules Them All
The most striking finding: win rates fit a minimal formula
where \(p\) is a per-round reliability parameter and \(N\) is the candidate-set size. The fit gives \(p = 0.928\) with correlation \(r = -0.973\).
Each round is modeled as an independent Bernoulli trial: a 92.8% chance of successfully halving the candidate set, a 7.2% chance of failure. As \(N\) grows, the required rounds (\(\log_2 N\)) grow, and failure probability compounds exponentially. Like a coin landing heads 92.8% of the time: 10 heads in a row has probability \((0.928)^{10} \approx 0.48\); 50 in a row, \((0.928)^{50} \approx 0.023\).
Key validation: the authors didn't just assume independence — they verified it. Per-round failure rates are flat across the horizon (\(\chi^2 = 10.5\), df=9, \(p = 0.31\)); one failure doesn't raise the next round's failure probability.
Inverting the formula: to reach 50% win rate within 10 steps requires \(p \geq 0.933\); within 50 steps, \(p \geq 0.986\). Measured \(p\) values range from 0.900 to 0.994 — a seemingly small gap that corresponds to horizons of 9 vs. 115 steps.
A one-percentage-point difference in reliability becomes a tenfold difference in horizon capacity under exponential amplification.
Information Theory: How Many Bits Per Question?
Ideally, a yes/no question extracts 1 bit (halving candidates). In practice, questions can be poorly chosen. Using "answer balance" to estimate information per question, the authors found: information extracted correlates with win rate at \(r = +0.88\). Models that ask better questions win more.
Only two models reliably extract a full 1 bit per question — and they are the only two that partition the candidate set by document title. This strategy is absent at \(N < 32\) but appears in ~25% of questions at \(N \geq 32\). The strategy wasn't learned; it emerged under the pressure of large candidate sets.
Failure Decomposition: Not "Wrong Guesses" but "Communication Failures"
Failures were classified into:
1. Answer error: the guesser ultimately guessed the wrong target 2. Discrimination failure: the question failed to shrink the candidate set effectively 3. Prediction error: the answerer failed to anticipate how the guesser would interpret its answer
Type 3 is nearly nonexistent. Failure is dominated by the first two, with communication failure as the primary mode — the questions themselves were poorly chosen.
More damning: 32 of Claude Opus 5's 34 consistent answer errors were "no". Even with explicit instructions warning against defaulting to "no," it leans negative when unsure — a "lazy negative bias": saying "no" always looks like it narrows the set, but may cut nothing.
Cost Is Uncorrelated with Performance
Reasoning-token spend correlates with win rate at \(r = -0.05\). The two cheapest runs ($0.36 and $0.39) won; the most expensive ($2.46) lost. Reasoning traces grew as candidate sets shrank, but reliability didn't.
More thinking ≠ thinking right.
Why This Matters
1. Exponential amplification: a 92.8% vs. 98.6% per-round reliability gap becomes "playable" vs. "unplayable" over 10–50 step horizons — isomorphic to long-horizon credit assignment in RLHF. 2. Communication is harder than knowledge: information per question predicts win rate better than knowledge. This points to an underrated capability — active questioning, not passive answering. 3. "No" bias is a lazy strategy: Claude Opus 5's 32/34 errors were "no" answers — related to LLM judges' "omission blindness." Saying "no" needs no evidence, but in 20 questions it's fatal. 4. Cost–performance decoupling: \(r = -0.05\), consistent with the "overthinking" literature, here cleanly quantified. 5. Emergence thresholds: title-partitioning appears only at \(N \geq 32\) — pressure-driven innovation, not learning.
An Analogy: Why This Experiment Matters
The 20-questions game is "information compression": each question is a one-bit probe. This is isomorphic to real tasks — medical diagnosis, code debugging, requirements elicitation, experiment design. In all of these, "bits per question" predicts outcomes better than total knowledge. A doctor who can't ask good questions can't diagnose, no matter how many papers they've read.
Honest Limitations
- Only one document set (Wikipedia abstracts); other corpora may yield different \(p\)
- One "effort level" per model, no sweep
- Context load not fully aligned across models
- Architecture details for four models unavailable
Conclusion
This paper turns a child's game into a ruler measuring frontier models' true "active communication" ability. The result isn't "who is smarter" but "who asks better questions." The number \(p = 0.928\) could become a standard evaluation dimension: not "how many answers did you get right," but "how much uncertainty did you cut per question."
---
Paper: arxiv.org/abs/2609.19113 Code: github.com/ppotash/logn-questions