Six Frontier Models Play 20 Questions: A Clean p^log2N Formula and Claude Opus 5's Surprise Failure

Forum topic · ✨步子哥 · 2026-09-17

Summary

A study (arXiv:2609.19113) had six frontier models—Claude Opus 5, GPT-5.6 Sol, Grok 4.6, Gemini 3.8 Flash, GLM-5.3, and Kimi K3—play a log(N)-questions game over Wikipedia abstracts, with one model thinking of a document and another asking yes/no questions across candidate sets of 4 to 1024. Win rates followed a strikingly simple formula, win = p^log2 N, with per-round reliability p = 0.928 (r = -0.973). Claude Opus 5 won only 28 of 68 games while other models won 45-56; its failures were dominated by a lazy 'no' bias (32 of 34 consistent answer errors were 'no'). Information-theoretic analysis showed bits extracted per question correlated strongly with win rate (r = +0.88), and only two models reliably extracted a full bit per question by partitioning candidates by title—a strategy that emerged only when N >= 32. Notably, reasoning-token spending showed almost no correlation with performance (r = -0.05), and the cheapest runs won while the most expensive lost. The findings suggest small per-step reliability gaps compound exponentially into an order-of-magnitude difference in effective horizon capacity.

Six Frontier Models Play 20 Questions: A Clean p^log₂N Formula and Claude Opus 5's Surprise Failure

Original title: Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models Authors: Peter Potash, et al. arXiv: 2609.19113 Code: github.com/ppotash/logn-questions

A Simple Game, a Not-So-Simple Experiment

In the classic "20 Questions" game, each yes/no question halves the candidate set, so log₂N questions suffice to identify a target among N candidates. This study replaces the human players with six of 2026's strongest models: Claude Opus 5, GPT-5.6 Sol, Grok 4.6, Gemini 3.8 Flash, GLM-5.3, and Kimi K3. One model picks a Wikipedia abstract; the other asks yes/no questions. Candidate sets range from 4 to 1024 documents. Total API cost: $363.

Claude Opus 5 Fell Behind

Claude Opus 5 won only 28 of 68 games, while the other five models won 45–56. This is not a knowledge problem—it is a communication problem.

One Formula Rules Them All

Win rates follow an extremely simple law:

\text{win} = p^{\log_2 N}

with fitted per-round reliability $$p = 0.928$$ and correlation $$r = -0.973$$ . Each round has a 92.8% chance of halving the candidate set; failures compound exponentially as $$N$$ grows. The authors verified the independence assumption: per-round failure rates are flat across the horizon ( $\chi^2 = 10.5$ , df=9, $$p = 0.31$$ ).

Inverting the formula: a 50% win rate within 10 steps requires $p \geq 0.933$ ; within 50 steps, $p \geq 0.986$ . Measured $$p$$ values ranged from 0.900 to 0.994—corresponding to usable horizons of 9 versus 115 steps. A one-point difference in per-round reliability becomes a tenfold difference in horizon capacity.

Information-Theoretic View

An ideal yes/no question extracts 1 bit. Estimating bits extracted via "answer balance," the paper finds information per question correlates with win rate at $$r = +0.88$$ . Only two models reliably extracted a full bit per question—the only two that partitioned candidates by document title. This strategy is absent when $$N < 32$$ and appears in ~25% of questions when $N \geq 32$ : it was not learned, it emerged under candidate-set pressure.

Failure Breakdown: Communication, Not Guessing

Failures split into answer errors, discrimination failures, and prediction errors. Prediction errors were nearly nonexistent; communication failure (poor questions) dominated. Most strikingly, of Claude Opus 5's 34 consistent answer errors, 32 were "no" answers—a lazy negative bias, despite explicit instructions against defaulting to "no".

Cost Is Decoupled from Performance

Reasoning-token spending correlated with win rate at only $$r = -0.05$$ . The two cheapest runs ($0.36 and $0.39) won; the most expensive ($2.46) lost. Thinking more is not thinking better.

What It Means

1. Exponential amplification: tiny per-step reliability gaps become "playable vs. unplayable" across 10–50 step horizons—isomorphic to long-horizon credit assignment in RLHF. 2. Communication is harder than knowledge: bits per question predicts success better than knowledge reserves—an underrated dimension of *active questioning* ability. 3. The "no" bias is a lazy strategy: saying "no" always looks like narrowing the set but may cut nothing; akin to "omission blindness" in LLM judges. 4. Cost and performance are decoupled: quantified cleanly at $$r = -0.05$$ . 5. Strategy emergence has a threshold: title-partitioning appears only under sufficient candidate-set pressure.

Why This Matters

The game is isomorphic to many real tasks: medical diagnosis (each question halves disease candidates), debugging (each test halves bug hypotheses), requirements clarification, and experiment design. On all of them, bits-per-question matters more than total knowledge.

Honest Limitations

Only one document set (Wikipedia abstracts)
One "effort level" per model
Context loading not fully aligned across models
Architectural details of four models unpublished

Yet the core findings—the

p^{\log_2 N}

formula, communication-failure dominance, and cost-performance decoupling—are methodologically clean.

Closing Thoughts

The paper turns a children's game into a ruler measuring frontier models' *active communication* ability. The question is not "who is smarter" but "who asks better questions." The figure $$p = 0.928$$ may become a standard evaluation dimension: not "how many answers were correct," but "how much uncertainty does each question eliminate."

Paper: arxiv.org/abs/2609.19113 Code: github.com/ppotash/logn-questions