The False Throne: Statistical Illusions on LLM Leaderboards

> You glance at a leaderboard. GPT-5.4 ranks first with 89.2; Claude Sonnet 4.6 ranks second with 88.7. The gap is 0.5 points, so GPT is better. > > Now ask yourself: is that gap statistically real? If we re-tested on another benchmark of the same difficulty and distribution, would GPT still win—or would it flip? A leaderboard doesn't answer this. It gives you one number and lets you assume it means "better."

In May 2026, independent researcher Anany Kotawala pinned this problem down with concrete numbers in *Resolution Diagnostics for Paired LLM Evaluation* (arXiv:2605.30315): of 40 pairwise comparisons on Open LLM Leaderboard v1, 11 fail to resolve under standard statistical criteria. Among adjacent ranks in MMLU-Pro's top ten, 4 to 6 of 9 pairs are indistinguishable. Nearly half of the top-ten ordering is, statistically speaking, a guess.

| Item | Detail | |------|--------| | Paper | Resolution Diagnostics for Paired LLM Evaluation | | Author | Anany Kotawala (independent researcher) | | arXiv | 2605.30315 (May 28, 2026, cs.CL) | | Key findings | 11/40 unresolved pairs on Open LLM Leaderboard v1; 4–6/9 unresolved adjacent pairs in MMLU-Pro top 10; common Cohen-style sample-size calculators biased ~2x for paired designs |

1. How leaderboards are built — and where they break

A typical leaderboard feeds each model the same benchmark (a few thousand questions), computes an accuracy, and sorts. But accuracy is an estimate, whose precision depends on sample size and effect size. With only a few hundred questions and a true gap of 0.3%, the ordering can flip on the next batch of questions.

Kotawala applied standard paired tests with alpha = 0.05 and power 1 − beta = 0.8—the conventions of social and medical research. Result: 11 of 40 pairs on Open LLM Leaderboard v1 are unresolved; 4 of 9 adjacent MMLU-Pro top-ten pairs are unresolved, rising to 6/9 when accounting for the clustered structure of items within the test set.

2. Worse than it looks

The remaining 29 resolved pairs may only barely clear the threshold. The paper offers a diagnostic: the resolution ratio q = N / N*, where N is actual sample size and N* is the minimum needed. If q < 1, your sample is insufficient; if q barely exceeds 1, you're ranking on noise.

Kotawala also found a tooling problem. Standard sample-size calculators—Cohen's 1988 formulas, G*Power, R's pwr package—assume independent samples. Users typically hand-correct for paired designs by multiplying by (1 − rho), where rho is the correlation between models' answers. The paper proves this correction deviates from the correct value by about 2x in large- or small-effect limits—meaning many researchers compute "sufficient" sample sizes that are actually half of what's needed.

3. Robust under multiple corrections

With 40 tests at alpha = 0.05, ~2 "significant" results are expected by chance alone. Kotawala applied family-wise error rate and FDR corrections: the indistinguishable pattern persists. He also tested anytime-valid sequential testing, which allows continuous monitoring without a fixed sample size—and the comparisons remain unresolved. The problem isn't the test; the signal in the data is simply too weak. No statistical method can rescue a 0.2% true gap measured on 1,000 questions.

4. What leaderboards omit

A leaderboard makes two claims: (1) the accuracy estimate is 89.2%, and (2) the model is better than the runner-up. The first is usually accompanied by a (rarely reported) confidence interval; the second, for over a quarter of the comparisons analyzed, has no statistical basis. Companies choosing models, researchers setting baselines, and investors are all deciding on claim (2).

Kotawala's minimum standard: report the resolution ratio q for every adjacent pair, and declare a tie when q < 1. Implementation requires only one extra formula on existing data. The obstacle is sociological: ties don't make headlines; "A beats B to take the crown" does. Statistical honesty and communicative efficiency are in direct conflict here.

5. Open questions

• How accurate is the second-order expansion behind the 2x bias at extreme effect sizes or very high rho?
• The paper covers only two leaderboards. What about Chatbot Arena or AlpacaEval, which use different designs?
• Is q >= 1 sufficient given distributional assumptions of paired tests (e.g., non-normal or outlier-prone per-item scores)?

6. Rankings are not measurements

Two models, 89.2 and 88.7, a gap of 0.5. If sample size is insufficient, you don't know which model is truly better—only that, on this particular finite set of questions, model A happened to answer slightly more correctly. This doesn't make leaderboards useless; it means the narrative of "first place," "beats," and "crushes" is often more dramatic than the data supports. Kotawala's contribution is not a new leaderboard but a set of diagnostics—resolution ratios, corrected sample-size formulas, anytime-valid monitoring—so you can ask: does the data actually support this ranking? If not, what you're seeing is not a better model, just bigger noise.

References: 1. Kotawala, "Resolution Diagnostics for Paired LLM Evaluation", arXiv:2605.30315, 2026. 2. Cohen, "Statistical Power Analysis for the Behavioral Sciences", 2nd ed., Erlbaum, 1988. 3. Beeching et al., "Open LLM Leaderboard", Hugging Face, 2023. 4. Hendrycks et al., "Measuring Massive Multitask Language Understanding" (MMLU), ICLR 2021. 5. Ramdas et al., "Admissible Anytime-Valid Sequential Inference", Statistical Science, 2023.