The Illusory Throne: Statistical Illusions on LLM Leaderboards

> You glance at a leaderboard. GPT-5.4 ranks first, Claude Sonnet 4.6 second. GPT scores 89.2, Claude 88.7 — a gap of 0.5. GPT is better. > > Now ask yourself: is this gap statistically real? If we tested on another dataset of the same difficulty and distribution, would GPT still win — or would the ranking flip? Over 100 runs, how often does each model win? > > Leaderboards don't answer this. They give you a single number. You assume it means "better." But across 11 public LLM leaderboard comparisons, the gaps fail standard statistical significance testing. The rankings you see may be orderings of noise — two models with no discernible capability difference, where small samples and randomness decide who comes first. > > In May 2026, independent researcher Anany Kotawala pinned this down in *Resolution Diagnostics for Paired LLM Evaluation*: in 40 pairwise comparisons on Open LLM Leaderboard v1, 11 pairs failed to resolve under conventional statistical criteria. Among adjacent rankings in the MMLU-Pro top ten, 4 to 6 pairs are indistinguishable — meaning nearly half of the ordering within the top ten is statistically a guess.

| Item | Detail | |------|--------| | Paper | Resolution Diagnostics for Paired LLM Evaluation | | Author | Anany Kotawala (independent research) | | arXiv ID | 2605.30315 | | Submitted | May 28, 2026 | | Category | cs.CL | | Key findings | 11/40 Open LLM Leaderboard v1 pairs unresolved; 4-6/9 adjacent MMLU-Pro top-10 pairs indistinguishable; common Cohen-style sample-size calculators biased ~2x in paired designs |

1. How leaderboards are built

The typical workflow: feed every model the same test set (a few thousand questions), score each answer, compute an accuracy, and rank models from high to low.

The problem is in the "ranking."

Accuracy is an estimate. Its precision depends on sample size and effect size. With only a few hundred questions and a true gap of 0.3% between two models, your sample is nowhere near enough — the ordering could easily flip on the next batch of questions.

This is old statistics applied to a new setting, yielding a new number: 11/40.

Kotawala applied a standard paired-test framework to two public leaderboards, using alpha = 0.05 and power 1 - beta = 0.8 — the conventional thresholds from social science and medical research. Under this standard, 11 of 40 pairs on Open LLM Leaderboard v1 fail to resolve. For MMLU-Pro, 4 of 9 adjacent pairs in the top ten are unresolved; accounting for the true clustering structure (topic correlations within the test set), the number rises to 6/9.

In other words, more than half of the orderings within the MMLU-Pro top ten are uncertain under commonly accepted statistical standards.

2. Why it's worse than you think

"Fine — 11 of 40 are indistinguishable. What about the other 29?"

Those 29 pairs have gaps large enough to detect at current sample sizes. But how many just barely crossed the line — and would fall back into the unresolved zone with a slightly smaller test set or slightly higher variance?

The paper offers a diagnostic tool: the resolution ratio q = N / N*, where N is the actual sample size and N* the minimum required. If q < 1, your sample is insufficient. If q barely exceeds 1 (say 1.05), you are ranking on noise.

Kotawala also found a tooling problem. Widely used sample-size calculators — Cohen's 1988 formulas, G*Power, R's pwr package — have a systematic flaw when applied to paired designs (the same test set used for both models). Users typically multiply independent-sample sizes by (1 - rho), where rho is the correlation between the two models' answers. The paper proves this correction deviates from the correct value by about 2x in the large- and small-effect limits — so many researchers compute "ample" sample sizes that are actually only half of what is needed.

If your sample-size calculator is off by 2x, you use half the necessary data, then declare "model A beats model B" — your statistical tooling has betrayed you.

3. Multiple comparisons and anytime-valid tests: do the defenses hold?

Run 40 statistical tests and, even with no true differences anywhere, you expect about 40 × 0.05 = 2 "significant" results by pure chance. That's the multiple-comparisons problem.

Kotawala applied multiplicity corrections — family-wise error rate and FDR control. The conclusion is unchanged: the indistinguishability pattern survives multiple-comparison correction.

The paper also tested anytime-valid sequential testing, a framework that fixes no sample size in advance and allows continuous monitoring with optional stopping. Even under this more flexible design, the unresolved comparisons remain unresolved. It's not that the tests aren't clever enough — the signal in the data itself is too weak.

This is the key point. You can swap tests, adjust alpha and beta, go Bayesian instead of frequentist — but if the true gap between two models is 0.2% and your test set has only 1,000 questions, no statistical method can save you. Insufficient information is insufficient.

4. What leaderboards lie about

This is not just a technical issue — it's a trust issue.

When a leaderboard shows "model A ranks first with 89.2," it makes two claims: (1) A's accuracy estimate on this benchmark is 89.2%, and (2) A is better than the runner-up B. The first claim usually has a reasonable confidence interval (though it's rarely reported). The second claim — for 11/40 comparisons in this analysis — has no statistical basis.

Leaderboard consumers — companies choosing models, researchers setting baselines, investors making judgments — base decisions on the second claim, which fails to hold up in more than a quarter of cases.

Kotawala's minimal standard: report the resolution ratio q = N / N* for every pair of adjacent rankings. If q < 1, list the models as tied. If q >= 1, one may claim superiority. The recommendation is so simple it only requires maintainers to run one extra formula on existing data.

So why doesn't the industry do it? Because "ties" don't make news. "A dethrones B" does. A leaderboard's social function — citations, discussions, tweets — depends on a clear, graded ordering. Once many rankings become ties, the leaderboard's narrative value collapses. Statistical honesty and communicative efficiency are in direct conflict here.

5. Honest uncertainty

The paper exposes an important problem with a rigorous framework, but leaves open questions:

• Second-order effects in small samples. The ~2x bias in common sample-size calculators is derived under a specific second-order expansion. Does it remain accurate for extremely small effect sizes or very high rho? The explicit second-order constants are given, but the expansion's domain of validity is not fully discussed.
• Other leaderboards? The paper analyzes two leaderboards. What about Chatbot Arena (human preference based) or AlpacaEval? Different designs and sample sizes could make the indistinguishability problem far more or less severe. A cross-leaderboard comparison would be valuable — but hasn't been done.
• Is q sufficient or necessary? q >= 1 implies adequate sample size — but is the paired test itself the "right" test? Paired tests typically assume a (normal-approximated) distribution of score differences. If that assumption fails — non-normal scores, extreme outliers on certain questions — the q calculation may over- or underestimate the required sample size.

6. Rankings are not measurements

Back to the opening example: two models, 89.2 and 88.7, a gap of 0.5.

This analysis tells you: if the gap is 0.5 and your sample is too small, you may not know which model is truly better. You only know that on this particular, finite set of questions, model A happened to answer slightly more correctly than model B.

This is not a claim that leaderboards are useless. It is a claim that leaderboard narratives — "first," "second," "beats," "crushes" — are routinely more dramatic than the data supports.

What Kotawala offers is coolant for that narrative heat: not a new leaderboard, but a set of diagnostic tools — the resolution ratio, correct sample-size formulas, anytime-valid monitoring — so that whenever you read a leaderboard you can ask: "Is the data enough to support this ranking?"

If the answer is no — then what you're seeing is not a better model. It's just bigger noise.

> | Item | Detail | > |------|--------| > | Core contributions | (1) First systematic quantification of statistical resolution in LLM leaderboard pairwise comparisons — 11/40 unresolved on Open LLM Leaderboard, 4-6/9 on MMLU-Pro; (2) proposes the resolution ratio as a minimum transparency standard; (3) finds common sample-size calculators biased ~2x for paired designs; (4) shows the indistinguishability pattern is robust to multiple-comparison corrections and anytime-valid sequential tests | > | Key limitations | Only two leaderboards covered; accuracy of the second-order approximation under small samples / high correlation not fully evaluated; distributional assumptions of paired tests may fail on extreme questions |

References: 1. Kotawala, "Resolution Diagnostics for Paired LLM Evaluation", arXiv:2605.30315, 2026. 2. Cohen, "Statistical Power Analysis for the Behavioral Sciences", 2nd ed., Erlbaum, 1988. 3. Beeching et al., "Open LLM Leaderboard", Hugging Face, 2023. 4. Hendrycks et al., "Measuring Massive Multitask Language Understanding" (MMLU), ICLR 2021. 5. Ramdas et al., "Admissible Anytime-Valid Sequential Inference", Statistical Science, 2023.