Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Preferences
Field: Machine Learning Authors: Jai Moondra, Ayela Chughtai, Bhargavi Lanka, Swati Gupta Published: 2026-05-07 arXiv: 2605.06656
Overview
Ranking LLMs via pairwise human feedback underpins today's leaderboards for open-ended tasks such as creative writing and problem solving. Analyzing ~89,000 comparisons of 52 LLMs across 116 languages in the Arena, the authors show that the best-fitting global Bradley-Terry (BT) ranking is misleading.
Key Findings
- Nearly 2/3 of decisive votes cancel each other out; even the top-50 models under the global BT ranking are statistically indistinguishable (pairwise win probabilities within the top 50 reach at most 0.53).
- The failure traces to strong, structured preference heterogeneity across languages, tasks, and time.
- Language plays a key role: grouping votes by language (and language family) sharply improves agreement, producing ELO score distributions up to two orders of magnitude wider — i.e., highly consistent rankings.
- What looks like global noise is in fact a mixture of coherent but conflicting subpopulations.
- On Arena data, the algorithm recovers only 5 distinct BT rankings, covering over 96% of votes at moderate λ, versus 21% for the global ranking.
- A portfolio of 6 LLMs covers twice as many votes as the global top-6 LLMs.
- On the COMPAS dataset, portfolios built from ensembles of fairness-regularized classifiers can detect blind spots in the data — of independent interest to policymakers.
The (λ, ν)-Portfolio Framework
To address this heterogeneity, the paper introduces (λ, ν)-portfolios: small sets of models whose prediction error is at most λ and which "cover" at least a fraction ν of users. The problem is formalized as a variant of set cover, with guarantees derived from the VC dimension of the underlying set system.