Paper Overview
- Field: Machine Learning
- Authors: Jai Moondra, Ayela Chughtai, Bhargavi Lanka, Swati Gupta
- Published: 2026-05-07
- arXiv: 2605.06656
Abstract (English translation)
Ranking LLMs via pairwise human feedback underpins today's leaderboards for open-ended tasks such as creative writing and problem solving. Analyzing ~89,000 comparisons of 52 LLMs across 116 languages from the Arena, we find that the best-fitting global Bradley-Terry (BT) ranking is misleading. Nearly two-thirds of decisive votes cancel each other out, and even the top 50 models under the global BT ranking are statistically hard to distinguish (pairwise win probabilities within the top 50 reach at most 0.53). We trace this failure to strong, structured opinion heterogeneity across languages, tasks, and time.Notably, language plays a key role. Grouping votes by language (and language family) substantially improves vote consistency, yielding ELO score distributions two orders of magnitude wider (i.e., highly consistent rankings). What appears to be global noise is in fact a mixture of coherent but conflicting subpopulations.
To address this heterogeneity in supervised machine learning, we introduce the (λ, ν)-portfolio framework: small collections of models whose prediction error is at most λ and which 'cover' at least a ν fraction of users. We formalize this as a variant of the set cover problem and provide guarantees using the VC dimension of the underlying set system. On Arena data, our algorithm recovers only 5 distinct BT rankings that cover over 96% of votes at moderate λ, whereas the global ranking covers only 21%. We also provide a portfolio of 6 LLMs covering twice as many votes as the global top-6 LLMs. We further build portfolios on the COMPAS dataset using ensembles of fairness-regularized classifiers, and show such portfolios can detect blind spots in data, which may be of independent interest to policymakers.