Your model ranks #1 on Chatbot Arena. You published the weights, wrote a nice paper, and claimed SOTA. But what if changing less than 1% of the battle data—say, deleting a single vote from certain users on certain models—drops your model from first to third? And another model can be pushed to #1 with the same tiny perturbation?
Oyarhoseini, Lin, and Karimi recently posted a paper on arXiv (2605.15761) doing exactly this. They built a unified perturbation framework to analyze how stable leaderboards—particularly pairwise human-preference leaderboards like Chatbot Arena—really are.
The answer: not very.
🎲 How the Leaderboard Is Built
Chatbot Arena does something clever: instead of asking people to score models on absolute scales (which humans are bad at), it runs AB tests—show two model outputs and ask which is better.
These pairwise preferences feed into a Bradley-Terry statistical model. Bradley-Terry dates back to 1952 and was originally designed for sports rankings: if Team A beats Team B, and Team B beats Team C, how much stronger should A be than C? It converts match outcomes into a "strength" parameter per player (or model), fit via maximum likelihood.
> When Bradley and Terry published the model in 1952, they were thinking about tennis tournaments. Seventy years later, the same model ranks AI models' intelligence. This is no coincidence—the ranking problem in pairwise comparisons has the same mathematical structure in sports and AI evaluation.
Chatbot Arena has accumulated hundreds of thousands of such preference votes and produces a leaderboard from them. The whole field watches this table. But Oyarhoseini et al. found that the ranking is unusually sensitive to changes in the input data.
🪚 Three Surgical Knives
They designed three classes of perturbations—delete, add, flip—to systematically test leaderboard robustness:
- Delete: remove a single match result from the battle data—one at a time.
- Add: insert a synthetic match result simulating one user's preference.
- Flip: reverse a match outcome—A's win becomes B's win.
The result is unsettling: less than 1% data perturbation can change who holds first place. This means current leaderboards may not be "objective rankings" but fragile constructions highly sensitive to data selection.
More critically, they show the same influence score can serve two purposes: auditing fragility ("can this leaderboard be manipulated?") and actually manipulating it ("how to achieve maximum rank change with minimum data changes"). The same math can be a transparency tool or an opaque manipulation tool—depending on who wields it.
🎯 Not Just a Theoretical Concern
They didn't test on toy setups alone. Experiments ran on Chatbot Arena and six other pairwise-comparison datasets, with consistent results everywhere: no robustness.
They also propose a normalized dataset-level robustness score for comparing leaderboard stability—so maintainers can tell the community how stable their leaderboard is, rather than just saying "we have 100k battle records." Under their measurement, Chatbot Arena's robustness score is not encouraging.
🤷 What I Don't Know
Several open questions:
1. Practical feasibility. Sub-1% perturbations are small in absolute terms, but I don't know how implementable they are operationally. If a maintainer can detect anomalous patterns—e.g., the same account repeatedly preferring the same loser—attackers would need cleverer strategies. The framework assumes the attacker knows the target model and influence scores; in reality that information may not be accessible. I'm unsure how effective the framework remains under limited-information scenarios.
2. Bradley-Terry as the ranking engine. It's what Chatbot Arena actually uses, but one of its assumptions is homogeneity—every match matters equally. In reality, some matches carry far more information (e.g., two top-tier models battling tells you more than top vs. bottom). Weighted matches or Bayesian variants like TrueSkill might be more robust. The paper doesn't compare against other ranking models.
3. How bad is poor robustness in practice? If a 0.5% perturbation flips #1, perhaps the leaderboard simply lacks the information resolution to distinguish "first" at all—the skill gap between #1 and #2 may be smaller than measurement noise. The problem may not be attackers but the leaderboard's resolution itself. The paper doesn't discuss this angle.
🔍 At Its Core, a Measurement Problem
Feynman's first principle: don't fool yourself. But if the tool you use to avoid fooling yourself is itself easily manipulated, you need to examine the tool first.
Leaderboards aren't inherently bad. What's bad is pretending they're sturdier than they are. This paper provides a toolbox: before releasing your new model, check whether your ranking survives less-than-1% data perturbation. If it does, you can rest easy. If not—you and your readers deserve to know.
---
References
1. Oyarhoseini, H., Lin, J., & Karimi, A.-H. (2026). *A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation*. arXiv:2605.15761 [cs.LG]. https://arxiv.org/abs/2605.15761 2. Bradley, R. A., & Terry, M. E. (1952). *Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons*. Biometrika, 39(3-4), 324-345. 3. Chiang, W.-L., et al. (2024). *Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference*. arXiv:2403.04132. 4. Herbrich, R., Minka, T., & Graepel, T. (2007). *TrueSkill: A Bayesian Skill Rating System*. Advances in Neural Information Processing Systems, 20. 5. Szymborski, B. (2024). *Benchmarking Benchmark Leakage: LLM Benchmark Manipulation and Protection*. arXiv:2404.12411.