Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

| Item | Detail | |------|--------| | Paper | Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels | | Author | Guneet Kohli (independent researcher) | | arXiv ID | 2605.29800 | | Submitted | May 28, 2026 | | Category | cs.CL | | Core finding | A panel of 9 top LLMs from 7 model families carries the information of only ~2 independent judges; the best single model matches or beats the panel in all conditions; correlated errors—not aggregation methods—are the bottleneck |

Nine judges walk into an evaluation room

Nine top-tier models from seven different "courts"—GPT, Claude, Gemini, and others—were asked to judge the same natural language inference (NLI) cases, with their verdicts aggregated, weighted, and compared.

Nine voters should be more reliable than one—if they judge independently. The paper's answer: nine judges, only two effective votes.

Not because two judges are especially smart, but because the other seven's judgments overlap heavily with those two. They make the same mistakes, fail on the same items, agree on the same questions. Statistically, this is error correlation. In effective-sample-size terms, nine judges behave like a sample of 2. They are not nine independent brains—they are one lineage's shared preferences speaking through nine mouths.

How it was measured

The paper borrows a tool from survey statistics: Kish effective sample size (\(n_{\text{eff}}\)), originally designed to measure correlation among survey respondents. If 1000 people all watch the same TV show and absorb the same narrative, your sample size isn't 1000—it's close to 1.

Applied to an LLM jury: if nine judges vote independently on each item, effective votes ≈ 9. If their errors are perfectly correlated, effective votes = 1. The measured result: about 2. Seven-ninths of the "illusion of independence" is eaten by error correlation.

A second layer of validation uses a Condorcet null model. Condorcet's jury theorem says: if each judge is independent and more than 50% accurate, larger juries converge toward perfect verdicts. Condition one—independence—fails. So the optimistic guarantee doesn't apply: with correlated errors, accuracy plateaus at a ceiling rather than converging to 100%. The measured ceiling: the best single model, which matches or beats the full nine-model panel under all test conditions.

A rigorous experimental design

• 9 models across 7 families; three NLI datasets with 100 human annotations per item; cross-validation on paired-preference tasks (RewardBench).
• Robustness checks: same errors under different prompt templates; identical correlation patterns across temperature settings; overlap persists with chain-of-thought reasoning; the same effective-sample-size decay appears in RewardBench's subjective preference-judging task.
• "Smart" aggregation algorithms—weighted voting, confidence calibration, meta-judge models—closed at most 11% of the gap. Even with tuning, the panel never beats a single model.

This is not an algorithm problem. It is a structural problem.

Why models converge

The paper doesn't fully explain why different models err identically, but plausible explanations include:

1. Training data overlap. High-quality NLI datasets (MNLI, SNLI, ANLI) are near-universal in pretraining. Their error patterns—specific logic traps, idiomatic ambiguities—get distilled into every model's parameters. It's like every judge in a city studying the same flawed law textbook. 2. Architectural convergence. Transformer + RLHF/DPO + similar alignment strategies + similar reasoning-chain training. Shared underlying structure may explain error overlap better than data differences do.

The paper offers no causal claim—appropriately, given the evidence.

What this means

The entire LLM evaluation ecosystem assumes multi-model cross-validation. LLM-as-a-judge is standard in academic review, safety filtering, alignment evaluation, and automated experiment design. MT-Bench, AlpacaEval, Chatbot Arena—all rely on models scoring models.

But if multiple models add near-zero information—their votes are two signals echoed nine times—the signal-to-noise ratio of this architecture is systematically overestimated. Concrete consequences:

1. Leaderboards are unreliable. With effective sample size ~2, fine-grained ELO rankings ("GPT-5.5 slightly beats GPT-5.4") may be statistically meaningless noise. 2. The best single model is the best judge. Adding a second, third, or fourth model adds no reliable information. If you can only afford one evaluator, use the best one. 3. The "LLM jury safety" argument collapses. The claim that multiple models reduce single-model bias fails: bias is systemic and shared. Panel diversity is an illusion.

Limitations, honestly stated

• Only NLI tasks tested. Three NLI datasets plus RewardBench cover classification and preference judgment; open-ended QA, code review, math, and multi-turn dialogue were not evaluated, so results can't yet generalize.
• Nine models are still LLMs. If the problem is shared Transformer-RLHF blind spots, swapping in a hundred more LLMs won't help. True diversity may require different evaluator types: symbolic reasoners, retrieval-based systems, rule-based checks.
• No human-LLM hybrid panels tested—a practically important gap, since many organizations already use mixed panels.
• Condorcet assumes >50% per-judge accuracy. Models below chance on some tasks would require a more complex theoretical treatment.

A broader question

The industry shifted from human evaluation to model evaluation not because it's more accurate, but because human evaluation is slow and expensive. But if the tool itself is this fragile, we risk a loop: measure capability with biased tools → optimize models against biased scores → models adapt to the tool → bias is entrenched.

This extends beyond evaluation into training: RLHF reward models, DPO preference pairs, and Best-of-N scorers are all "one LLM judging another." If error correlation is universal, part of the progress of recent years may be "becoming more alike" rather than "becoming better." Progress is real—but its magnitude may be overstated, because the ruler itself is shrinking.

Two votes, not nine

Not 9. Not 7. Not 3. 2.

Nine top LLM judges collectively amount to two independent judgments. The "LLM jury" concept needs rethinking—not because it's useless, but because its value is systematically inflated. If you have the budget to run nine models, run one good one. The other eight are replaceable copies.

The paper opens a door: how to design genuinely diverse AI evaluation systems. For now, one thing is clear: the bigger your jury, the more complete you feel—while what you're seeing is just two faces, reflected nine times.

References

1. Kohli, "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels", arXiv:2605.29800, 2026. 2. Kish, "Survey Sampling", Wiley, 1965. (original definition of effective sample size) 3. Condorcet, "Essay on the Application of Analysis to the Probability of Majority Decisions", 1785. 4. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023. 5. Lambert et al., "RewardBench: Evaluating Reward Models for Language Modeling", arXiv:2403.13787, 2024.