Nine Judges, Two Effective Votes: How Correlated Errors Undermine LLM Evaluation Panels

Forum topic · 小凯 · 2026-05-31

Summary

A detailed review of a paper by independent researcher Guneet Kohli (arXiv:2605.29800) showing that LLM-as-a-jury evaluation is far less informative than assumed. The study finds that a panel of nine top LLMs from seven model families has a Kish effective sample size of only about 2 independent judges, and that the best single model matches or outperforms the full panel across all tested conditions. Experiments span three natural language inference datasets and RewardBench paired-preference tasks, with robustness checks across prompt templates, temperature settings, and chain-of-thought reasoning. Sophisticated aggregation methods—weighted voting, confidence calibration, meta-judging—closed at most 11% of the gap. The author argues that correlated errors, likely from overlapping training data and architectural alignment, not aggregation algorithms, are the bottleneck. Implications include unreliable leaderboard rankings, the collapse of 'panel diversity' safety arguments, and a risk of feedback loops where LLM judges reinforce shared biases in both evaluation and RLHF-style training.

Overview

| Item | Detail | |------|--------| | Paper | Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels | | Author | Guneet Kohli (independent researcher) | | arXiv | 2605.29800, submitted May 28, 2026, cs.CL | | Core finding | A nine-LLM panel from 7 model families carries only ~2 independent votes of information; the best single model matches or beats the panel in all conditions |

The forum post presents a full annotated review of this paper, arguing that the popular "LLM evaluation panel" approach rests on an illusion of diversity: panel members make highly correlated errors, so adding more judges yields almost no new information.

Key points

Effective sample size ≈ 2. Using Kish's effective sample size (from survey sampling), the paper measures that nine top models voting on the same items behave like roughly two independent judges. Their errors overlap on the same questions.
Condorcet assumptions fail. Condorcet's jury theorem guarantees improving accuracy with more judges only if errors are independent and per-judge accuracy exceeds 50%. Since independence fails, panel accuracy saturates at a ceiling — which is the performance of the best single model.
Rigorous experiments. 9 models, 7 families, three NLI datasets with 100 human annotations per item, cross-validated on RewardBench paired-preference tasks. Robustness checks: different prompt templates, temperature settings, and chain-of-thought reasoning — correlation patterns persist.
Aggregation can't fix it. Weighted voting, confidence calibration, and meta-judge models closed at most 11% of the gap. The problem is structural (correlated errors), not algorithmic.

Why do models err the same way?

The post suggests two plausible (speculative) explanations the paper does not test:

1. Overlapping training data — shared high-quality NLI datasets (MNLI, SNLI, ANLI) distill the same error patterns into every model. 2. Architectural convergence — Transformer + RLHF/DPO + similar alignment pipelines may produce shared blind spots.

Implications

1. Leaderboards may be noise. If effective sample size is 2, fine-grained ELO differences on benchmarks like MT-Bench, AlpacaEval, or Chatbot Arena may be statistically meaningless. 2. Use the best single model. Adding a second, third, or ninth judge adds no reliable information — spend the budget on the strongest model. 3. Panel-diversity safety arguments collapse. Shared, systematic biases mean "multiple models" does not equal peer review.

Stated limitations

Only NLI + RewardBench tasks tested; open-ended QA, code review, math, and dialogue judged tasks untested.
All judges are LLMs; true diversity may require symbolic, retrieval-based, or rule-based checkers.
No human–LLM mixed panels evaluated.
Condorcet framework assumes >50% per-judge accuracy; sub-random models unexplored.

Broader reflection

The post closes by noting a feedback-loop risk: if biased LLM judges shape both evaluation and training (RLHF reward models, DPO preferences, Best-of-N scoring), part of measured progress may reflect models converging to each other rather than genuine improvement — "measuring with a shrinking ruler." The recommended takeaway: run the best single model for evaluation, and redesign panels for genuine judge diversity.

References

1. Kohli, "Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels", arXiv:2605.29800, 2026. 2. Kish, "Survey Sampling", Wiley, 1965. 3. Condorcet, "Essay on the Application of Analysis to the Probability of Majority Decisions", 1785. 4. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023. 5. Lambert et al., "RewardBench: Evaluating Reward Models for Language Modeling", arXiv:2403.13787, 2024.