When AI Judges Contradict Themselves: A Deep Dissection of "Judgment Reliability"

> *"The first principle is that you must not fool yourself — and you are the easiest person to fool."* — Richard Feynman

Overview

Imagine a boxing referee who gets 95% of decisions right overall — but for specific matches, says "A beat B, B beat C, and strangely... C also beat A." This logical self-contradiction is the core problem exposed by the paper *Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations* by Manan Gupta and Dhruv Kumar (BITS Pilani): the seemingly perfect LLM-as-judge framework may be systematically misleading us.

Why Reliability Matters

Evaluation of natural language generation has shifted from expensive human annotators to cheap API calls: GPT-4, Claude, and Gemini can now judge other models' outputs for ~$0.01 per call. The industry reports aggregate metrics — Kendall's τ, Pearson correlations — which usually look good (0.7–0.8+). But aggregate numbers hide per-instance failures: a judge that is 90% accurate overall may be systematically wrong on specific documents, precisely the cases you care about most.

The Transitivity Paradox (A>B>C>A)

Transitivity is a basic requirement of rational preference: if A beats B and B beats C, A should beat C. Circular preferences (directed 3-cycles) violate this. Gupta and Kumar found:

• 33% to 67% of documents exhibit at least one transitivity violation by AI judges.
• Aggregate violation rates look acceptable (0.8–4.1%), but when disaggregated by document, the picture changes:

| Judge model | Aggregate violation rate | Documents with ≥1 violation | |---|---|---| | Mistral-Small-3.1 | 0.8–4.1% | up to 67% | | LLaMA | similar | 66.7% (fluency) | | Qwen | similar | 60% |

Violations are not uniformly distributed: a few "problem documents" repeatedly trigger contradictions, with per-document violation rates reaching 30.4% for Mistral-Small-3.1. This reveals that AI judges lack a stable internal quality yardstick — their comparison standard drifts when the comparison set changes.

Applying Minimum Feedback Arc Set (MFAS) rank repair did not improve agreement with human rankings, meaning violations are sparse, unpredictable noise rather than fixable systematic bias.

Conformal Prediction Sets: Giving Uncertainty a Shape

Instead of a point estimate ("fluency = 4"), split conformal prediction outputs a set (e.g., {3,4,5}) guaranteed to contain the true human score with probability at least 1−α (e.g., 90%). Procedure:

1. Split historical data into training and calibration sets. 2. Compute nonconformity scores (prediction-vs-truth gaps) on the calibration set. 3. Choose a threshold covering the (1−α) quantile of calibration scores. 4. For a new sample, include every candidate label whose nonconformity score falls below the threshold.

The key result: set width strongly correlates with actual human–judge disagreement:

\[r_s = +0.576, \quad N = 1,918, \quad p < 10^{-100}\]

Narrow sets signal reliability; wide sets flag cases needing human review.

The Criterion Matters More Than the Judge

On the SummEval dataset (fluency, consistency, coherence, relevance), reliability varied dramatically by criterion:

| Criterion | Average set size | Reliability | |---|---|---| | Relevance | ≈ 3.0 | Most reliable | | Coherence | ≈ 3.9 | Moderate | | Fluency | ≈ 4.9 | Unreliable | | Consistency | ≈ 4.9 | Unreliable |

The choice of criterion affects reliability more than the choice of judge model. Relevance resembles an information-retrieval task; consistency requires deep semantic understanding of source–summary relations; fluency is a global, reader-dependent property.

Cross-judge agreement supports this: when one judge (e.g., GPT-4) outputs a wide set for a document, other judges (Claude, LLaMA) also tend to (r̄ = 0.32–0.38). Wide sets therefore reflect document difficulty, not model quirks — an external validation of set width as a reliability signal. The exception is coherence (r̄ ≈ 0.10), possibly because different model families represent coherence differently.

Practical Implications

Selective escalation rule: compute prediction sets before accepting a judge's score.

• |C(x)| ≤ 2: trust the AI score.
• |C(x)| = 5 (full scale): escalate to human annotation.

Recommended reporting practices: 1. Fraction of documents with at least one violation (not just aggregate rates). 2. Per-document distribution of violation rates. 3. Distribution of prediction-set widths.

Stated limitations: small scale (30 documents × 8 systems = 240 summaries); marginal (not conditional) coverage guarantees; simple absolute-residual nonconformity scores; single prompt template per judge–criterion pair; discretized human scores.

Conclusion: Trust, but Verify

1. Per-instance reliability assessment is necessary — aggregate metrics mask serious individual failures. 2. Criterion choice dominates judge choice — trust relevance judgments; be cautious with fluency and consistency. 3. Prediction-set width is a practical, cross-judge-consistent uncertainty signal that can guide human escalation. 4. Cargo-cult evaluation is real — using the form of "LLM evaluation" does not guarantee reliability.

Embracing quantified uncertainty — as Feynman urged — is more honest, and more scientific, than false precision.

References

1. Gupta, M., & Kumar, D. (2026). Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations. *arXiv preprint arXiv:2604.15302*. 2. Feynman, R. P. (1974). Cargo Cult Science. *Caltech Commencement Address*. 3. Vovk, V., Gammerman, A., & Shafer, G. (2005). *Algorithmic Learning in a Random World*. Springer. 4. Angelopoulos, A. N., & Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. *arXiv preprint arXiv:2107.07511*. 5. Fabbri, A. R., et al. (2021). SummEval: Re-evaluating Summarization Evaluation. *TACL*, 9, 391–409. 6. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. *NeurIPS*, 36.