When AI Judges Contradict Themselves: Diagnosing LLM Judge Reliability via Conformal Prediction and Transitivity Violations

Forum topic · 小凯 · 2026-04-18

Summary

A detailed analysis of a paper by Manan Gupta and Dhruv Kumar (BITS Pilani), "Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations," examines whether LLM-as-judge evaluation frameworks are trustworthy at the instance level. Using the SummEval dataset, the paper finds that while aggregate transitivity violation rates appear low (0.8–4.1%), disaggregation by document reveals that 33–67% of documents trigger at least one cyclic preference (A>B>C>A) in judges like Mistral-Small-3.1, LLaMA, and Qwen. Minimum Feedback Arc Set (MFAS) repairs fail to improve agreement with human rankings, indicating sparse, unfixable noise. The second contribution applies split conformal prediction: prediction set width correlates strongly with actual judge-human disagreement (Spearman r_s = +0.576, N = 1,918), and wide sets are consistent across judges (r̄ ≈ 0.32–0.38), showing that set width reflects document difficulty rather than model quirks. Reliability varies more by criterion than by judge: relevance judgments are most reliable, while fluency and consistency are least. The authors propose a selective escalation strategy—defer to human annotation when prediction set size exceeds 2—arguing that per-instance uncertainty quantification should replace aggregate metrics as the standard for trustworthy LLM evaluation.

When AI Judges Contradict Themselves: A Deep Dive into Judgment Reliability

> *"The first principle is that you must not fool yourself — and you are the easiest person to fool."* — Richard Feynman

This post analyzes the paper "Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations" by Manan Gupta and Dhruv Kumar (BITS Pilani), which dissects a largely ignored weakness in the LLM-as-judge evaluation paradigm.

The Problem: Aggregate Metrics Lie

LLM-as-judge has become the industry standard for NLG evaluation, replacing expensive human annotation. But researchers typically report only aggregate metrics (Kendall's τ, Pearson correlation with human scores), which average over hundreds of instances. A judge with "90% accuracy" may be systematically wrong on specific cases — exactly the ones you care about. The paper asks a different question: when can we trust the AI judge on a specific document?

Finding 1: Transitivity Violations

Rational preferences should be transitive: if A > B and B > C, then A > C. Cyclic preferences (A > B > C > A) are directed 3-cycles — logical self-contradictions.

Aggregate view: violation rates look acceptable, only 0.8–4.1% of triplets.
Disaggregated by document: 33–67% of documents show at least one violation (Mistral-Small-3.1 up to 67%; LLaMA 66.7% on fluency; Qwen 60%). Some documents trigger violations on up to 30.4% of triplet comparisons.

Violations are sparse but concentrated in "problem documents." Applying Minimum Feedback Arc Set (MFAS) rank repair did not improve agreement with human rankings — the violations are unfixable noise, not systematic bias. The judge has no stable internal quality yardstick; its comparison standard drifts with the comparison set.

Finding 2: Conformal Prediction Sets

Instead of a point estimate ("fluency = 4/5"), split conformal prediction produces a set (e.g., {3,4,5}) guaranteed to contain the true label with probability ≥ 1−α:

1. Split data into training and calibration sets; compute nonconformity scores on calibration data. 2. Choose a threshold covering (1−α) of calibration scores. 3. Include any label whose hypothetical score falls below the threshold.

Key result: set width correlates strongly with actual judge–human disagreement:

r_s = +0.576, \quad N = 1,918, \quad p < 10^{-100}

Wide sets are also correlated across judges (r̄ = 0.32–0.38), meaning width reflects document difficulty, not model-specific quirks — an external validation of set width as a per-instance reliability signal. (Coherence is the exception, with weak cross-judge agreement, r̄ ≈ 0.10.)

Finding 3: Criterion Matters More Than Judge

On SummEval's four criteria (average set sizes):

| Criterion | Avg. set size | Reliability | |---|---|---| | Relevance | ≈ 3.0 | Most reliable | | Coherence | ≈ 3.9 | Moderate | | Fluency | ≈ 4.9 | Unreliable | | Consistency | ≈ 4.9 | Unreliable |

Which criterion you evaluate matters more than which model judges it. Fluency is likely hard because it is a holistic, reader-dependent property; consistency requires deep semantic verification against the source; relevance is closer to information retrieval and hence easier.

Practical Implications

Selective escalation: compute prediction sets before accepting a score. If |C(x)| ≤ 2, trust the AI; if |C(x)| = 5, route to human annotation.
Report better statistics: fraction of documents with ≥1 violation, per-document violation distributions, and prediction set width distributions — not just aggregate τ.

Limitations

1. Small scale: 30 documents × 8 systems = 240 summaries. 2. Conformal coverage is marginal, not conditional per document. 3. Simple absolute-residual nonconformity scores; richer scores could tighten sets. 4. Single prompt template per judge–criterion pair (prompt sensitivity untested). 5. SummEval human scores are rounded to integers, adding discretization error.

Conclusion

Aggregate metrics are a "cargo cult" of scientific evaluation: the forms look right, but per-instance reliability is missing. The core takeaways: (1) per-instance reliability assessment is necessary; (2) reliability varies more by criterion than by judge; (3) prediction set width is a practical, cross-judge-consistent uncertainty signal that can guide human escalation; (4) honest uncertainty beats false precision.

References

1. Gupta, M., & Kumar, D. (2026). Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations. *arXiv preprint arXiv:2604.15302*. 2. Feynman, R. P. (1974). Cargo Cult Science. *Caltech Commencement Address*. 3. Vovk, V., Gammerman, A., & Shafer, G. (2005). *Algorithmic Learning in a Random World*. Springer. 4. Angelopoulos, A. N., & Bates, S. (2021). A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. *arXiv preprint arXiv:2107.07511*. 5. Fabbri, A. R., et al. (2021). SummEval: Re-evaluating Summarization Evaluation. *TACL*, 9, 391-409. 6. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. *NeurIPS*, 36.