Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Analysis

Forum topic · 小凯 · 2026-04-18

Summary

This arXiv paper (2504.13084) by Manan Gupta and Dhruv Kumar presents a two-pronged diagnostic toolkit for evaluating the per-instance reliability of LLM-as-judge frameworks in NLG evaluation, applied to the SummEval benchmark. First, a transitivity analysis reveals widespread per-input inconsistency hidden by low aggregate violation rates (mean violation rate of 0.8%-4.1%), with 33%-67% of documents exhibiting at least one directed 3-cycle in judge rankings. Second, split conformal prediction sets over 1-5 Likert scores provide theoretically guaranteed coverage of at least (1-α), with prediction set width serving as a per-instance reliability indicator (Spearman rs = +0.576, N=1,918, p < 10^-100, pooled across all judges). Prediction set width shows consistent cross-judge agreement (r̄ = 0.32-0.38), indicating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criteria matter more than judges, with coherence judged most reliably (mean set size ≈3.9) and relevance best (≈3.0), while fluency and consistency remain unreliable (≈4.9). The authors release all code, prompts, and cached results.

Paper Overview

Field: NLP
Authors: Manan Gupta, Dhruv Kumar
Published: 2025-04-17
arXiv: 2504.13084

Abstract

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. This paper presents a two-pronged diagnostic toolkit applied to SummEval:

1. Transitivity analysis: reveals widespread per-input inconsistency masked by low aggregate violation rates (ρ̄ = 0.8%-4.1%), with 33%-67% of documents exhibiting at least one directed 3-cycle.

2. Split conformal prediction sets over 1-5 Likert scores: provides theoretically-guaranteed ≥(1-α) coverage, with set width serving as a per-instance reliability indicator (rs = +0.576, N=1,918, p < 10^-100, pooled across all judges).

Key Findings

Prediction set width shows consistent cross-judge agreement (r̄ = 0.32-0.38), demonstrating that it captures document-level difficulty rather than judge-specific noise.
Across four judges and four criteria, both diagnostic methods converge: criteria matter more than judges.
Relevance is judged most reliably (mean set size ≈ 3.0), followed by coherence (≈ 3.9).
Fluency and consistency remain unreliable (mean set size ≈ 4.9).

The authors release all code, prompts, and cached results to support reproducibility.

---

*Auto-collected on 2026-04-18.*