[论文] Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transi...

论文概要

研究领域: NLP
作者: Manan Gupta, Dhruv Kumar
发布时间: 2025-04-17
arXiv: 2504.13084

中文摘要

LLM-as-judge框架正越来越多地用于自然语言生成自动评估，但其逐实例可靠性仍 poorly understood。我们在SummEval上呈现了一个双管齐下的诊断工具包：(1) 传递性分析揭示了被低聚合违规率（ρ̄ = 0.8%-4.1%）掩盖的广泛逐输入不一致性，33%-67%的文档至少表现出一次有向3-循环；(2) 在1-5李克特分数上的分裂共形预测集提供理论上保证的≥(1-α)覆盖率，预测集宽度作为逐实例可靠性指标（rs = +0.576，N=1,918，p < 10^-100，跨所有评判者汇总）。关键的是，预测集宽度显示出一致的跨评判者一致性（r̄ = 0.32-0.38），证明它捕捉的是文档级难度而非评判者特定噪声。跨四个评判者和四个标准，两种诊断方法趋于一致：标准比评判者更重要，相关性被评判得最可靠（平均集大小≈3.0），连贯性次之（平均集大小≈3.9），而流畅性和一致性仍不可靠（平均集大小≈4.9）。我们发布了所有代码、提示和缓存结果。

原文摘要

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: (1) a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (ρ̄ = 0.8%-4.1%), with 33%-67% of documents exhibiting at least one directed 3-cycle; and (2) split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed ≥(1-α) coverage, with set width serving as a per-instance reliability indicator (rs = +0.576, N=1,918, p < 10^-100, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (r̄ = 0.32-0.38), demonstrating it captures document-level difficulty rather than ...

自动采集于 2026-04-18

#论文 #arXiv #NLP #小凯