English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

The War on a Hair: A One-Centimeter Ruler Claiming to Measure 0.1 Millimeters — Auditing LLM-as-a-Judge Benchmarks

Forum topic · 小凯 · 2026-09-24

Summary

A Chinese forum post discusses arXiv paper 2609.28082, 'Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM' by Atul Anand, which performs a metrological audit of LLM-as-a-judge evaluation. Drawing on 373,019 judge verdicts and Generalizability Theory, the paper shows that single-judge, point-wise 1–5 scoring has a structural reliability ceiling determined by the system-by-judge interaction variance, which adding more test items cannot overcome — 'items saturate; judges do not.' The measurable 'floor' — the smallest reliably detectable score difference — ranges from 0.41 to 1.24 points on 0–5 scales, while the median claimed improvement in the literature is only 0.28 points. Of 17 published MT-Bench improvement claims, all 17 fell below this noise floor. Switching to pairwise preference with swapped presentation order cuts judge-interaction variance by two orders of magnitude, reaching a generalizability coefficient of 0.986, though an 8.6 percentage-point position bias favoring first-presented answers remains. An audit of 628 arXiv papers found fewer than a quarter report evaluation run counts and only 46–67% report any uncertainty estimates.

The War on a Hair: A One-Centimeter Ruler Claiming to Measure 0.1 Millimeters

A detailed Chinese-language analysis of arXiv:2609.28082 — *Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM* (Atul Anand, 2026), a meta-scientific audit of LLM-as-a-judge evaluation. Below is a structured English rendering of the post's argument and findings.

Key points

  • The core question is metrological, not about bias. Prior critiques of LLM judges (verbosity bias, position bias, self-preference) ask "how much is the clock fast?" This paper asks "what is the clock's smallest division?" — i.e., can the instrument resolve the 0.1–0.2 point differences that dominate leaderboards?
  • Background on LLM-as-a-judge. Since MT-Bench (Zheng et al., 2023) reported >80% agreement between GPT-4 preferences and human judgments, point-wise scoring (MT-Bench style) and win-rate comparison (AlpacaEval style) replaced expensive human evaluation.
  • The anatomy of score variance. Following Generalizability Theory (Cronbach et al., 1972), observed scores decompose into system variance (real differences), item variance, judge variance, and the critical interaction terms: system×item and system×judge interactions.
  • Main findings

  • 373,019 judge verdicts analyzed. The generalizability coefficient (Eρ²) measures how reliable rankings are. Under a single judge, the coefficient asymptotes to σ²s/(σ²s + σ²sj) — item count cancels out entirely. "Items saturate; judges do not." More questions cannot push past the ceiling set by the system-by-judge interaction.
  • The noise floor vs. claimed progress. For 0–5 scales, the minimum reliably detectable difference is 0.41–1.24 points (at native benchmark sizes). The median claimed improvement across the field is 0.28 points — below even the lowest floor. Over half of "we lead by X points" claims fall in a region the instrument cannot see.
  • Diminishing returns near the ceiling. The item count needed for each marginal reliability gain approaches divergence; the ceiling itself is unreachable by adding items at all.
  • MT-Bench: 17/17 claims below the floor. All 17 published improvement claims on MT-Bench fall below the benchmark's noise floor. About 70% of win-rate claims in the literature are below the floor of pairwise comparison as well.
  • The fix: ask "which", not "how good"

  • Switching to pairwise preference (run both presentation orders) drops system×judge interaction variance to under 1% of system variance — two orders of magnitude — yielding a single-judge generalizability coefficient of 0.986 (bootstrap CI [0.934, 1.000], 11 systems).
  • New cost: position bias. The first-presented answer wins 8.6 percentage points more often — larger than the median claimed improvement (~7pp) across 53 published win-rate papers. Mitigation: evaluate both presentation orders and average.
  • This partially vindicates AlpacaEval's pairwise protocol; its real flaw was only unswapped presentation order.
  • Audit of evaluation practice

  • Anand audited 628 arXiv papers using two independent LLM coders cross-checked against blind human coding (kappa = 0.73):
  • 1. Fewer than 25% report how many times the evaluation was run. 2. Only 46–67% report any form of uncertainty (error bars, confidence intervals, variance). 3. Given this disclosure level, most conclusions are unverifiable and incomparable even setting structural issues aside.

    Takeaway

    The paper proposes no new benchmark and trains no model — it performs calibration, like 19th-century metrologists checking the prototype kilogram. Its conclusions compress to three sentences:

    1. Single-judge point-wise scoring has a structural ceiling; more items cannot fix it. The disease belongs to the measurement format, not the judge. 2. Asking "which is better" makes one judge sufficient (Eρ² = 0.986) — but swap presentation order, or the 8.6pp seat premium will consume your conclusion. 3. Most claimed improvements in the current literature are smaller than the instrument's own floor — not rhetoric, but an execution record of 17/17 on MT-Bench.

    The post closes with practical advice: before celebrating a "0.21-point lead," ask — how many items? how many judges? how many runs? And if those go unanswered: whose ruler, and how coarse are its divisions?

    Reference

  • Anand, A. (2026). *Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM*. arXiv:2609.28082. https://arxiv.org/abs/2609.28082

Tags

#llm-evaluation#llm-as-a-judge#meta-science#benchmarks#measurement-theory#mt-bench#pairwise-comparison#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178635167