Clean Engineering, Unstable Measurement: A Preregistered Reliability Audit of LLM Judges

Forum topic · 小凯 · 2026-09-05

Summary

Language-model judges now gate training data, score generations, and drive leaderboards, relying on the rarely stated assumption that the same request to the same model name reads the same tomorrow. In two preregistered audit campaigns with all thresholds fixed in advance (arXiv:2609.04198, by Haoyaun Zhu and Jie Zhang), the authors test this assumption and find it fails instrument validation. Across 52,988 audited requests, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99. Three mechanisms explain the gap: label-to-semantics mapping introduces bias as strong as the signal, candidate gaps sit seven orders of magnitude below the instrument's own noise floor, and byte-identical inputs return different rankings, a noise amplified by rank readings. Follow-ups show waiting does not help (0.805 vs 0.800), switching providers does not help (four providers share a noise floor, median 0.74-0.88), and self-hosting on batch-invariant kernels works only on quiet servers. The paper distills evidence into a three-tier snapshot identity ladder, eight design rules, and a reporting checklist.

Overview

Field: Machine Learning Authors: Haoyaun Zhu, Jie Zhang Published: 2026-09-03 arXiv: 2609.04198

Key Findings

Language-model judges now gate training data, score generations, and drive leaderboards. As measurement instruments, they depend on a rarely stated assumption: that the same request sent to the same model name should read the same tomorrow. The authors examined this assumption in two preregistered audit campaigns with all thresholds fixed in advance — and both studies failed instrument validation.

Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.
Byte-identical next-day replays agreed at 0.78 against a required 0.99; every execution log hit the cap.
Three mechanisms explain the gap:
Label-to-semantics mapping biases readings as strongly as the signal itself.
Candidate gaps lie seven orders of magnitude below the instrument's own noise floor.
Byte-identical inputs return different rankings — noise that rank-aligned readings amplify.
Neither metric substitution nor sampling fixed the issue on the tested grid.

Preregistered Follow-ups

Waiting did not help: 0.805 vs 0.800 over five additional days of repeats.
Switching providers did not help: four providers share a noise floor (median 0.74 to 0.88), unpredictable from any metadata field.
Self-hosting on batch-invariant kernels worked only when the server was quiet.

Contributions

The paper distills the evidence into:

1. A three-tier snapshot identity ladder. 2. Eight design rules. 3. A reporting checklist. 4. A pilot covering roughly 2% of study call volume that can pre-expose both unreachable thresholds.

All results concern external measurement against shared service infrastructure. On shared endpoints, a model name is not a frozen instrument; preregistered evaluation must measure its instrument before fixing any threshold.

---

*Auto-collected 2026-09-05*