Preregistered Audit Shows LLM-as-Judge Measurements Are Unreliable Across Repeated Identical Requests

Forum topic · 小凯 · 2026-09-05

Summary

A preregistered study by Haoyaun Zhu and Jie Zhang (arXiv:2609.04198) audits a hidden assumption behind language-model judges: that identical requests to the same model name yield reproducible results. Across 52,988 audited request attempts, same-window repeat rankings achieved Spearman consistency of only 0.400 against a required 0.90, while byte-identical next-day replays scored 0.78 against a required 0.99. Three mechanisms explain the gap: label-to-semantic mapping noise rivaling the signal itself, candidate gaps seven orders of magnitude below the instrument's noise floor, and rank shifts on byte-identical inputs. Follow-up preregistered studies showed waiting did not help (0.805 vs 0.800), switching providers did not help (four providers shared a noise floor, median 0.74-0.88, unpredictable from metadata), and self-hosting only worked on quiet servers. The authors propose a three-level snapshot identity ladder, eight design rules, and a reporting checklist; a pilot at roughly 2% of study calls can expose unreachable thresholds in advance. The paper concludes that on shared endpoints, a model name is not a frozen instrument, and preregistered evaluations must first validate their own measurement instruments before fixing any thresholds.

Paper Overview

Field: ML Authors: Haoyaun Zhu, Jie Zhang Published: 2026-09-03 arXiv: 2609.04198

Abstract (translated)

Language-model judges now gate training data, score generations, and drive leaderboards. As measurement instruments, they rely on a rarely stated assumption: that the same request sent to the same model name should read the same tomorrow. The authors tested this assumption in two preregistered audit campaigns with all thresholds fixed in advance — and both failed instrument validation. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, with every execution log reaching its cap.

Three mechanisms explain the gap:

Label-to-semantic mapping introduces deviations as strong as the signal itself
Candidate gaps are seven orders of magnitude below the instrument's own noise floor
Byte-identical inputs return different rankings — noise amplified by exactly aligned readings

Neither metric substitution nor sampling could fix the issue on the tested grid.

Preregistered Follow-ups

Waiting doesn't help: 0.805 vs 0.800 across an additional five-day repeat on the sampled dates
Switching providers doesn't help: four providers share a noise floor (median 0.74–0.88), unpredictable from any metadata field
Self-hosting on batch-invariant kernels only works when the server is quiet

Contributions

The authors distill the evidence into:

A three-level snapshot identity ladder
Eight design rules
A reporting checklist
A pilot costing ~2% of study calls that can expose two unreachable thresholds in advance

Conclusion

All findings concern external measurement against shared service infrastructure. On shared endpoints, a model name is not a frozen instrument; preregistered evaluations must measure their instrument first before fixing any thresholds.

Original Abstract

> Language-model judges now gate training data, score generations, and drive leaderboards. We audited the assumption that the same request sent to the same model name reads the same tomorrow. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99.

--- *Auto-collected on 2026-09-05*