Paper: Preregistered Audit Shows LLM-as-Judge Measurements Are Unreliable Across Repeated Requests

Forum topic · 小凯 · 2026-09-05

Summary

This paper (arXiv:2609.04198) by Haoyaun Zhu and Jie Zhang audits the reliability of language-model judges, which increasingly gate training data, score generations, and drive leaderboards. Across 52,988 audited requests in two preregistered campaigns, same-window repeat rankings achieved Spearman correlation of only 0.400 (threshold: 0.90), and byte-identical next-day replays reached 0.78 (threshold: 0.90 target 0.99). Three mechanisms explain the gap: label-to-semantic mapping introduces bias as strong as the signal; candidate score gaps sit seven orders of magnitude below the instrument's noise floor; and byte-identical inputs return different rankings, amplified by exact-arrangement readings. Follow-up preregistered tests found waiting, switching providers (four providers shared a median 0.74–0.88 noise floor), or self-hosting offered limited or no fixes. The authors propose a three-tier snapshot identity ladder, eight design rules, and a reporting checklist, with a pilot at roughly 2% of study volume able to expose unreachable thresholds in advance. The core message: on shared inference endpoints, a model name is not a frozen instrument—preregistered evaluation must validate its measuring tool before fixing thresholds.

Clean Engineering, Unstable Measurement: A Preregistered Reliability Audit of LLM-as-Judge

Field: Machine Learning Authors: Haoyaun Zhu, Jie Zhang arXiv: 2609.04198

Key Findings

Language-model judges now gate training data, score generations, and drive leaderboards. This work audits a rarely stated assumption behind such use: that the same request sent to the same model name should read the same tomorrow.

Across 52,988 audited request attempts in two preregistered audit campaigns (all thresholds fixed in advance), both studies failed instrument validation.
Same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.
Byte-identical next-day replays agreed at 0.78 against a required 0.99, with every execution record hitting its cap.

Three Mechanisms Behind the Gap

1. Label-to-semantic mapping shifts readings as strongly as the signal itself. 2. Candidate gaps sit seven orders of magnitude below the instrument's own noise floor. 3. Byte-identical inputs return different rankings — noise amplified by exact-arrangement readings.

Neither metric substitution nor sampling could fix the problem on the tested grid.

Preregistered Follow-Ups

Waiting didn't help: 0.805 vs 0.800, replicated over five additional days.
Switching providers didn't help: four providers shared a noise floor (median 0.74 to 0.88), unpredictable from any metadata field.
Self-hosting on batch-invariant kernels worked only when servers were quiet.

Deliverables

The authors distill the evidence into:

A three-tier snapshot identity ladder
Eight design rules
A reporting checklist, with a pilot at roughly 2% of study volume able to expose the two unreachable thresholds in advance

All results concern external measurement against shared service infrastructure. On shared endpoints, a model name is not a frozen instrument — preregistered evaluation must measure its instrument before fixing any thresholds.

--- *Auto-collected on 2026-09-05*