100% Consistent but Only 51% Correct: The Certainty Trap of LLM Judges

An Awkward Number

Suppose you use GPT-4o as a judge, scoring the same 100 questions. You repeat the run 10 times and get identical results every time — 100% consistency. You relax: the judge is stable.

Then you check its accuracy: only 51%.

This is not a hypothetical. A paper from October 2026, "All Verdicts are Not Equal: Rethinking LLM Judge Reliability" (arXiv: 2610.12083), conducted a systematic reliability audit of 6 frontier models and exposed a problem the field has largely ignored: an LLM judge can be 100% deterministic yet only 51% correct.

Worse, the industry-standard "accuracy" metric cannot even see this problem.

Where the Problem Lies

LLM-as-a-Judge has become the standard NLP evaluation paradigm: have GPT-4o score two responses, pick the better one, and treat that judgment as ground truth. The paradigm carries a hidden assumption: same question, same prompt, same model → stable output.

The audit reveals three cracks in this assumption:

Crack 1: Sampling variance. Same question, same prompt, same temperature — run 10 times and results may differ. 7%–17% of verdicts changed across 10 repetitions. The "accuracy" you see may just be one lucky sample out of 10.

Crack 2: Order bias. Put response A first vs. response B first, and the verdict may flip. In pairwise evaluation, order-flipping caused verdict changes at rates up to 98% — some "verdicts" depend entirely on who appears first.

Crack 3: Format sensitivity. Same prompt, Direct format vs. Verdict-first JSON format: accuracy differs by 25 percentage points. You think you're measuring model capability, but you're measuring prompt engineering.

Core Concept: Trustworthy Verdict Rate (TT)

The paper's central contribution is a unified metric — Trustworthy Verdict Rate (TT) — which measures not "accuracy" but "the probability of satisfying three conditions simultaneously":

1. Reproducible: at least 9 out of 10 repeated runs give the same verdict 2. Order-invariant: swapping A/B positions doesn't change the verdict 3. Accurate: the verdict matches ground truth

TT has an elegant mathematical form:

where C is stochastic consistency, A is accuracy, and φ is the flip rate. The formula carries a deep implication: accuracy is capped by the flip rate. If φ = 98%, the ceiling on A is 1 − 98%/2 = 51%. No matter how "smart" the model is, order sensitivity pins accuracy near 51%.

This is why Gemini-2.0-Flash can be "100% consistent yet only 51% correct" — its CC (stochastic consistency) reaches 96%–100%, but its high flip rate crushes the accuracy ceiling to 51%.

Experimental Design: A 2×2 Quadrant

Four benchmarks spanning a 2×2 design:

| | Subjective | Objective | |---|---|---| | Standard | MT-Bench | JudgeBench-MC | | Adversarial | AlpacaFarm | JudgeBench |

Six frontier models: GPT-4o, Claude-3.5-Sonnet, Gemini-2.5-Pro, Gemini-2.0-Flash, DeepSeek-V3, Llama-3.1-405B.

5 prompt formats × 2 presentation orders × 3 sampling temperatures × 10 repetitions = 300 data points per model×benchmark combination.

Key Findings

Finding 1: Accuracy severely overestimates reliability.

On JudgeBench with Gemini-2.5-Pro using Direct prompting: accuracy 63.5%, TT only 28.9%. Switch to Verdict-first JSON: accuracy 88.5%, TT 77.5%. A 25-point accuracy gain came with a 49-point TT gain — because the format change simultaneously reduced flip rate and sampling variance.

Finding 2: The most "certain" judge may be the least trustworthy.

Gemini-2.0-Flash's CC was 96%–100% across all settings — nearly fully deterministic. Yet TT was only 2.0%–52.3%. The reason is structural: in TT = C × (A − φ/2), when (A − φ/2) is small, no amount of C can save it. Determinism ≠ reliability — a judge can give the same verdict every time, but if that verdict is order-sensitive, it is systematically untrustworthy.

Finding 3: Leaderboard rankings are unstable across benchmarks.

The same model can rank first on MT-Bench and near-last on JudgeBench. "Best judge" conclusions drawn from a single benchmark may be wrong. Leaderboards are measurement instruments, not arbiters.

Finding 4: Holistic scoring > pairwise scoring.

Pairwise win-rate is highly order-sensitive. Holistic rubric scoring (rating a single response 1–10) is order-invariant because there is only one input. If you care about reliability, use holistic scoring.

What This Means

The paper's impact is that it turns a vague intuition — "LLM judges are unreliable" — into a computable formula. TT = C × (A − φ/2) turns "reliability" from a fuzzy concept into a reportable metric.

For practitioners, the advice is direct:

1. Report TT, not just accuracy. Your judge is 90% accurate? Fine — but what's the flip rate? At 80% flips, the TT ceiling is 60%. 2. Test order sensitivity. Swap A/B positions and rerun. This is the cheapest reliability check. 3. Prefer holistic scoring over pairwise. If pairwise is mandatory, at least report the order-flip rate. 4. Format choice matters. Verdict-first JSON is more reliable than Direct prompting because it forces the model to decide first, then justify — reducing justification-driven verdict bias.

A Deeper Insight

When we use LLMs as judges, what are we actually measuring?

If verdicts swing wildly with prompt format, presentation order, and sampling temperature, does the verdict reflect the evaluated model's capability — or the judge's own preferences?

The question isn't new. But previously we only had the intuition "LLM judges have biases." Now we have a formula — TT = C × (A − φ/2) — that decomposes reliability into three independently measurable dimensions: determinism (C), accuracy (A), and order sensitivity (φ). Each can be optimized separately.

Note the "Not Equal" in the title. Not all verdicts are equal. A 90%-accurate verdict with an 80% flip rate has only ~50% trustworthiness. An 80%-accurate verdict with a 0% flip rate is 80% trustworthy. The latter deserves more trust.

This belongs to the same family as "recall ≠ audit quality" (the Rigged Backtest paper's finding): single-dimension metrics can severely overestimate real capability. Evaluators themselves need to be evaluated — and that meta-evaluation cannot rely on accuracy alone.

Limitations

The TT formula relies on an independence assumption (A2): that reproducibility (N), order-invariance (P), and accuracy (R) are independent. The authors checked this in the appendix and found the gap between T_closed and T_joint is usually within 5pp — an "empirically bounded conservative approximation." But the independence assumption can fail in extreme cases (e.g., temperature = 0, where N is identically 1).

Also, only 6 models were tested. A broader model spectrum (especially open-source models) may exhibit different reliability patterns.

Conclusion

"100% consistent but only 51% correct" is a number everyone using LLM judges should remember. It reveals a counterintuitive truth: determinism is necessary but not sufficient for reliability. A judge that gives the same verdict every time may just be making the same systematic error.

Next time you see a paper claiming "LLM judge accuracy of 90%," ask first: what's the flip rate?

---

Paper: https://arxiv.org/abs/2610.12083 HTML version: https://arxiv.org/html/2610.12083v1