LLM Judging Is Harder Than Generating: A Three-Year-Old Default Assumption Falsified

Forum topic · ✨步子哥 · 2026-06-29

Summary

A controlled study by Sambaran Bandyopadhyay (Adobe Research) challenges the foundational assumption of LLM-as-a-Judge pipelines: that judging answers is easier than generating them. Using in-context QA on four benchmarks (SQuAD 2.0, DROP, HotpotQA, MuSiQue) to eliminate parameter-knowledge confounds, the study finds that generation accuracy exceeds self-evaluation accuracy on three of four benchmarks. Attention analysis shows models pay 3-5x less attention to the context when judging, effectively taking shortcuts instead of verifying answers against the context. A systematic acquiescence bias—models defaulting to accepting answers—further degrades judgment quality. LoRA fine-tuning experiments show generation and evaluation are coupled: training one degrades the other, and only joint training improves both, with limited gains. The exception is MuSiQue, where judging slightly beats generation, suggesting judging is easier only when reasoning complexity is high. The findings cast doubt on LLM-as-a-Judge evaluation pipelines and offer guidance for detection of lazy judging via attention analysis.

For the past three years, nearly every LLM application pipeline has been built on an unverified assumption:

Judging is easier than generating.

LLM-as-a-Judge uses large models to score other models. Self-reflection has the same model generate, critique, and revise. RLHF reward models are themselves LLMs. All of these architectures assume that a model's ability to judge whether an answer is correct should exceed its ability to generate a correct answer itself.

Intuitively this seems reasonable—grading exams is easier than taking them; reviewing papers is easier than writing them. Sambaran Bandyopadhyay of Adobe Research decided to formally test this assumption under controlled conditions.

The result: on three of four benchmarks, generation accuracy exceeded self-evaluation accuracy. Judging is not easier than generating—it is harder.

Controlled experiment: eliminating parameter-knowledge interference

Previous work questioning the "judging is easier" assumption tested open-domain QA, where a model judging an answer could draw on its stored knowledge, mixing knowledge sources with generation and preventing a clean comparison.

This paper's design is elegant: use in-context QA—give the model a passage and have it answer based solely on that text. Both generation and judging are restricted to the same passage, eliminating parameter-knowledge interference.

Four benchmarks:

SQuAD 2.0: extractive QA on single passages
DROP: reading comprehension requiring discrete reasoning
HotpotQA: multi-hop QA
MuSiQue: multi-hop QA, harder than HotpotQA

Two models were tested. Each model first generated an answer, then judged whether its own answer was correct.

On 3/4 benchmarks, generation beat judging

The results were surprising:

On SQuAD 2.0, DROP, and HotpotQA, generation accuracy was higher than self-evaluation accuracy. Only MuSiQue (the hardest) showed evaluation slightly above generation.

This means models are better at *solving problems* than at judging whether their solutions are correct.

Why? Attention analysis provided the answer.

When judging, the model barely reads the context

The team analyzed attention distributions in generation vs. judging modes and found a striking difference:

In judging mode, the model's attention to the context is only 1/3 to 1/5 of that in generation mode.

When generating, the model carefully reads the context, finds answers, and organizes language. When judging, the model barely looks back at the context and pays little attention to the candidate answer—it makes a quick "does this look right?" intuitive judgment rather than careful verification.

This explains why judging is worse: judging itself isn't harder—the model is being lazy when judging. It skips the "carefully compare answer with context" operation and takes a shortcut.

Acquiescence bias: the model is a people-pleaser

Deeper analysis revealed a systematic bias: models tend to say "correct." In self-evaluation, the rate at which models accept wrong answers (saying "yes") far exceeds the rate at which they reject correct answers (saying "no").

This is *acquiescence bias*—the model defaults to "yes" unless there is strong evidence for "no." In generation, the model must actively produce content with no default to lean on; in judging, "correct" is a low-resistance default.

LoRA fine-tuning: can training fix this?

The team ran four LoRA experiments: train generation only, train judging only, train both, train neither (baseline).

The results were more discouraging:

Generation-only training: generation accuracy rose, but judging accuracy *dropped*—the model became more inclined to accept any answer (over-acceptance), including wrong ones.

Judging-only training: judging barely improved, and generation regressed.

Training both: the only configuration improving both, but with limited gains.

This shows generation and judging are not independent skills—they share underlying mechanisms. Training one interferes with the other; there is no free lunch of "improving judging without touching generation."

The MuSiQue exception: the peculiarity of multi-hop reasoning

The only benchmark where judging slightly beat generation was MuSiQue. Why?

MuSiQue requires multi-hop reasoning—answers must chain information across multiple passages. During generation, the model must complete every reasoning step itself; any error breaks the final answer. During judging, the model only needs to check whether a given answer is consistent with the context—it doesn't need to complete the reasoning chain itself.

This suggests a more precise conclusion: the "judging is easier" assumption holds only when reasoning complexity is high enough. On simple tasks, generation is actually easier, because generating forces the model to read the context carefully, while judging lets it be lazy.

Warnings for LLM-as-a-Judge

This paper is a serious warning for current LLM evaluation pipelines:

1. Don't assume judging is easier than generating. On simple tasks this assumption is wrong. Using an LLM as a judge may be *less* accurate than having the LLM solve the problem directly.

2. Attention allocation is key. Judgment quality depends on whether the model actually reads the context and candidate answers. "Lazy judging" can be detected via attention analysis.

3. LoRA cannot decouple generation and judging. Fine-tuning a model to "judge without learning to generate" is unrealistic; the two share underlying capabilities.

4. Multi-hop tasks are the exception. Only when reasoning complexity is high enough is judging truly easier than generating.

A deeper lesson

The deepest lesson may not be the specific finding that judging is harder than generating, but the methodology: when you have a widely accepted assumption, don't assume it's correct—design a controlled experiment to test it.

LLM-as-a-Judge has become the industry default, powering leaderboards, automated benchmarks, and production quality evaluation. Yet its foundational assumption was never cleanly verified. Now we know: on at least 3/4 of tasks, that assumption is wrong.

Next time you see evaluation results using "GPT-4 as judge," remember: the judge may be less reliable than the player.

---

Paper link: https://arxiv.org/abs/2606.28050