LLM-as-Judge Systematically Prefers Worse Answers: User Feedback Is a Signal Judges Can't Hear

You ask an AI a question, and it gives a flawed answer. You point out an error, and the AI corrects itself. Then you show both versions to another LLM acting as a judge and ask which is better.

It picks the uncorrected one.

This is not a joke—it is the core experimental finding of the paper discussed in this post.

The Counterintuitive Finding

Shachar Don-Yehiya et al. (Hebrew University + IBM Research) found a systematic phenomenon: when a model improves its answer based on user feedback, LLM judges systematically prefer the original, unimproved version.

More specifically: across four different corruption types, LLM judges consistently rated the uncorrected version as better than the corrected one—even though the corrected version objectively fixed more issues.

Experimental Design

The researchers built two datasets to isolate the real effect of feedback:

Synthetic data: The researchers deliberately injected four types of corruption into answers—factual errors, logic gaps, formatting issues, and relevance drift. The model was then given user feedback to see whether it could repair the answer.

Natural data: Feedback and corrections collected from real conversations, verifying whether the synthetic results hold in real-world scenarios.

The key comparison: the same question, with the model generating corrections under "with feedback" and "without feedback" conditions, evaluated using issue resolution rate and LLM judge preferences.

Feedback Does Help

On issue resolution rates, the effect of feedback is clear:

• Synthetic data: versions with feedback fixed more issues than versions without, across all four corruption types
• Natural data: the same pattern holds; the improvement from feedback is consistent

Feedback is not noise. It carries information the model itself cannot generate—the user knows what the answer should look like, and that knowledge is not in the model's weights.

But Judges Can't Hear the Signal

The problem appears at the evaluation stage. When LLM judges perform pairwise comparisons:

• On samples with large improvements, judges consistently prefer the uncorrected version
• The rate at which judges prefer uncorrected versions far exceeds the rate for corrected ones
• Even when the corrected version objectively fixes the corruption, judges still pick the uncorrected one

This means: if you use LLM-as-judge to evaluate the effect of feedback, you will conclude that "feedback doesn't help"—but that conclusion is wrong.

Why Feedback Is a "Unique Signal"

The paper's key argument: the signal carried by user feedback is not encoded in model weights, and therefore cannot be obtained through distillation or self-improvement.

The logic is straightforward:

1. When a model generates an answer, it can only rely on knowledge in its weights. 2. User feedback comes from external world knowledge—the user knows the correct answer; the model does not. 3. When the model corrects an answer based on feedback, the corrected answer contains information the model itself could not produce. 4. The LLM judge is the same model (or a same-family model), so it also lacks this external knowledge. 5. Therefore the judge cannot recognize the value of the correction—it cannot see what it does not know.

This is isomorphic to "omission blindness": LLMs can detect what is present but not what should be present. Here, an LLM can assess the quality of an answer but cannot assess whether it contains information it does not itself possess.

A Challenge to the Evaluation Paradigm

The paper's deeper contribution is exposing a systematic bias in the current evaluation paradigm:

Current LLM evaluation assumes that "the model knows what a high-quality answer looks like." But when answer quality depends on external information the model itself does not know, this assumption collapses.

This cannot be fixed by tuning. It is structural: when model A evaluates whether model A (or a same-family model B) benefits from external signals, model A can never see the part it does not know.

The paper calls on the community to develop "evaluation frameworks that reflect the true utility of feedback." But how, specifically? The paper does not answer—that open question may matter more than the paper's findings themselves.

Practical Implications

For teams doing RLHF and feedback-based training:

1. Do not rely solely on LLM-as-judge to evaluate feedback. If your feedback signal genuinely carries information the model lacks, LLM judges may systematically underestimate it. 2. Build evaluation sets with ground truth. Assess feedback effectiveness in settings with known correct answers rather than relying on model self-assessment. 3. The value of feedback is not in "making the model better" but in "telling the model what it doesn't know." If feedback only makes the model repeat what it already knows, it really is noise.

Conceptual Context

This paper resonates with several existing concepts:

• Omission blindness: LLMs can detect what is present but not what is missing. Feedback signals are precisely information about "what should be there but isn't."
• Judgment-gate decoupling: a model may "know" feedback is useful (at the training-signal level), but the evaluation gate never consults that knowledge.
• Scalar illusion: managing "model quality" as a scalar ignores that "what the model knows" and "what the model doesn't know" are independent dimensions.

The finding that "user feedback is a unique signal" fundamentally means: a model's knowledge boundary is a real boundary, not a point on a scalar. Information beyond the boundary is invisible to the model—including to its own evaluator.

---

Paper: arXiv:2609.02859 Authors: Shachar Don-Yehiya, Leshem Choshen, Omri Abend, et al. (Hebrew University / IBM Research / MIT) Code: github.com/shachardon/feedback-blindspot Data: huggingface.co/datasets/shachardon/feedback-blindspot