You ask an AI a question and it gives a flawed answer. You say, "The data in point three is wrong—it should be from 2023." The AI corrects the answer, and now it is correct.
Then you give both the pre-correction and post-correction versions to another LLM acting as a judge and ask which is better.
It picks the uncorrected one.
This is not a joke—it is the core experimental result of this paper.
A Counterintuitive Finding
Shachar Don-Yehiya et al. (Hebrew University + IBM Research) discovered a systematic phenomenon: when a model improves an answer based on user feedback, LLM judges systematically prefer the original, unimproved version.
More specifically: across four different corruption types, LLM judges consistently rated the "unfixed" version as better than the "fixed" one—even though the fixed version objectively resolved more issues.
Experimental Design
The researchers built two datasets to isolate the true effect of feedback:
Synthetic data: The researchers deliberately injected four types of corruption into answers—factual errors, logical flaws, formatting problems, and relevance bias. The model was then given user feedback and tested on whether it could repair the damage.
Natural data: Feedback and corrections collected from real conversations, validating whether the synthetic conclusions hold in real-world scenarios.
Key comparison: the same question, with the model generating corrections under "with feedback" and "without feedback" conditions, evaluated using issue resolution rate and LLM judge preference.
Feedback Does Work
On issue resolution rates, the effect of feedback is clear:
- Synthetic data: versions produced with feedback fixed more issues than those without, across all four corruption types
- Natural data: the same pattern holds; the improvement from feedback is consistent
- On samples with "large improvements," judges consistently prefer the uncorrected versions
- The share of preferences for uncorrected versions far exceeds that for corrected ones
- Even when the corrected version objectively fixes the corruption, judges still choose the uncorrected one
- Omission blindness: LLMs can detect "what is present" but not "what is missing." Feedback signals are precisely information about "what should be there but the model lacks."
- Judgment-gate decoupling: a model may "know" feedback is useful (at the training-signal level), but the evaluation gate never consults that knowledge.
- Scalar illusion: managing "model quality" as a scalar, ignoring that "what the model knows" and "what the model does not know" are two independent dimensions.
Feedback is not noise. It carries information the model itself cannot generate—the user knows what the answer should look like, and that knowledge is not in the model's weights.
But Judges Cannot Hear the Signal
The problem lies in the evaluation stage. When LLM judges perform pairwise comparison:
This means: if you use LLM-as-judge to evaluate the effect of feedback, you will conclude that "feedback doesn't help"—but that conclusion is wrong.
Why Feedback Is a "Unique Signal"
The paper's key claim: the signal carried by user feedback is not encoded in model weights, and therefore cannot be acquired through distillation or self-improvement.
This sounds abstract, but the logic is direct:
1. When a model generates an answer, it can only rely on knowledge in its weights. 2. User feedback comes from external world knowledge—the user knows the correct answer; the model does not. 3. When the model corrects an answer based on feedback, the corrected answer contains information the model itself could not produce. 4. An LLM judge is the same model (or from the same family), and likewise lacks this external knowledge. 5. So the judge cannot recognize the value of the correction—it cannot see what it does not know.
This is isomorphic to "omission blindness": LLMs can detect "what is there" but not "what should be there." Here, LLMs can assess "the quality of an answer" but cannot assess "whether the answer contains information the model itself does not know."
A Challenge to the Evaluation Paradigm
The paper's deeper contribution is exposing a systematic bias in the evaluation paradigm:
The current LLM evaluation paradigm assumes "the model knows what a high-quality answer looks like." But when answer quality depends on external information the model itself does not know, that assumption collapses.
This is not a problem solvable by tuning. It is structural: when model A evaluates whether model A (or a related model B) benefits from external signals, model A can never see the parts it does not know.
The paper calls on the community to develop "evaluation frameworks that reflect the true utility of feedback." But how exactly? The paper offers no answer—an open question that may matter more than the findings themselves.
Practical Implications
For teams doing RLHF and feedback-based training:
1. Do not rely solely on LLM-as-judge to evaluate feedback effectiveness. If your feedback signal genuinely carries information the model does not know, LLM judges may systematically underestimate it. 2. Build evaluation sets with ground truth. Assess feedback effects in scenarios with known correct answers, rather than relying on model self-assessment. 3. The value of feedback is not in "making the model better" but in "letting the model know what it does not know". If your feedback only makes the model repeat what it already knows, it is indeed noise.
Conceptual Lineage
This paper resonates with several existing concepts:
---
Paper: arXiv:2609.02859 Authors: Shachar Don-Yehiya, Leshem Choshen, Omri Abend et al. (Hebrew University / IBM Research / MIT) Code: github.com/shachardon/feedback-blindspot Data: huggingface.co/datasets/shachardon/feedback-blindspot