LLM Judges Verify Presence, Not Absence: Omission Blindness Breaks AI Clinical Note Review
A Counterintuitive Finding
Give an LLM judge the task of reviewing AI-generated medical notes. Deliberately delete a critical piece of information — say, a patient's penicillin allergy — and ask the judge: "Is anything wrong with this note?"
You'd expect it to say yes. But across 8 different judge designs and 500 pairs of single-error notes, paired discrimination (the probability that a flawed note scores lower than a clean note) was only 0.50–0.63 for omissions — essentially a coin flip. For commissions (erroneous content added), the number was 0.79–0.94.
In other words, LLM judges can reliably find "what was added wrongly," but almost completely fail to find "what was left out."
This is not a problem with one model. It persisted across 8 judge designs, two model families, prompt optimization, voting, and rewording. The paper's title says it all: LLM Judges Verify Presence, Not Absence.
Why This Matters More Than You Think
Medical AI scribes (ambient AI note-takers) are being deployed at scale. A companion audit of three commercial AI scribes across 142 encounters and 565 notes found: one in three notes (31.3%) carried verified failures, concentrated in allergy information, medication details, fabricated patient identities, and history turned into physical exam findings in telephone visits.
The dominant failure mode is omission — notes missing information actually established during the encounter.
The industry-standard detection method is the LLM judge: a second model reads the transcript and the note, flagging problems. The paradigm assumes judges can detect omissions. That premise is wrong.
A paired discrimination of 0.5 means: notes with omissions score essentially the same as clean notes. No threshold can separate them — there is no separable signal at all.
What They Tried, and Why Nothing Worked
The paper systematically ruled out "bad prompting":
- Different model families (GPT and Claude lines)
- Different wording ("find problems" vs. "find omissions" vs. "list all inconsistencies")
- Voting (majority over multiple samples)
- GEPA prompt optimization (evolutionary search over prompt text)
- QuoteBench: matching scores masked pipeline damage (-64.3 damage + 60.7 compensation = a seemingly normal -3.6 gap)
- CROP: token-level filtering lacks counterfactual sensitivity
- Gender Bias: BBQ measures explicit bias but misses linguistic dialect bias
- Calibrated Enough to Know: models 90% know when they're unpredictable, but the action gate doesn't consult the judgment module
- Code review: LLMs find "added bugs," but can they find "missing error handling"?
- Documentation review: LLMs find "wrong descriptions," but can they find "missing sections"?
- Security review: LLMs find "present vulnerabilities," but can they find "missing safeguards"?
- Compliance review: LLMs find "violations," but can they find "required steps never executed"?
Result: all of these moved the operating point, but none restored usable detection. They shifted the trade-off between alert volume and precision, but the curve itself didn't change — the omission detection ceiling was near random.
More counterintuitively: human physicians do not show this asymmetry. On the same notes, human reviewers detected omissions and commissions symmetrically. So it's not that "omissions are inherently hard" — it's a specific blind spot of LLM judges.
The Mechanism: Omission Leaves Nothing to Point At
The paper's explanation is elegantly simple: an omission leaves nothing to point at.
A commission is "text in the note that shouldn't be there." The judge can point to that text and say "this contradicts the transcript." Detection is: find evidence in the note, compare against transcript.
An omission is "text missing from the note that should be there." Detecting it requires first building the expectation "this information should appear in the note" from the transcript, then verifying the expectation fails. But the LLM judge's working mode is "read the note, find problems" — it has no expectation list of what should be there, only a perception of what is there.
In one sentence: presence detection is a perception task; absence detection is a reasoning task. LLM judges are good at the former, poor at the latter.
The Fix That Actually Works: Convert Absence into Presence
The paper's core contribution is a surprisingly simple repair:
Don't ask the judge whether the note has problems. First, have another module list every fact established in the transcript, then run a closed presence check on each fact — "does this fact appear in the note?"
This reframes absence detection as a set of presence checks. The judge no longer needs to "reason about what should be there" — only to "perceive whether it is there."
Two independent methods converged on this restructuring:
1. Per-fact pipeline: one module produces a fact list; a second module runs a closed check per fact. At a 2.7% false-positive rate, it can pinpoint which fact is missing and how severe it is. 2. GEPA-evolved prompt: an evolutionarily optimized single-call prompt that internally executes the same "list + check each item" procedure. Detection rate 36.9% (vs. 24.6% for the pipeline, p=0.002), 6.2% false positives, roughly 1/10 the cost.
The trade-off is clear: the pipeline is precise but expensive; the evolved prompt is sensitive but noisier. Both lift omission detection from "random" to "usable."
Where Human Physicians Stand
A key experiment: on 10 notes where the pipeline and the best monolithic judge disagreed, a physician author blindly adjudicated. The physician sided with the pipeline on all 10 (p=0.002).
Subsequently, an independent physician blind-rated severity — agreeing with the first within one severity grade in both assessments.
This is the paper's heaviest hit: it's not that the pipeline is slightly better than the judge — the pipeline aligns with physician judgment while the judge systematically diverges from it.
Another Instance of the Evaluation Blind-Spot Law
This paper precisely hits a recurring pattern — the evaluation blind-spot law: evaluation systems systematically miss a class of failures, and it's exactly the most important class.
Previous cases:
Omission blindness is the purest form: the evaluator's detection capability for an entire failure class (omissions) equals random — and that class is the most frequent error type in real deployment (31.3% of notes carried omission-type failures).
A deeper isomorphism: judgment–gate decoupling. The LLM judge "knows" what a note should contain (ask it "should this note include allergy info?" and it says yes), but that judgment doesn't propagate to the "find problems" action gate. Prompt optimization can only reword the action module, not restructure it.
The essence of the fix: don't try to teach the action module to reason; decompose the reasoning task into a set of perception tasks so the action module only does what it's good at.
Implications for All LLM-as-Judge Systems
The conclusions extend far beyond clinical notes. Any scenario using an LLM judge for "completeness checking" faces the same blind spot:
The general repair template: any LLM judge task that checks completeness should first enumerate what should exist, then run a closed check on each item. This isn't a prompt trick — it's a task-architecture-level restructuring.
Limitations and Open Questions
The paper honestly acknowledges several limitations:
1. No transferable benchmark thresholds for real vendor notes. The benchmark was constructed from transcript-derived fact sheets; real notes have a different distribution, and all thresholds need recalibration. 2. The omission detection ceiling remains low. The best method (evolved prompt) reaches only 36.9% detection. The "random" problem is fixed, but "reliable" is far away. 3. Cost trade-offs. The pipeline is precise but expensive; the evolved prompt is cheap but noisier. Production deployments must weigh the specifics.
But the most important open question: is omission blindness an intrinsic limit of LLMs, or an artifact of current training paradigms? The paper doesn't answer. If future models break through on absence reasoning, this fix may become redundant. Until then, task restructuring is the only reliable defense.
A Deeper Philosophical Question
The most fascinating part of the paper is its mechanistic explanation — omission leaves nothing to point at. This is not just an LLM problem; it's a human cognition problem too: we notice what's present more easily than what's absent.
But human physicians learn "systematic checklists" through training — they don't rely on "feeling something is wrong," but on "confirming item by item what should be there." Which is exactly the paper's fix.
In other words: the LLM judge's omission blindness is, at its core, the failure to learn human checklist thinking. The repair isn't making the LLM smarter — it's giving it an externalized checklist.
This isomorphism reinforces a conclusion: many "capability deficits" of AI systems are really architecture-level task-allocation problems, not model capability problems. Decompose the task at the right level, and much of the "impossible" becomes "obvious."
---
Authors: Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
Published: 2026-08-31
Dataset: 500 pairs of single-error notes + a survey of 565 real notes across 142 encounters