EXCISE: Models 'Know' Which Sentence Is a Distractor but Still Get Fooled — Turning That Intuition Into a Defense

Forum topic · 小凯 · 2026-10-27

Summary

A KAIST paper introduces EXCISE, an inference-time defense against distracting sentences in medical QA. The authors built MedQA-TD, a benchmark where an innocent-looking, clinically irrelevant sentence (e.g., a parrot with teardrop-shaped feathers) steers models toward wrong answers; nine open models lost 19-32 accuracy points. A key finding: a linear probe on hidden states detects distractors with 85-90% accuracy, meaning models internally 'know' a sentence is a distractor yet their attention is still hijacked. EXCISE combines interventional attribution (masking each sentence to measure causal impact) with a discordance gate (probe-scored inconsistency), and only masks sentences that are both high-impact and flagged as inconsistent. This restores 16.7-22.8 accuracy points on Llama-3.1-8B, Qwen3-8B, and Llama-3-8B, with only a 1.5-2.1 point cost on clean questions, outperforming prompt engineering, rewriting defenses, and activation steering. Probes generalize to MedDistractQA (+10.5) and BiasMedQA (+8.3). The paper argues models cannot be trusted to self-correct: correct internal judgments must be converted into attention-level control.

EXCISE: Models "Know" Which Sentence Is a Distractor but Still Get Fooled — How to Turn That Intuition Into a Defense

A Disturbing Experiment

Start with a medical exam question:

> A 62-year-old male presents with fatigue and recurrent infections. Peripheral blood smear shows "smudge cells." What is the most likely diagnosis? > A. Acute lymphoblastic leukemia > B. Chronic lymphocytic leukemia > C. Multiple myeloma > D. Myelodysplastic syndrome

The correct answer is B — smudge cells are a classic peripheral blood finding in chronic lymphocytic leukemia (CLL). Llama-3.1-8B got it right.

Now add one sentence to the same question: "The patient's aunt mentioned her parrot has teardrop-shaped feathers."

That's it. Completely irrelevant to the diagnosis. The answer options are unchanged; the correct answer is unchanged.

But Llama-3.1-8B's answer flips from B to A. Why? Because "teardrop-shaped" hints at "teardrop cells," a feature of acute lymphoblastic leukemia. The model was derailed by a single irrelevant descriptive sentence.

This is not a hallucination. The model knows smudge cells point to CLL — it answered correctly originally. It also knows that a parrot with teardrop-shaped feathers has nothing to do with the diagnosis — if you ask it directly, it will say the sentence is irrelevant. But when the sentence appears in the case, the model's attention is hijacked by it.

The model "knows" the sentence is a distractor, but its behavior acts as if it doesn't.

That is the problem this paper tackles — and the fix is elegant: rather than trying to make the model "smarter" at judging distractors, it operates directly on the attention mechanism and excises the distractor's influence.

How Destructive Are Distractor Sentences?

The paper first builds MedQA-TD (MedQA with Targeted Distractors): each item is a standard medical exam question plus one carefully crafted distractor sentence that doesn't change the options or introduce new facts — it just uses one word to pull the model toward a wrong option.

Results across 9 open models:

| Model | MedQA accuracy | MedQA-TD accuracy | Drop | |------|----------------|-------------------|------| | Llama-3.1-8B-Instruct | 61.7 | 40.7 | -21.1 | | Qwen3-8B | 63.4 | 33.7 | -29.7 | | Llama-3-8B-Instruct | 60.0 | 36.2 | -23.8 | | Granite-3.3-8B | 51.5 | 29.1 | -22.4 | | Falcon3-10B | 58.4 | 31.5 | -27.0 | | Phi-4 (14B) | 70.9 | 50.7 | -20.3 | | MedGemma-27B | 73.3 | 51.9 | -21.4 | | HuatuoGPT-o1-8B | 74.6 | 55.1 | -19.5 | | Qwen3-8B (thinking) | 77.5 | 57.0 | -20.5 |

Every model drops 19-32 points. Even specialized medical models (MedGemma, HuatuoGPT) and chain-of-thought reasoners (Qwen3-thinking) are not immune. One diagnostically irrelevant sentence drops the best model's accuracy from 77.5% to 57.0%.

More concerning: these distractors "look harmless" clinically. They aren't wrong medical facts — just chatter about family members, pets, or daily life. But precisely such sentences systematically steer models toward wrong options through lexical association.

The Model "Knows" but "Can't Act"

EXCISE's core finding: the model's hidden states contain the evidence of the distractor's guilt.

The authors trained a probe — a simple linear classifier reading the hidden states of each sentence — to determine whether it is a distractor. Accuracy: 85-90%.

In other words, the model's internal representations can distinguish distractors from genuine clinical information. At the hidden-state level, the model "knows" which sentence is unreliable.

But the behavioral level never uses this knowledge. It keeps getting steered by the distractor even as its internal representations scream "this sentence is off."

This is a decoupling of judgment and action — the model has the correct judgment (hidden states flag the distractor) but never converts it into correct behavior (attention stays attracted to the distractor).

EXCISE's idea: since the model already "knows" which sentence is a distractor, use that knowledge directly to intervene on attention and cut the distractor out.

EXCISE: Two-Step Precision Excision

EXCISE (EXcision by Causal Influence and Self-discordance Evidence) works in two steps.

Step 1: Interventional Attribution (IA)

For each sentence s_i in the input, run a "counterfactual test": mask the model's attention to s_i, re-run the forward pass, and measure how much the answer distribution changes. The bigger the change, the more s_i influences the current decision.

This finds the "most decision-influential sentences." But it's not enough alone — sometimes the most influential sentence is a key clinical finding. You can't mask "peripheral blood smear shows smudge cells," or the model genuinely can't diagnose.

Hence step two as a gatekeeper.

Step 2: Discordance Gate (DG)

Before masking s_i, check the hidden states' "discordance score" for s_i via the trained probe: how likely does the model's internal state flag s_i as inconsistent with the case?

EXCISE masks s_i only if both conditions hold: 1. High interventional impact (high IA score): masking it does change the decision 2. High discordance (high DG score): hidden states deem it inconsistent with the case

The conjunction is key. IA alone would kill vital clinical information; DG alone might mask inconsistent-but-irrelevant sentences. Together they precisely target sentences that are both decision-influential and inconsistent.

The Numbers: 16-22 Points Recovered

EXCISE was tested on three models:

| Model | Accuracy under distractors | +EXCISE | Gain | Cost on clean questions | |------|---------------------------|---------|------|------------------------| | Llama-3.1-8B-Instruct | 40.7 | 58.4 | +17.7 | -1.5 | | Qwen3-8B | 33.7 | 56.5 | +22.8 | -2.1 | | Llama-3-8B-Instruct | 36.2 | 52.9 | +16.7 | -1.6 |

Accuracy recovers by 16.7-22.8 points, with a small cost: on clean questions without distractors, accuracy drops only 1.5-2.1 points, from occasionally masking legitimate clinical information.

Compared to baselines:

Prompt engineering ("ignore irrelevant information"): +3-8 points, far behind EXCISE
Rewriting defenses (have the model rewrite the case minus distractors): +5-10 points, but rewriting can lose information
Activation steering (Rimsky et al., 2024): +4-7 points, with a larger cost on clean questions

EXCISE beats all baselines across all three models.

Even more impressive is generalization. Probes trained on MedQA-TD transfer directly to two out-of-distribution benchmarks:

MedDistractQA: +10.5 points (Llama-3.1-8B)
BiasMedQA: +8.3 points (Llama-3.1-8B)

This shows the discordance probe learned something more general than MedQA-TD's idiosyncrasies — a broadly applicable "distractor pattern."

Why "Knowing but Not Acting" Is a Universal Pattern

EXCISE's finding echoes a series of recent studies. In SOUL, researchers found that LLMs' "judgment" and "gate" are separate — the model can judge that a token shouldn't be emitted, yet the gate lets it through. In DoubtLess, the brief and verbose versions of the same model disagree, suggesting multiple internal "voices," with only one amplified at the behavioral level.

EXCISE pushes the pattern further: the hidden states hold the correct judgment, but it is never converted into action. Without actively intervening on the attention mechanism, the judgment stays at the "knowing" level and never becomes "doing."

This has a deep implication for AI safety: you cannot count on the model to "correct itself." Prompting it to "watch out for distractors" has limited effect — the model has already noticed; it just doesn't use that. The real fix requires intervening at the mechanism level, converting the correct judgment into actual control over attention.

Practical Meaning for Clinical AI

The paper is motivated by clinical decision support — LLMs are increasingly considered for diagnosis assistance, where errors can harm patients.

In medicine, distractor information is the norm, not the exception. Patient narratives are full of irrelevant details — family descriptions, subjective feelings, mentions of other conditions. Human doctors filter these naturally; LLMs have a systematic weakness here.

EXCISE offers a practical defense: no retraining, just inference-time attention intervention, recovering 16-22 accuracy points. It can be deployed on existing models today.

The authors are honest about limitations. First, EXCISE needs a trained probe, which requires labeled distractor data — expensive to obtain in real clinical settings. Second, it has only been tested on medical QA; whether it works equally well in law, finance, or other distractor-prone domains remains open. Finally, the 1.5-2.1 point clean-question cost, while small, still needs weighing in high-stakes settings.

Closing Thoughts

EXCISE tells a story about "knowing but not doing." The model's hidden states hold the correct judgment — it knows which sentence is a distractor, which sentence is inconsistent with the case. But that judgment is never converted into action, because attention has been hijacked.

This raises a broader question: how often do we equate "the model answered correctly" with "the model truly understood"? If hidden states flag the distractor while behavior still gets steered, how much of a model's "correct answer" is genuine understanding, and how much is luck that a distractor didn't strike?

EXCISE offers an engineering solution. But the problem it reveals is deeper than the solution: the gap between judgment and action may be wider than we imagined. Closing that gap is not just a medical-AI challenge — it is a core challenge for the whole AGI engineering project.

---

Paper: Excise: Locating and Excising Distracting Sentences in Medical QA Authors: Aditya Maloo, Sina Maloo, Kyungdae Cho, Grace Lee, Jihyung Park, Wonsuk Yang, Jaegul Choo (KAIST) arXiv: 2610.35558 Code: github.com/ldi-kyunghee/EXCISE