MedMisBench: Medical LLMs Show Severely Overestimated Epistemic Resilience Under Misleading Contexts

Forum topic · 小凯 · 2026-06-16

Summary

MedMisBench (arXiv:2606.12291), from Oxford, Washington, UCL, and Waterloo, is a benchmark measuring how well large language models resist misleading medical information. Using 10,932 medical questions and 48,889 paired misleading contexts validated by a 14-member clinical expert panel across 7 countries, the study shows models averaging 71.1% clean accuracy drop to 38.0% after a single false claim is injected, with a 51.5% attack success rate and 38.2% clinical harm rate. Misinformation framed as coming from authoritative sources (clinical guidelines, discharge records) succeeds 69.5% of the time, versus only 18.5% when attributed to patient self-report. Counterintuitively, higher reasoning effort does not guarantee more resilience—Gemini-3.1-pro's attack success rate rose from 61.7% to 65.0% in high-reasoning mode. Web search and defensive prompting reduce but do not eliminate the vulnerability, especially for weaker models. The benchmark argues medical AI evaluation must shift from knowledge testing to resilience testing.

> Paper: MedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical Context > arXiv: 2606.12291 | June 2026 > Institutions: Oxford × University of Washington × UCL × University of Waterloo > Code: https://github.com/AI4HealthResearch/MedMisBench

Key points

High exam scores are deceptive: LLMs averaging 71.1% clean accuracy fall to 38.0% when a single misleading medical claim is injected, with a 51.5% attack success rate (ASR) on previously correct answers and a 38.2% clinical harm rate assessed by experts.
Authority is the most dangerous vector: misinformation framed as coming from authoritative sources achieves 69.5% ASR, versus 65.2% for neutral false statements and only 18.5% for patient self-report.
More reasoning ≠ more resilience: Gemini-3.1-pro's clean accuracy barely changed in high-reasoning mode (83.1% → 83.5%), but Type 1 ASR rose from 61.7% to 65.0%. Gemini-3.1-pro scored highest on exams (83.5%) yet showed a 65.0% ASR; GPT-5.4 was comparatively most resilient (36.1% ASR).
Mixed-evidence scores can be illusory: in Type 2 (mixed evidence) settings accuracy is nearly unchanged (70.5% vs 71.1%), but 84.3% of Type 2 failure cases involve models retaining false judgments while coincidentally selecting the correct option.

Benchmark design

MedMisBench uses 10,932 medical questions and 48,889 paired misleading contexts, validated by a 14-member clinical expert panel from 7 countries (average 3 years of clinical experience). Misleading contexts are generated along two orthogonal dimensions:

Content types (5):

Relation/sequence reversal
Threshold/reference tampering (ASR 60.9%)
Cue remapping
Spurious anchoring (weakest, ASR 20.9%)
Exception poisoning (ASR 64.1%) — fabricating contraindications or exceptions

Source credibility framing (3):

Patient self-report (ASR 18.5%)
Neutral false statement (ASR 65.2%)
Authoritative source — "according to the latest clinical guidelines…" (ASR 69.5%)

Paired testing protocol: 1. Confirm the model answers the original question correctly (clean accuracy) 2. Inject misleading context 3. Test whether the model still answers correctly

If step 1 is correct but step 2 fails, resilience is lost (attack succeeded).

Clinical harm assessment

Expert review of 89 cases found:

38.2% rated as worst-case output (wrong answer + accepted misinformation + serious potential harm)
46.1% rated as wrong answers with low-to-moderate harm
Only 5.6% rated as correct rejection of misinformation

Mitigations: what helps and what doesn't

Web search: reduced Gemini-3.1-pro's Type 1 ASR from 81.5% to 16.1%, but Gemini-3.1-flash-lite only improved from 54.0% to 40.7%. External evidence helps only if the model can arbitrate between retrieved evidence and injected misinformation.
Defensive prompting: explicitly warning that context may contain misinformation reduces ASR by 10–14 percentage points, but residual resilience loss remains significant. Knowing a threat exists ≠ being able to resist it.

Limitations

1. Static benchmark with potential data contamination risk 2. Primarily English medical QA; resilience in other languages may differ 3. Misinformation generated by an LLM (Gemini-3.1-flash), albeit with filtering and clinical validation 4. Real-world deployment scenarios (e.g., RAG systems repeatedly retrieving wrong documents) may be more subtle and persistent than one-shot injection