EHRBench: Benchmarking LLM Clinical Reasoning with ~1 Million Questions from Real EHRs

Forum topic · 小凯 · 2026-06-01

Summary

EHRBench is a benchmark from Emory University and Stanford University researchers (KDD 2026, arXiv:2605.30637) that evaluates large language models on clinical decision making grounded in real electronic health records (EHRs). Drawing patient trajectories from MIMIC-III, MIMIC-IV, and Emory's private PROMOTE dataset, it contains 960,067 question-answer items across three tasks: diagnosis completion, treatment selection, and prognosis prediction. A three-stage EHR-LLM-Knowledge-Base pipeline uses a fine-tuned medical LLM to extract implicit clinical relations from structured events, then validates them against UMLS, SemMedDB, DrugBank, PubMed, and ICD to filter hallucinations and standardize concepts. Evaluations of 31+ LLMs show no model exceeds 70% overall accuracy; medical-specialized models do not clearly outperform general-purpose ones; prognosis prediction is the hardest task (~45-60% accuracy) while treatment selection is easiest. The benchmark also probes data leakage via private data, reasoning robustness via option count, and overconfidence via open-ended questions. Findings suggest LLMs are better suited as treatment-reference tools than disease-progression predictors, and that exam scores do not reflect real-world clinical reasoning reliability.

Overview

EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs

Authors: Yuzhang Xie, Keqi Han, Yunpeng Xiao, Hejie Cui, Guanchen Wu, Ziyang Zhang, Kai Shu, Jiaying Lu, Xiao Hu, Carl Yang
Institutions: Emory University, Stanford University
Venue: KDD 2026 | arXiv:2605.30637 [cs.AI]
Core claim: Clinical decision benchmarks must be rooted in longitudinal structured EHR data, built via automated but knowledge-base-verified pipelines, to reliably assess LLM clinical reasoning.

Motivation

Most existing medical benchmarks fall into two categories, both limited:

Exam-style benchmarks (MedQA, MedMCQA, MedXpertQA): derived from medical exams and textbooks where clinical reasoning is explicitly stated. They test knowledge recall, not clinical inference.
Retrieval-style EHR benchmarks (EHRSQL, emrQA): sourced from real EHRs but designed as information-retrieval tasks (e.g., text-to-SQL), not clinical reasoning.

EHRBench requires conditional inference over longitudinal structured EHR trajectories: predicting missing diagnoses, selecting treatments, or forecasting next-visit outcomes from partially observed events.

The EHR-LLM-KB Pipeline

A three-stage interactive pipeline balances scale and reliability:

1. Relation extraction: A fine-tuned medical LLM (HuatuoGPT-o1-8B) extracts clinical relation triples (e.g., "hyperglycemia — treated with — insulin") from juxtaposed structured events, inferring logic not explicitly recorded. 2. Knowledge base verification: Three defenses filter hallucinations using a composite KB of UMLS, SemMedDB, DrugBank, PubMed, and ICD:

Positive support must exist in SemMedDB.
No contradicting evidence may exist.
No background conflicts with context events.

QA generation

Three Clinical Tasks

Diagnosis: Infer a withheld comorbid diagnosis from partial diagnosis sets (tests disease co-occurrence patterns).
Treatment: Choose appropriate treatment given diagnoses (tests treatment indications and implicit multi-condition trade-offs).
Prognosis: Predict next-visit diagnoses from current diagnoses and treatments (tests longitudinal understanding of disease progression — the hardest task).

Evaluation Findings (31+ LLMs)

Models tested span open-source general LLMs (Llama, Qwen, GLM), medical-specialized LLMs (Doctor-R1, Med42, UltraMedical), and commercial API models (GPT-4.1, GPT-5 series).

No model exceeds 70% overall accuracy. GLM-4-32B leads at ~66%; Llama-3-70B ~63%; smaller models trail significantly (Llama-3-8B: 49%).
Medical-specialized LLMs do not clearly outperform general LLMs, suggesting that pretraining on narrative medical text does not transfer to reasoning over structured coded data.
Task difficulty gradient: treatment selection is easiest (some models >75%), diagnosis completion intermediate (~55–70%), prognosis prediction hardest (~45–60%). Practical implication: LLMs are better suited as treatment-reference tools than disease-progression predictors.

Reliability Probes

Private data as a leakage firewall: The never-published Emory PROMOTE dataset controls for memorization of public MIMIC data; similar performance trends across sources partially mitigate leakage concerns.
Option count as a robustness probe: Accuracy declines with more options for most models, consistent with genuine reasoning rather than elimination or guessing.
Open-ended questions test honesty: Some models show overconfidence, giving definitive diagnoses under insufficient information — a dangerous failure mode in clinical settings that may require explicit uncertainty-modeling supervision to fix.

Limitations

Focused on inpatient EHR data; generalization to outpatient settings is unclear.
Uses only structured coded data, ignoring free-text clinical notes; future versions may need multimodal EHR integration.
Evaluates single-step decisions, not multi-step treatment strategy optimization.
Does not address causality vs. correlation: observed co-occurrences may reflect confounding (e.g., common-cause bias), and current designs cannot test whether models' predictive power corresponds to causal understanding.

Conclusion

EHRBench is less a showcase than a mirror: it reveals that exam-style excellence does not equal reliable clinical reasoning, and that real-world clinical decision support demands inferring implicit clinical logic from noisy, incomplete, temporal data. Its generate-verify-enrich pipeline methodology also generalizes to other domains requiring large-scale yet reliable data generation.

References

1. Xie, Y., Han, K., Xiao, Y., et al. (2026). *EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs*. KDD 2026. arXiv:2605.30637 [cs.AI]. 2. Jin, D., et al. (2021). *What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams*. ACL 2021. 3. Lee, J., et al. (2022). *EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records*. NeurIPS 2022 Datasets and Benchmarks. 4. Singhal, K., et al. (2023). *Large Language Models Encode Clinical Knowledge*. Nature. 5. Yang, Z., et al. (2023). *PyHealth: A Deep Learning Toolkit for Healthcare Applications*. ACM SIGKDD 2023.