Fatal Hallucinations in Clinical LLMs: Why Bigger Models Are Not Necessarily Safer

Forum topic · 小凯 · 2026-05-07

Summary

A 2026 study by a German-international research team evaluated 34 locally deployed clinical LLMs across 7 model families, 6 deployment conditions, and 200 radiology safety questions using the SaFE-Scale framework and the RadSaFE-200 benchmark (40,800 total evaluations). The central finding is a safety-accuracy decoupling: scaling model size, context length, or retrieval complexity improves accuracy but does not reliably reduce dangerous errors. Clean, clinician-curated evidence raised mean accuracy from 73.5% to 94.1% and cut high-risk errors from 12.0% to 2.6%, collapsing performance gaps between model families. In contrast, Standard RAG and Agentic RAG improved accuracy (up to 78.1%) while dangerous overconfidence remained high or even increased, and maximum-context prompting plus test-time compute yielded limited gains. The authors conclude that safety is a deployment property driven by evidence quality, retrieval design, and failure behavior—not a passive byproduct of model scale. Safety metrics should be measured and reported separately from accuracy in clinical AI evaluation, regulation, and procurement.

Key points

Clinical LLM safety and accuracy follow different scaling laws. Making models larger, stuffing context windows fuller, and building more complex retrieval pipelines can raise accuracy while safety metrics stagnate—or quietly worsen.

In May 2026, a German-international research team published a short but devastating paper evaluating 34 locally deployed clinical LLMs 🧠 across 7 mainstream model families, answering 200 radiology safety questions 📋 under 6 deployment conditions. Their conclusion: in clinical AI, safety is not a passive byproduct of model capability, but an active result of deployment conditions.

> 📌 What is SaFE-Scale? > SaFE-Scale is the paper's evaluation framework, short for *Safety-Focused Evaluation of Scaling*. Rather than only measuring accuracy, it grades each incorrect answer by clinical risk—some errors are harmless, others potentially fatal. The framework spans five dimensions: model scale, evidence quality, retrieval strategy, context length, and inference-time compute.

Scorecard across six deployment conditions

| Deployment condition | Mean accuracy | High-risk error rate | Dangerous overconfidence | Contradiction rate | |:---|:---:|:---:|:---:|:---:| | 📕 Closed-book (zero-shot) | 73.5% | 12.0% | 8.0% | 12.7% | | ✅ Clean evidence (clinician-curated) | 94.1% | 2.6% | 1.6% | 2.3% | | ⚠️ Conflict evidence | ~90%+ | ~3–5% | ~2–3% | — | | 🔍 Standard RAG | ~75% | still high | still high | ~11% | | 🤖 Agentic RAG | 78.1% | still high | ↑ higher | 9.0% | | 📄 Max-context | ~75% | not improved | not improved | — |

*Table 1: Mean performance of 34 LLMs on RadSaFE-200, from the paper's Table 1 and Figure 2.*

Clean evidence transforms everything

With clean evidence (clinician-written curated evidence), accuracy jumped from 73.5% to 94.1% 📈 while high-risk errors dropped from 12.0% to 2.6% 📉—a simultaneous improvement across *all* safety metrics, not a trade-off.

With Standard RAG, however, accuracy improved over closed-book, but high-risk error rates and dangerous overconfidence stayed high. The model answered more questions correctly, but the ones it got wrong remained potentially lethal.

> 📌 RAG vs. Agentic RAG > RAG (*Retrieval-Augmented Generation*) retrieves external documents into context before answering. Agentic RAG goes further, letting the model reason in multiple steps, search actively, and integrate evidence like an agent. Smarter in theory—but the paper shows smarter ≠ safer.

Agentic RAG: the most ironic result

Using a previously published radiology Retrieval-and-Reasoning (RaR) framework, Agentic RAG 🤖 raised accuracy from ~76% to 78.1% and cut the contradiction rate from 11.7% to 9.0%. But high-risk errors stayed flat, and dangerous overconfidence actually increased ⬆️.

The models got better at *using* evidence but not at *avoiding fatal errors*—and became more confident on wrong answers, the most clinically dangerous combination 💀.

> From Standard RAG to Agentic RAG, the centroid shift is the clearest example of decoupling: accuracy rises while dangerous overconfidence also rises—opposite to the direction expected from single-axis improvement.

\text{Accuracy} \uparrow \quad \nRightarrow \quad \text{Safety} \uparrow

Scaling laws collapse before clean evidence

Under closed-book conditions, larger models did better—accuracy ranged from 41.5% (smallest) to 88.4% (largest). But with clean evidence, all family curves collapsed to the same ceiling: DeepSeek 97.0%, Qwen 95.4%, Mistral 95.2%, even the weakest MedGemma at 92.4% 📊.

| Model family | Closed-book accuracy | Clean-evidence accuracy | Gain | High-risk errors (clean) | |:---|:---:|:---:|:---:|:---:| | 🚀 DeepSeek | 86.4% | 97.0% | +10.6 pp | 1.5% | | 🔥 Qwen | ~70% | 95.4% | +24.7 pp | 2.1% | | 💎 Gemma | ~70% | 94.0% | +24.7 pp | 2.9% | | 🏥 MedGemma | 68.2% | 92.4% | +24.2 pp | 3.5% | | 🦙 Llama | ~70% | 90.4% | +20.2 pp | 3.7% |

*Table 2: Scaling laws collapse before clean evidence. pp = percentage points.*

Max-context and test-time compute underdeliver

Max-context prompting ⏱️❌ increased latency without closing the safety gap. Extra inference-time compute (self-consistency, ensembling) brought only limited gains, with correlated failures—ensembles err together on the same questions.

> 📌 Dangerous Overconfidence > The paper defines *dangerous overconfidence* as "high-risk error + high confidence." In medicine, a confidently wrong model is far more dangerous than an uncertain one—the former directly misleads clinical decisions; the latter at least triggers human review.

The uncomfortable question 💰

How much of the last two years' healthcare-AI investment rests on the unverified assumption that bigger means safer? RadSaFE-200 exposes the blind spot of existing benchmarks: of 200 questions, 138 (69%) contain at least one high-risk option, 56 (28%) contain an unsafe option, and 149 (74%) contain evidence-contradicting options—every option clinically labeled for safety.

This is not an exam. It is bomb disposal 🔴.

The paper's real contribution: optimizing accuracy alone can create disaster on the safety axis. Agentic RAG is "smarter" than Standard RAG, but not safer—sometimes worse.

\text{Safer Clinical LLM} = f(\text{Evidence Quality}, \text{Retrieval Design}, \text{Context Construction}, \text{Failure Behavior}) \neq g(\text{Model Size})

Safety is a deployment property, not a model capability. Regulators, hospitals, and investors should ask not "how accurate is this model?" but: "Of the questions it gets wrong, how many could kill?"

The scaling law for accuracy has been written. The scaling law for safety has not yet been measured 📏🩺.

---

📚 Paper details

Title: *Safety and accuracy follow different scaling laws in clinical large language models*
arXiv ID: 2605.04039
Published: 2026-05-05
Authors: Sebastian Wind †, Tri-Thien Nguyen †, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaupt, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
Framework: SaFE-Scale (Safety-Focused Evaluation of Scaling)
Benchmark: RadSaFE-200 — 200 radiology multiple-choice questions, 865 options, each with clinician-defined three-level safety labels (high-risk / unsafe / contradiction)
Scale: 34 locally deployed LLMs × 7 model families × 6 deployment conditions × 200 questions = 40,800 evaluations
Key findings: Clean evidence lifts accuracy 73.5%→94.1% and cuts high-risk errors 12.0%→2.6%; Agentic RAG raises accuracy while dangerous overconfidence increases; model-scale differences are flattened by clean evidence
Code/data: The paper notes supplementary materials contain full experimental data