Fatal Hallucinations in Clinical LLMs: Why Bigger Models Are Not Necessarily Safer

Safety and accuracy in clinical LLMs follow different scaling laws. Making models larger, stuffing contexts longer, and building more complex retrieval pipelines may raise accuracy — but safety can stay flat, or even silently worsen.

In May 2026, a German-international research team published a short but devastating paper. They evaluated 34 locally deployed clinical LLMs across 7 mainstream model families under 6 deployment conditions, answering 200 radiology safety questions. The conclusion is not a gentle reminder but a loud slap: in clinical AI, safety is not a passive byproduct of model capability — it is an active result of deployment conditions.

> 📌 Note: What is SaFE-Scale? > SaFE-Scale is the evaluation framework proposed in the paper, short for *Safety-Focused Evaluation of Scaling*. Rather than only measuring accuracy, it grades each wrong answer by clinical risk — some errors are harmless, others can be lethal. It spans five dimensions: model scale, evidence quality, retrieval strategy, context length, and inference-time compute.

The Scoreboard: Same Models, Six Deployment Conditions

| Deployment Condition | Avg. Accuracy | High-Risk Error Rate | Dangerous Overconfidence | Contradictory Evidence Rate | |:---------|:----------:|:-----------:|:-----------:|:---------:| | 📕 Closed-book (zero-shot) | 73.5% | 12.0% | 8.0% | 12.7% | | ✅ Clean evidence (clinician-curated) | 94.1% | 2.6% | 1.6% | 2.3% | | ⚠️ Conflict evidence | ~90%+ | ~3–5% | ~2–3% | — | | 🔍 Standard RAG | ~75% | Still high | Still high | ~11% | | 🤖 Agentic RAG | 78.1% | Still high | ↑ Higher | 9.0% | | 📄 Max-context | ~75% | Not improved | Not improved | — |

*Table 1: Average performance of 34 LLMs on RadSaFE-200. Data from the paper's Table 1 and Figure 2.*

The most striking part is not the numbers but the direction.

With clean evidence (curated, clinician-written evidence), accuracy jumps from 73.5% to 94.1% 📈, and high-risk error rates plummet from 12.0% to 2.6% 📉. This is an across-the-board leap — all safety metrics improve together, not a trade-off where one metric gains at another's expense.

But with Standard RAG, accuracy does improve over closed-book — yet high-risk errors and dangerous overconfidence remain stubbornly high. The model gets more answers right, but the ones it gets wrong can still be fatal.

> 📌 Note: RAG vs. Agentic RAG > RAG (*Retrieval-Augmented Generation*) retrieves external documents into the context before answering. Agentic RAG goes further, letting the model reason in multiple steps, actively search, and integrate evidence like an agent. In theory, smarter — but the paper shows: smarter ≠ safer.

Agentic RAG: Smarter but Not Safer

The most ironic result is Agentic RAG 🤖. The researchers used a previously published radiology Retrieval-and-Reasoning (RaR) framework. The result? Accuracy rose from ~76% (Standard RAG) to 78.1% — progress. The contradictory-evidence rate fell from 11.7% to 9.0% — also progress. But high-risk errors and dangerous overconfidence did not budge — dangerous overconfidence actually rose ⬆️.

What does this mean? The model got better at *using* evidence, but it did not get better at *avoiding fatal errors*. It shows *higher* confidence on wrong answers — and confident errors are the most dangerous combination in clinical settings 💀.

In the paper's own words:

> The centroid shift from Standard RAG to Agentic RAG is the clearest example of decoupling: accuracy increases while dangerous overconfidence also increases — the opposite direction from what single-axis improvement would predict.

This is the core evidence for Safety–Accuracy Decoupling. Not an abstract theory, but an empirical fact across 34 models, 200 questions, and 6 conditions.

Scaling Laws Collapse in the Face of Clean Evidence

The researchers plotted accuracy, high-risk error rates, and dangerous overconfidence against model parameter counts. Under closed-book conditions, larger models do perform better — accuracy rises from 41.5% (smallest) to 88.4% (largest), with significant family-level differences. But give all models clean evidence, and every family's curve collapses onto the same ceiling: DeepSeek 97.0%, Qwen 95.4%, Mistral 95.2%, even the weakest MedGemma reaching 92.4% 📊.

Evidence quality erased the model-scale gap. One clean, human-curated evidence passage is more effective than scaling a model 10×.

| Model Family | Closed-book Accuracy | Clean-evidence Accuracy | Gain | Clean-evidence High-Risk Errors | |:---------|:-----------------:|:--------------------:|:-------:|:------------------------:| | 🚀 DeepSeek | 86.4% | 97.0% | +10.6 pp | 1.5% | | 🔥 Qwen | ~70% | 95.4% | +24.7 pp | 2.1% | | 💎 Gemma | ~70% | 94.0% | +24.7 pp | 2.9% | | 🏥 MedGemma | 68.2% | 92.4% | +24.2 pp | 3.5% | | 🦙 Llama | ~70% | 90.4% | +20.2 pp | 3.7% |

*Table 2: Scaling laws collapsing before clean evidence. pp = percentage points.*

And max-context prompting (cramming the context window)? More tokens, longer prompts, more compute — a hot industry direction. Result: latency increased, the safety gap did not close ⏱️❌. Extra inference-time compute (self-consistency, ensembling) brought only limited gains, and ensembles exhibited correlated failures — they failed on the same questions together.

> 📌 Note: Dangerous Overconfidence > The paper defines *dangerous overconfidence* as the combination of "high-risk error + high confidence." In medicine, a model that is wrong with high confidence is far more dangerous than one that is wrong but uncertain — the former directly misleads clinical decisions; the latter at least triggers human review.

The Uncomfortable Question

Over the past two years, how many of the billions of dollars invested in medical AI rested on the untested assumption that "bigger models are safer"? Hospitals, pharma companies, and AI firms have raced to buy bigger GPUs, deploy longer contexts, and build more complex RAG pipelines — yet almost no one has measured "safety" as a separate dimension.

RadSaFE-200 exposes the blind spot of existing benchmarks. Traditional medical QA benchmarks only record right vs. wrong — they don't tell you whether an error is benign or potentially lethal. In RadSaFE-200, 138 of 200 questions (69%) contain at least one high-risk option, 56 (28%) contain unsafe options, and 149 (74%) contain options contradicting the evidence. Every option is labeled by clinicians with a three-level safety tag (high-risk / unsafe / contradiction).

This isn't an exam. It's bomb disposal 🔴.

The paper's real contribution is not "clean evidence is good" — that's nearly tautological. It is the demonstration that if you optimize only on accuracy, you may be creating disaster on another axis. Agentic RAG is "smarter" than Standard RAG, but on safety it is no better — and in some ways worse.

Safety is a deployment property, not a model capability.

Regulators approving medical AI products, hospitals procuring clinical decision-support systems, and investors pouring money into medical AI should ask not "how accurate is this model?" but: "Of the questions this model gets wrong, how many could be lethal?"

The scaling law for accuracy has been written. The scaling law for safety hasn't even started being measured 📏🩺.

---

📚 Paper Details

• Title: *Safety and accuracy follow different scaling laws in clinical large language models*
• arXiv ID: 2605.04039
• Published: 2026-05-05
• Authors: Sebastian Wind †, Tri-Thien Nguyen †, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaupt, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
• Core framework: SaFE-Scale (Safety-Focused Evaluation of Scaling)
• Benchmark: RadSaFE-200 (Radiology Safety-Focused Evaluation) — 200 radiology multiple-choice questions, 865 options, each with a clinician-defined three-level safety label (high-risk / unsafe / contradiction)
• Experimental scale: 34 locally deployed LLMs × 7 model families × 6 deployment conditions × 200 questions = 40,800 evaluations
• Key findings: Clean evidence raised accuracy from 73.5% to 94.1% and cut high-risk errors from 12.0% to 2.6%; Agentic RAG improved accuracy while dangerous overconfidence rose; model-scale differences vanished under clean evidence
• Code/Data: The paper notes that supplementary materials contain the full experimental data