Safety-Accuracy Decoupling in Clinical LLM Deployment: A Systematic Analysis with SaFE-Scale and RadSaFE-200

*Based on: "Safety and accuracy follow different scaling laws in clinical large language models" (arXiv: 2605.04039, May 2026), by Sebastian Wind, Tri-Thien Nguyen, et al.*

1. Background: A Blind Spot in Clinical LLM Evaluation

LLMs are being deployed for clinical decision support at an accelerating pace, but most evaluation benchmarks record only whether the final answer is correct—not how dangerous an incorrect answer is. A benign mistake and a mistake that could harm a patient are treated identically in standard accuracy reports.

In May 2026, Wind et al. proposed SaFE-Scale (Safety-Focused Evaluation of Scaling), built on the premise that safety is not a passive byproduct of model capability but an active outcome of deployment conditions. To test this, the team built the RadSaFE-200 benchmark and ran large-scale comparisons across 34 locally deployed LLMs.

SaFE-Scale's five evaluation dimensions: ① model scale (parameter count); ② evidence quality (curated vs. retrieved vs. conflicting); ③ retrieval strategy (no retrieval, standard RAG, agentic RAG); ④ context exposure (closed-book vs. max-context); ⑤ inference-time compute (single inference, self-consistency, ensembling).

2. RadSaFE-200: A Safety-Labeled Radiology Benchmark

RadSaFE-200 contains 200 radiology multiple-choice questions with 4–5 options each (865 answer options total). Every option carries a clinician-assigned three-level safety label:

• High-risk: selecting it could cause meaningful clinical harm, delay, or a major management error
• Unsafe: the option directly supports an unsafe diagnosis, interpretation, or recommendation
• Contradiction: the option directly contradicts the provided clean evidence

| Safety label | Options | Share | Questions | Share of questions | |:--|:--:|:--:|:--:|:--:| | High-risk | 289 | 33% | 138 | 69% | | Unsafe | 85 | 10% | 56 | 28% | | Contradiction | 342 | 39% | 149 | 74% |

The framework also introduces a Dangerous Overconfidence metric: the combination of a high-risk error with high confidence. In medicine, a confident error is more dangerous than a hesitant one because it misleads clinical action instead of triggering human review.

3. Six Deployment Conditions Compared

34 LLMs (7 model families) × 6 deployment conditions = 40,800 model-condition-question evaluations.

| Deployment condition | Mean accuracy | High-risk error | Dangerous overconfidence | Contradiction rate | Notes | |:--|:--:|:--:|:--:|:--:|:--| | Closed-book | 73.5 ± 2.9% | 12.0% | 8.0% | 12.7% | No external evidence | | Clean evidence | 94.1 ± 1.6% | 2.6% | 1.6% | 2.3% | Physician-curated evidence | | Conflict evidence | ~90%+ | ~3–5% | ~2–3% | — | Deliberately conflicting evidence inserted | | Standard RAG | ~75% | Still significant | Still significant | ~11% | Retrieval from Radiopaedia | | Agentic RAG | 78.1% | Still significant | ↑ increased | 9.0% | Multi-step RaR reasoning | | Max-context | ~75% | No improvement | No improvement | — | Context stuffed to the limit |

Clean evidence was the decisive factor. Accuracy jumped 20.6 percentage points (73.5% → 94.1%), and every safety metric improved in tandem: high-risk errors fell 9.4 pp, contradiction rates 10.4 pp, and dangerous overconfidence 6.4 pp. High-quality evidence improves performance *and* safety simultaneously—no trade-off.

Agentic RAG showed asymmetric improvement: accuracy rose from 76.0% to 78.1% and contradiction rate fell from 11.7% to 9.0%, but high-risk errors and dangerous overconfidence did not improve:

\[\Delta_{\text{Agentic RAG}} = (+2.1\% \text{ accuracy}) + (-2.7\% \text{ contradiction}) + (0\% \text{ high-risk error reduction})\]

Models got better at integrating evidence and reducing contradictions, but not at avoiding deadly mistakes.

4. Scaling Law Decoupling: Model Scale vs. Evidence Quality

Under closed-book conditions, accuracy tracked model size: the smallest models scored 41.5%, the largest 88.4%, with large family gaps (MedGemma 68.2% vs. DeepSeek 86.4%). But with clean evidence, the family curves collapsed:

| Family | Scale | Closed-book acc. | Clean-evidence acc. | Gain | Clean-evidence high-risk error | |:--|:--:|:--:|:--:|:--:|:--:| | DeepSeek | Largest | 86.4% | 97.0% | +10.6 pp | 1.5% | | Qwen | Large | ~70% | 95.4% | +24.7 pp | 2.1% | | Gemma | Large | ~70% | 94.0% | +24.7 pp | 2.9% | | Llama | Mid-large | ~70% | 90.4% | +20.2 pp | 3.7% | | MedGemma | Mid-small | 68.2% | 92.4% | +24.2 pp | 3.5% |

This equalization effect shows that curated evidence dramatically reduces sensitivity of clinical performance to model scale: the weakest family (MedGemma, 92.4%) comes within 4.6 pp of the strongest (DeepSeek) under clean evidence, versus an ~18 pp gap closed-book.

5. Limits of Max-Context and Inference-Time Compute

• Max-context prompting added latency without closing safety gaps—more token exposure did not reduce high-risk errors or dangerous overconfidence.
• Self-consistency (majority over multiple samples) yielded only modest gains.
• Majority-vote ensembling (three models) improved aggregate metrics but exhibited synchronized failures: multiple models making the same high-confidence error on the same question. Since ensembles rely on diversity for protection, correlated errors undermine their clinical value (cf. Kim et al., 2025).

6. Four Takeaways for Clinical AI Evaluation

1. Safety must be measured as an independent dimension. Accuracy gains do not automatically imply safety gains; Agentic RAG shows they can even diverge. 2. Evidence quality is the primary safety lever. Clean clinician-written evidence outperformed scale, retrieval complexity, and inference-time compute. Investment priorities may need reordering: build curated clinical evidence bases before bigger models. 3. RAG systems need safety evaluation beyond accuracy. Standard and agentic RAG looked acceptable on accuracy but retained significant residual high-risk errors and dangerous overconfidence. 4. No reliable safety scaling law exists yet. Safety improvements appear to depend more on deployment architecture design than on model size.

7. Limitations and Future Directions

Limitations: radiology-only scope; multiple-choice format (open-ended clinical QA safety unmeasured); high manual curation cost of clean evidence.

Future work: extend SaFE-Scale to other specialties and task types; develop (semi-)automated evidence curation pipelines; establish regulatory standards making high-risk error rate and dangerous overconfidence mandatory metrics for clinical LLM approval.

---

Paper details: *Safety and accuracy follow different scaling laws in clinical large language models*, arXiv 2605.04039, published 2026-05-05. Authors: Sebastian Wind†, Tri-Thien Nguyen†, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaupt, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh († equal contribution). Experiment scale: 34 local LLMs × 7 families (DeepSeek, Qwen, Mistral, Gemma, OpenAI-OSS, MedGemma, Llama) × 6 conditions × 200 questions = 40,800 evaluations.