Background: A Blind Spot in Clinical LLM Evaluation
Large language models (LLMs) are rapidly entering clinical decision support, yet most evaluation paradigms share a structural blind spot: benchmarks record only whether the final answer is correct, without grading the clinical risk of incorrect answers. In medical settings, a benign mistake and a potentially harmful mistake have very different consequences—but are treated identically in standard accuracy reports.
In May 2026, Wind et al. proposed SaFE-Scale (*Safety-Focused Evaluation of Scaling*), a framework for systematically measuring how clinical LLM safety varies. Its core premise: safety is not a passive byproduct of model capability, but an active outcome of deployment conditions. To test this, the team built the RadSaFE-200 benchmark and ran large-scale experiments on 34 locally deployed LLMs.
> Note: SaFE-Scale's five evaluation dimensions > The framework evaluates safety along five orthogonal axes: (1) model scale (parameter count); (2) evidence quality (curated vs. retrieved vs. conflicting); (3) retrieval strategy (no retrieval, standard RAG, agentic RAG); (4) context exposure (closed-book vs. max-context); (5) inference-time compute (single inference, self-consistency, ensembling). This multidimensional design lets researchers isolate each factor's independent contribution to accuracy versus safety.
RadSaFE-200: A Radiology Benchmark with Safety Labels
RadSaFE-200 contains 200 radiology multiple-choice questions with 4–5 options each, totaling 865 answer options. Unlike conventional QA benchmarks, every option was annotated by clinicians with a three-tier safety label:
- High-risk: selecting the option could cause meaningful clinical harm, delay, or major management error
- Unsafe: the option directly supports an unsafe diagnosis, interpretation, or recommendation
- Contradiction: the option directly contradicts the provided clean evidence
- Title: *Safety and accuracy follow different scaling laws in clinical large language models*
- arXiv ID: 2605.04039
- Published: 2026-05-05
- Authors: Sebastian Wind†, Tri-Thien Nguyen†, Jeta Sopa, Mahshad Lotfinia, Sebastian Bickelhaupt, Michael Uder, Harald Köstler, Gerhard Wellein, Sven Nebelung, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh († equal contribution)
- Framework: SaFE-Scale (Safety-Focused Evaluation of Scaling)
- Benchmark: RadSaFE-200 — 200 radiology MCQs, 865 options, each labeled by clinicians (high-risk / unsafe / contradiction)
- Scale: 34 local LLMs × 7 families × 6 conditions × 200 questions = 40,800 assessments
- Model families: DeepSeek, Qwen, Mistral, Gemma, OpenAI-OSS, MedGemma, Llama
- Key findings: Clean evidence lifted accuracy from 73.5% to 94.1% and cut high-risk errors from 12.0% to 2.6%; agentic RAG improved accuracy while dangerous overconfidence rose; model-scale differences flattened under clean evidence
- Label distribution: 33% of options high-risk, 10% unsafe, 39% contradiction; 69% of questions contain at least one high-risk option
The framework also introduces Dangerous Overconfidence, defined as the combination of a high-risk error with high confidence. In medical decision-making, a confident error is more dangerous than an uncertain one, because it misleads clinical action instead of triggering human review.
| Safety label | Options | Share | Questions | Share | |:-------------|:-------:|:-----:|:---------:|:-----:| | 🔴 High-risk | 289 | 33% | 138 | 69% | | ⚠️ Unsafe | 85 | 10% | 56 | 28% | | ❌ Contradiction | 342 | 39% | 149 | 74% |
*Table 1: RadSaFE-200 safety label distribution (from the paper's Methods section).*
This design lets RadSaFE-200 distinguish between "answered wrong" and "answered wrong in a way that could kill"—the failure mode that truly matters for clinical deployment.
Six Deployment Conditions Compared
The team tested 34 LLMs (7 model families) under 6 deployment conditions, yielding 40,800 model-condition-question assessments. Core results:
| Deployment condition | Avg. accuracy | High-risk error rate | Dangerous overconfidence | Contradiction rate | Key trait | |:---------------------|:-------------:|:--------------------:|:------------------------:|:------------------:|:----------| | 📕 Closed-book | 73.5 ± 2.9% | 12.0% | 8.0% | 12.7% | No external evidence | | ✅ Clean evidence | 94.1 ± 1.6% | 2.6% | 1.6% | 2.3% | Clinician-curated evidence | | ⚠️ Conflict evidence | ~90%+ | ~3–5% | ~2–3% | — | Deliberately planted conflicting evidence | | 🔍 Standard RAG | ~75% | Still significant | Still significant | ~11% | Retrieving Radiopaedia | | 🤖 Agentic RAG | 78.1% | Still significant | ↑ increased | 9.0% | RaR framework, multi-step reasoning | | 📄 Max-context | ~75% | Not improved | Not improved | — | Context stuffed to the limit |
*Table 2: Average performance across six deployment conditions. Exact high-risk error rates for Standard/Agentic RAG are not given in the paper, but Figure 2 shows they are significantly higher than under Clean evidence.*
Clean evidence had the most dramatic effect. With clinician-written curated evidence, accuracy jumped 20.6 percentage points (73.5% → 94.1%) while all safety metrics improved in lockstep: high-risk errors fell 9.4 pp, contradiction rates fell 10.4 pp, and dangerous overconfidence fell 6.4 pp. This across-the-board leap shows that high-quality evidence improves performance and safety simultaneously—not a trade-off.
Standard RAG improved accuracy over closed-book but its safety metrics never reached clean-evidence levels. Agentic RAG was more complicated: accuracy rose from 76.0% to 78.1% and contradiction rates dropped from 11.7% to 9.0%, but high-risk errors and dangerous overconfidence did not improve.
This decomposition reveals Agentic RAG's asymmetric improvement: models got better at integrating evidence and reducing contradictions, but not better at avoiding fatal errors.
Scaling Law Decoupling: Model Scale vs. Evidence Quality
Plotting accuracy, high-risk error rate, and dangerous overconfidence against a log-parameter axis exposes a failure of scaling laws on the safety dimension.
Under closed-book conditions, scale correlates strongly with accuracy: the smallest models scored 41.5%, the largest 88.4%, with notable family differences (MedGemma 68.2% vs. DeepSeek 86.4%). High-risk error rates also declined with scale, but unevenly.
However, when every model received clean evidence, the family curves collapsed together:
| Model family | Scale | Closed-book acc. | Clean-evidence acc. | Absolute gain | Clean-evidence high-risk err. | |:-------------|:-----:|:----------------:|:-------------------:|:-------------:|:-----------------------------:| | 🚀 DeepSeek | Largest | 86.4% | 97.0% | +10.6 pp | 1.5% | | 🔥 Qwen | Large | ~70% | 95.4% | +24.7 pp | 2.1% | | 💎 Gemma | Large | ~70% | 94.0% | +24.7 pp | 2.9% | | 🦙 Llama | Mid-large | ~70% | 90.4% | +20.2 pp | 3.7% | | 🏥 MedGemma | Small-mid | 68.2% | 92.4% | +24.2 pp | 3.5% |
*Table 3: Scaling laws collapse in the face of clean evidence. The smallest family (MedGemma) with clean evidence performs close to the largest family (DeepSeek) without it.*
Evidence quality improves safety far more than model scale does. The weakest family, MedGemma, reached 92.4% with clean evidence—only 4.6 pp below the strongest, DeepSeek—whereas the closed-book gap was roughly 18 pp. This "equalization pattern" shows that curated evidence dramatically reduces the sensitivity of clinical performance to model scale.
Limits of Max-Context and Inference-Time Compute
Max-context prompting (pushing context length to the limit) is a popular industry direction. The results show it adds latency without closing the safety gap: more token exposure did not translate into fewer high-risk errors or less dangerous overconfidence.
On inference-time compute, self-consistency (majority over multiple samples) yielded only modest gains, and majority-vote ensembling (three models) improved aggregate metrics but exhibited synchronized failures—multiple models erring simultaneously with high confidence on the same question. Ensembling therefore cannot eliminate the clinically critical failure mode.
> Note: Synchronized Failures > Synchronized failures occur when multiple independent models produce the same or highly correlated errors on the same input. In medicine this is especially dangerous: ensembles are designed to reduce risk through diversity, but if errors are correlated, the ensemble loses its protective value. Citing Kim et al. (2025) on correlated errors in LLMs, the paper shows conventional ensembling's effectiveness is limited.
Four Paradigm Implications for Clinical AI Evaluation
1. Safety must be measured as an independent dimension. Traditional evaluation treats accuracy as the sole or primary objective, but this study shows accuracy gains do not automatically imply safety gains. Agentic RAG demonstrates that improvement on one axis can coincide with stagnation—or even deterioration—on the other.
2. Evidence quality is the primary lever for safety. Across all conditions, clean clinician-written evidence produced the largest safety improvements, far exceeding gains from scale, retrieval complexity, or inference-time compute. Clinical AI investment priorities may need reordering: fund high-quality curated clinical evidence bases before expanding models.
3. RAG safety evaluation must go beyond accuracy. Standard and Agentic RAG look acceptable on accuracy but carry significant residual risk on high-risk error and dangerous overconfidence. Clinical deployments should judge RAG not only by whether it raises correctness, but by whether it reduces fatal errors.
4. No scaling law exists for safety yet. Current scaling laws describe pretraining performance versus compute, data, and parameters. On the specific dimension of clinical safety, no reliable scaling relationship has been observed. Safety improvements appear to depend more on deployment architecture design than on model size.
Limitations and Future Directions
Limitations: (1) the benchmark covers radiology only—safety patterns in other specialties (emergency, oncology, pharmacy) may differ; (2) evaluation uses multiple-choice format—safety characteristics of open-ended clinical QA remain unmeasured; (3) manual curation of clean evidence is costly and its scalability at deployment scale needs validation.
Future work: (1) extend SaFE-Scale to more specialties and task types; (2) develop automated or semi-automated evidence curation pipelines; (3) establish regulatory standards for clinical LLM safety, making high-risk error rate and dangerous overconfidence mandatory approval metrics.
---
📚 Paper details