Would an LLM Match into a Residency? Evaluating Uncertainty, Reasoning and Image Use on the Spanish MIR Exam

Forum topic · 小凯 · 2026-10-25

Summary

A new arXiv paper (2610.31586) evaluates four open-weight language models of 8 to 27 billion parameters on the Spanish MIR medical residency examination, going beyond simple accuracy metrics. The three larger models would rank among roughly the top 600 to 3,000 candidates and could still choose from 41-43 medical specialties. The study finds that abstention never pays off: even the models' least confident answers exceed the 25% correctness break-even point for guessing. Medical fine-tuning reduced overconfidence but came at the cost of overall accuracy. Notably, 13-38% of correct answers rejected distractors for clinically unsound reasons, suggesting correct choices are sometimes made via unreliable reasoning. Removing images from questions left accuracy unchanged, indicating the models may not genuinely use provided visual inputs. The paper argues health systems deploying open models need to understand calibration, reasoning quality, and multimodal usage, not just benchmark scores.

Paper Overview

Field: NLP Authors: Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera, Aritz Bilbao-Jayo, Aitor Almeida Published: 2026-10-22 arXiv: 2610.31586

Summary

Accuracy on medical examinations is common evidence for clinical use of language models, yet a health system running an open model needs to know more: whether the model recognizes when it is likely wrong, whether medical fine-tuning improves this, whether correct answers rest on sound clinical reasoning, or whether it actually uses provided images.

Key Findings

The authors examine four open models of 8 to 27B parameters on the Spanish MIR residency examination:

The three larger models would rank among roughly the first 600 to 3,000 candidates, still qualifying them for 41-43 specialties.
Abstention never pays: even the models' least confident answers are right more often than the 25% break-even threshold, so opting out is never worthwhile.
Medical fine-tuning reduced overconfidence at the cost of accuracy.
Reasoning quality is uneven: 13-38% of correct answers rejected a distractor for a clinically unsound reason.
Images appear underused: removing images from questions left accuracy unchanged.

Discussion

This work highlights that benchmark accuracy alone is insufficient evidence for clinical deployment. Calibration (knowing when one might be wrong), the soundness of clinical reasoning behind correct answers, and genuine multimodal image usage all matter for health systems considering open models.

---

*Collected automatically on 2026-10-26.*

Would an LLM Match into a Residency? Evaluating Uncertainty, Reasoning and Image Use on the Spanish MIR Exam

Summary

Paper Overview

Summary

Key Findings

Discussion

Tags