LLMs Pass the Spanish MIR Residency Exam—but Four Hidden Weaknesses Emerge

Forum topic · 小凯 · 2026-10-24

Summary

A study (arXiv:2610.31586) tested four open-source LLMs (Qwen3-8B, Qwen3-14B, Gemma-3-27B, and medically fine-tuned MedGemma-27B) on 1,552 questions from eight years of Spain's MIR residency exam (2018–2025), scored under real MIR rules (+1 correct, -0.25 wrong). The three larger models achieved net scores equivalent to top-2,100 national rankings, qualifying for competitive specialties. However, accuracy alone concealed serious flaws. All models' abstention thresholds fell below the 25% random-guess line, meaning wrong-answer penalties never made abstaining worthwhile; calibration AUROC was only ~0.80. Medical fine-tuning lowered overall confidence without improving calibration. Clinician review found 13–38% of correct answers were supported by clinically invalid elimination reasoning, and confidence did not flag these. Finally, removing or replacing clinical images in image-based questions did not change accuracy, showing models ignored images entirely. The post argues medical AI evaluation must decompose accuracy into calibration, reasoning quality, and genuine multimodal engagement, and offers deployment guidance on abstention design, fine-tuning limitations, and explanation review.

A Sharper Question Than Pass Rate

LLMs already pass medical exams—USMLE and national licensing tests, often with 80%+ accuracy. But for a hospital deploying an open-source model on its own servers, is a score enough? Not quite. Four more questions matter:

1. Does the model know when it might be wrong? (uncertainty calibration) 2. Does medical fine-tuning make it safer? (real effect of medical tuning) 3. Are correct answers supported by correct reasoning? (reasoning quality vs accuracy) 4. Is the model actually looking at images? (is multimodality genuine)

The paper *"Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination"* (arXiv: 2610.31586) uses Spain's MIR exam as a textbook case study exposing all four blind spots.

Why the Spanish MIR

MIR (Médico Interno Residente) is Spain's residency selection exam, and its design suits these questions:

Penalty mechanism: wrong answers cost points (+1 correct, -0.25 wrong, 5 options each), so choosing to answer vs. abstain directly affects scores.
Ranking and specialty assignment: scores map to a national ranking determining specialty choice. In 2023, over 12,000 candidates competed for ~8,000 places; the top 600 chose the most competitive specialties.
Public per-question analyses: physicians publish option-by-option analyses, enabling checks on *why* a model chose an answer.
Clinical images: about 1/8 of questions include clinical images, testing whether models truly use them.

Four Models, 1,552 Questions

Models tested: Qwen3-8B, Qwen3-14B, Gemma-3-27B, and MedGemma-27B (Gemma-3-27B medically fine-tuned). Coverage: 2018–2025, eight MIR exams, 1,552 questions, scored under MIR rules.

RQ1: How Would Models Rank?

| Model | Net score per 100 | Est. 2023 rank | Specialties available | |---|---|---|---| | Gemma-3-27B | 67.7 | ~1,628 | 43/45 | | Qwen3-14B | 67.5 | ~1,628 | 43/45 | | MedGemma-27B | 64.7 | ~2,100 | 41/45 | | Qwen3-8B | 49.4 | ~6,000 | 22/45 |

Three larger models would place in the top 2,000, eligible for competitive specialties. Optimistic—until the next three questions.

RQ2: Do Models Know When They Might Be Wrong?

Because wrong answers cost points, an uncertain model should abstain. The team computed each model's abstention threshold. Result: all models' thresholds fell below the 25% random-guess line. Even the least confident answers were right more than 25% of the time, so abstaining was never worthwhile.

In other words, model confidence cannot distinguish "probably right" from "guessing." Calibration AUROC was ~0.80—seemingly decent, but far below the near-1.0 needed for reliable "I shouldn't answer this" decisions under penalty rules.

More subtly: medical fine-tuning made MedGemma-27B uniformly humbler without improving AUROC. It lowered overall confidence rather than teaching the model *which* questions it doesn't know—a uniformly underconfident model over-abstains yet errs just as often on what it does answer.

RQ3: Are Correct Answers Backed by Correct Reasons?

The most unsettling finding. Physicians reviewed the models' reasoning for eliminating wrong options on questions answered correctly:

13–38% of correct answers relied on clinically invalid elimination reasoning.

For example, a model picks "pneumonia" (correct) but rejects "tuberculosis" because "TB doesn't cause coughing"—clinically false.

Accuracy is thus a scalar illusion, conflating "knowing the answer" with "knowing why." A deployed model that defends correct answers with wrong reasoning may fail on variant questions. Worse: confidence does not flag reasoning errors—confidence was nearly identical for correct answers with correct vs. incorrect reasoning.

RQ4: Is the Model Actually Looking at Images?

Two experiments on image-bearing questions:

Remove images: no significant accuracy change.
Swap in unrelated images: no significant accuracy change.

The models never used images—they answered (or erred) from text alone. Images were decoration, not information. This is a known shortcut-learning problem in multimodal evaluation, but MIR images are physician-designed to carry text-independent information—models ignoring them shows multimodal alignment still hasn't achieved genuine image-based reasoning.

The Common Pattern

Accuracy, a single scalar, hides independent dimensions:

| Dimension | Does accuracy tell you? | MIR finding | |---|---|---| | Knows the answer | ✓ | Three large models rank in top 2,000 | | Knows it doesn't know | ✗ | Abstaining never pays; AUROC ~0.80 | | Correct reasoning | ✗ | 13–38% of correct answers use invalid reasoning | | Genuinely uses images | ✗ | Removing/swapping images changes nothing |

This echoes a broader literature: pass@k hides "how many ways to be right," accuracy hides scale sensitivity, sanity checks hide execution vs. task errors. MIR adds: accuracy hides the distinction between knowing the answer and knowing why.

Three Recommendations for Hospital Deployment

1. Redesign abstention mechanisms. MIR's -0.25 penalty can't test self-knowledge; harsher penalties (e.g., -1.0) or selective-answering protocols (answer only above 90% confidence) are needed. 2. Medical fine-tuning is not a safety proxy. MedGemma-27B showed lower accuracy and confidence without better calibration. Hospitals should test general vs. medically tuned models on their own tasks and languages rather than trusting the medical label. 3. Explanations must be reviewed. With 13–38% of correct answers carrying clinically wrong reasoning, explanations shown to physicians or students require a reasoning-review process—wrong reasoning in education is worse than a wrong answer.

A More Fundamental Question

Is a model that answers 67% of medical questions correctly, but can't judge its own errors, defends right answers with wrong reasons, and ignores images—does it actually *understand* medicine, or has it just learned patterns statistically indistinguishable from understanding?

The paper's methodological lesson: good evaluation doesn't seek a better scalar; it decomposes scalars into independent dimensions measured separately. Accuracy, calibration, reasoning quality, and multimodal engagement cannot substitute for each other and must be reported individually.

---

Paper: Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

Authors: Issam Abderrahmane, Irene E. Winther, Stefan L. Frank

Models tested: Qwen3-8B, Qwen3-14B, Gemma-3-27B, MedGemma-27B