MRI-Eval: A Tiered Benchmark for LLM Performance on MRI Physics and GE Scanner Operations

Forum topic · 小凯 · 2026-05-08

Summary

MRI-Eval is a tiered benchmark for evaluating large language models on MRI physics and GE scanner operations knowledge. It contains 1,365 scored multiple-choice items across nine categories and three difficulty tiers, drawn from textbooks, GE scanner manuals, programming course materials, and expert-generated questions. Five model families were tested: GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, and Llama 3.3 70B. Under standard MCQ conditions, overall accuracy ranged from 93.2% to 97.1%, with GE scanner operations the lowest category for every model (88.2% to 94.6%). However, when options were removed (stem-only free-text condition), frontier-model accuracy dropped to 58.4% to 61.1% and Llama 3.3 70B to 37.1%; GE scanner operations fell to just 13.8% to 29.8%. The findings show that high MCQ performance can mask weak free-text recall, especially for vendor-specific operational knowledge, and the authors recommend caution when using raw LLM outputs for GE-specific protocol guidance. (arXiv: 2605.05175)

Paper Overview

Field: Medical Imaging Author: Perry E. Radau Published: 2026-05-06 arXiv: 2605.05175

Abstract

Background: Existing MRI LLM benchmarks rely mainly on review-book multiple-choice questions, where top proprietary models already score highly, limiting discrimination. No systematic benchmark has evaluated vendor-specific scanner operational knowledge central to research MRI practice.

Purpose: We developed MRI-Eval, a tiered benchmark for relative model comparison on MRI physics and GE scanner operations knowledge using primary multiple-choice questions (MCQ), with stem-only and primed diagnostic conditions as complementary analyses.

Methods: MRI-Eval includes 1365 scored items across nine categories and three difficulty tiers from textbooks, GE scanner manuals, programming course materials, and expert-generated questions. Five model families were evaluated (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, Llama 3.3 70B). MCQ was primary; stem-only removed options and used an independent LLM judge; primed stem-only tested responses to incorrect user claims.

Results: Overall MCQ accuracy was 93.2% to 97.1%. GE scanner operations was the lowest category for every model (88.2% to 94.6%). In stem-only, frontier-model accuracy fell to 58.4% to 61.1%, and Llama 3.3 70B fell to 37.1%; GE scanner operations stem-only accuracy was 13.8% to 29.8%.

Conclusion: High MCQ performance can mask weak free-text recall, especially for vendor-specific operational knowledge. MRI-Eval is most informative as a relative comparison benchmark rather than an absolute competency measure and supports caution in using raw LLM outputs for GE-specific protocol guidance.

Key Findings

MCQ accuracy is nearly saturated for top proprietary models (93.2%–97.1%), limiting benchmark discrimination.
Free-text (stem-only) recall is substantially weaker: 58.4%–61.1% for frontier models, 37.1% for Llama 3.3 70B.
Vendor-specific GE scanner operational knowledge is the weakest area across all conditions (13.8%–29.8% stem-only).
MRI-Eval is best used for relative model comparison rather than as an absolute measure of clinical competency.