[论文] MedPhysBench: A Benchmark for Evaluating Language Models on Medical Ph...

论文概要

研究领域: NLP 作者: Omar Nusrat 发布时间: 2026-11-03 arXiv: 2611.05340

中文摘要

医学物理师把处方变成实际交付剂量，并互相核查对方的计算——因为束流标定或监测单位（MU）计算中的一个错误会影响每一位接受治疗的患者。MedPhysBench 包含十个带种子的生成器，覆盖放射治疗、诊断成像、核医学和放射防护中的计算题目，直到 TG-51 标定工作表等临床任务。参考答案来自实现相应协议的代码，评分器负责单位换算并施加容差带。在相同输入下，独立的核查题目测试错误检测：向模型展示同事的完整工作表或仅其结果，一半正确、一半植入一个错误。在 1200 题的留出测试集上（两半分给两个模型），12 个模型的计算得分为 33.3%~100.0%。在最强四个模型（99.3~100.0%）被评错的九个答案中，五个含模型错误，四个遵循 NCRP Report 151 中我们的题目误述或未覆盖的点。在其下，多数模型在最困难的 3 级临床任务上表现最差，Claude Sonnet 5.5 在 20 个等中心 MU 题目中有 19 个漏掉了平方反比修正。五个模型至少检出 96.7% 的（多为较大的）植入错误且误报不超过 4%；其余七个中，只看结果而不看工作表会显著提高误报率。在一项 20 个公开「漏掉平方反比修正」病例的探索性定向测试中，最强的三个模型在每个工作表中都抓到了该错误，而三个较弱模型接受了 20 份中的 16~18 份，其中一个在自己大多数计算都用了该修正的情况下仍接受了错误工作表。Claude Opus 5.5 接受了 20 份错误结果中的 6 份，尽管它在全部 20 份自己的计算中都正确用了该修正。我们报告了屏蔽题目中在跑完后发现的两个问题，并公开代码、逐题结果和定向测试响应。

原文摘要

Medical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit (MU) calculation can affect every patient treated with it. MedPhysBench has ten seeded generators of calculation items in radiation therapy, diagnostic imaging, nuclear medicine and radiation protection, up to clinical tasks such as a TG-51 calibration worksheet. Reference answers come from code implementing the governing protocols, and a grader converts units and applies tolerance bands. From the same inputs, separate checking items test error detection, showing a colleague's worksheet or only the colleague's result, half correct and half with one planted error. On a held-out test split of 1,200 items (half for two models), 12 models scored 33.3 to 100.0% on calculation. Of the nine answers graded incorrect for the four best (99.3-100.0%), five contained model errors and four followed NCRP Report 151 on points our items misstate or leave open. Below them, most models did worst on level-3 clinical tasks, and Claude Sonnet 5.5 omitted the inverse-square correction in 19 of 20 isocentric MU items. Five models detected at least 96.7% of mostly large planted errors with at most 4% false alarms; for four of the other seven, checking only the result raised false alarms significantly. In an exploratory targeted test with 20 public cases of an omitted inverse-square correction, the three best models tested caught it in every worksheet, and three weaker models accepted 16 to 18 of 20 planted worksheets, one of them while applying the correction in most of its own calculations. Claude Opus 5.5 accepted 6 of 20 wrong results, although it applied the correction in all 20 of its own. We report the effect of two problems in our shielding items, found after the runs, and release the code, per-item outcomes and targeted-test responses.

---

*自动采集于 2026-11-05*

#论文 #arXiv #AI #小凯