MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations

• Field: NLP
• Lead author: Omar Nusrat
• Published: 2026-11-03
• arXiv: 2611.05340

Abstract (original)

Medical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit (MU) calculation can affect every patient treated with it. MedPhysBench has ten seeded generators of calculation items in radiation therapy, diagnostic imaging, nuclear medicine and radiation protection, up to clinical tasks such as a TG-51 calibration worksheet. Reference answers come from code implementing the governing protocols, and a grader converts units and applies tolerance bands. From the same inputs, separate checking items test error detection, showing a colleague's worksheet or only the colleague's result, half correct and half with one planted error. On a held-out test split of 1,200 items (half for two models), 12 models scored 33.3 to 100.0% on calculation. Of the nine answers graded incorrect for the four best (99.3-100.0%), five contained model errors and four followed NCRP Report 151 on points our items misstate or leave open. Below them, most models did worst on level-3 clinical tasks, and Claude Sonnet 5.5 omitted the inverse-square correction in 19 of 20 isocentric MU items. Five models detected at least 96.7% of mostly large planted errors with at most 4% false alarms; for four of the other seven, checking only the result raised false alarms significantly. In an exploratory targeted test with 20 public cases of an omitted inverse-square correction, the three best models tested caught it in every worksheet, and three weaker models accepted 16 to 18 of 20 planted worksheets, one of them while applying the correction in most of its own calculations. Claude Opus 5.5 accepted 6 of 20 wrong results, although it applied the correction in all 20 of its own. We report the effect of two problems in our shielding items, found after the runs, and release the code, per-item outcomes and targeted-test responses.

Key Points

• Benchmark design: Ten seeded generators of calculation items spanning radiation therapy, diagnostic imaging, nuclear medicine, and radiation protection, including realistic clinical tasks such as TG-51 calibration worksheets.
• Grading: Reference answers are produced by code implementing the governing protocols; the grader converts units and applies tolerance bands.
• Error-checking tasks: Parallel checking items present a colleague's worksheet (or only its final results), half correct and half containing one planted error.
• Results: On a held-out test split of 1,200 items, 12 models scored 33.3%–100.0% on calculations.
• Top-model errors: Of nine answers graded incorrect for the four strongest models (99.3–100.0%), five were genuine model errors and four stemmed from points in NCRP Report 151 that the benchmark items misstate or leave open.
• Weakness pattern: Lower-scoring models performed worst on level-3 clinical tasks; Claude Sonnet 5.5 omitted the inverse-square correction in 19 of 20 isocentric MU items.
• Error detection: Five models caught ≥96.7% of (mostly large) planted errors with ≤4% false alarms; among the other seven, checking only results (not worksheets) significantly raised false-alarm rates in four models.
• Targeted test: In 20 public cases of an omitted inverse-square correction, the three strongest models caught the error in every worksheet, while three weaker models accepted 16–18 of 20 flawed worksheets. Claude Opus 5.5 accepted 6 of 20 wrong results despite applying the correction correctly in all 20 of its own calculations.
• Transparency: The authors report two problems found post-run in shielding items and release code, per-item outcomes, and targeted-test responses.