Towards a Deterministic Math Solver for Clinical Language Models

Forum topic · 小凯 · 2026-09-13

Summary

Large language models are unreliable at arithmetic, which is problematic for clinical calculators where a single numerical error can change a medical recommendation. This paper (arXiv:2609.10728) tests a Program-Solve interface: instead of computing directly, the model writes case-specific Python code executed by a restricted local solver, reducing the model's task to deciding how to use the tool. The approach is evaluated on MedCalc-Bench Verified (1,100 cases, 55 calculators) against direct model arithmetic and a hand-written 22-calculator library, using Qwen2.5-7B and Qwen2.5-32B-AWQ, after auditing benchmark formulas against current clinical guidelines and flagging 16 of 55 for version, use, or coefficient concerns. With formulas and gold variables supplied, solver hand-off is not a reliable advantage at 7B (75.31% vs 72.02%, +3.29, CI [-3.49, 10.38]) but is at 32B (90.53% vs 83.47%, +7.05, CI [0.47, 14.60]). The hand-written library is exact on its 440 supported cases but abstains elsewhere (40.0% overall). Adding an executor helps some open-weight models more than others and does not substitute for verified formulas or reliable variable extraction.

Overview

Field: cs.AI, cs.SE
Authors: Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange, Rafi Al Attrach, Sahil Kapadia, Zakaria Laouabdia Sellami, Angelo Antonio Talio, Leo Anthony Celi
arXiv: 2609.10728

Abstract

Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time. The authors test an alternative: the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver, and the model's task reduces to deciding how to use it.

They evaluate this Program-Solve interface on MedCalc-Bench Verified (1,100 cases, 55 calculators) against direct model arithmetic and a hand-written 22-calculator library, using Qwen2.5-7B and Qwen2.5-32B-AWQ, after auditing the benchmark's formulas against current clinical guidelines and flagging 16 of 55 with version, use, or coefficient concerns.

Key Findings

With formulas and gold variables supplied and both routes reading the whole note, handing off to the solver is not a reliable advantage at 7B (75.31% vs 72.02%, paired +3.29 points, 95% calculator-cluster interval of [-3.49, 10.38]).
At 32B it is an advantage (90.53% vs 83.47%, +7.05 [0.47, 14.60], clear of zero).
The hand-written library is exact on its 440 supported cases but abstains elsewhere (40.0% overall).
Adding an executor helps some open-weight models more than others even under matched formula, variable, and note access, and is not a substitute for verified formulas or reliable variable extraction either way.

---

*Auto-collected on 2026-09-13.*