Towards a Deterministic Math Solver for Clinical Language Models

• arXiv: 2609.10728
• Categories: cs.AI, cs.SE
• Authors: Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange, Rafi Al Attrach, Sahil Kapadia, Zakaria Laouabdia Sellami, Angelo Antonio Talio, Leo Anthony Celi
• Posted: 2026-09-13

Abstract

Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time. The authors test an alternative: the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver, and the model's task reduces to deciding how to use it.

Evaluation

The Program-Solve interface is evaluated on MedCalc-Bench Verified (1,100 cases, 55 calculators) against:

1. Direct model arithmetic 2. A hand-written 22-calculator library

Models tested: Qwen2.5-7B and Qwen2.5-32B-AWQ. The benchmark's formulas were audited against current clinical guidelines, with 16 of 55 calculators flagged for version, use, or coefficient concerns.

Results

With formulas and gold variables supplied, and both routes reading the whole note:

• 7B: handing off to the solver is *not* a reliable advantage — 75.31% vs 72.02% (paired +3.29 points, 95% calculator-cluster interval [-3.49, 10.38])
• 32B: the solver *is* an advantage — 90.53% vs 83.47% (+7.05, [0.47, 14.60], clear of zero)
• Hand-written library: exact on its 440 supported cases but abstains elsewhere (40.0% overall)

Takeaway

Adding a code-execution solver helps some open-weight models more than others, even under matched formula, variable, and note access — and it is not a substitute for verified formulas or reliable variable extraction either way.

---

*Auto-collected on 2026-09-13.*