Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time. This paper tests an alternative: the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver, and the model's task reduces to deciding how to use it.
Key points
- Approach: A "Program-Solve" interface where the LLM generates case-specific Python code executed by a restricted local solver, rather than performing arithmetic directly.
- Benchmark: MedCalc-Bench Verified (1,100 cases, 55 calculators), after auditing the benchmark's formulas against current clinical guidelines and flagging 16 of 55 with version, use, or coefficient issues.
- Baselines: Direct model arithmetic and a hand-written 22-calculator library.
- Models: Qwen2.5-7B and Qwen2.5-32B-AWQ.
- With formulas and gold variables provided, and both paths reading the full record, handing off to the solver was not a reliable advantage on Qwen2.5-7B: 75.31% vs 72.02% (+3.29 points, 95% clustered CI [-3.49, 10.38], overlapping zero).
- On Qwen2.5-32B-AWQ, the solver was a reliable advantage: 90.53% vs 83.47% (+7.05 points, 95% clustered CI [0.47, 14.60], clearly greater than zero).
- The hand-written calculator library was fully accurate on its 440 supported cases but abstained elsewhere, scoring 40.0% overall.
Results
Conclusion
Adding a deterministic executor benefits some open-weight models more than others (notably larger ones), even under matched formula, variable, and record access. In any case, program generation cannot replace validated formulas or reliable variable extraction.
Paper: arXiv:2509.05821