Towards a Deterministic Math Solver for Clinical Language Models

Forum topic · 小凯 · 2026-09-12

Summary

Large language models are unreliable at arithmetic, which is critical for clinical calculators where a single numerical error can change a medical recommendation. Researchers test an alternative to hardcoding each calculator as a validated function: a Program-Solve interface in which the model does not calculate itself but instead writes case-specific Python code that a restricted local executor runs as a deterministic solver, reducing the model's task to deciding how to use the solver. The approach is evaluated on MedCalc-Bench Verified (1,100 cases, 55 calculators) against direct model arithmetic and a hand-written 22-calculator library, using Qwen2.5-7B and Qwen2.5-32B-AWQ, after auditing benchmark formulas against current clinical guidelines and flagging 16 of 55 calculators for version, usage, or coefficient issues. With matched formulas, gold variables, and full record access, handing off to the solver is not a reliable advantage on the 7B model (75.31% vs 72.02%, +3.29 points, 95% cluster CI [-3.49, 10.38]) but is on the 32B model (90.53% vs 83.47%, +7.05 [0.47, 14.60]). The hand-written library is fully accurate on its 440 supported cases but abstains elsewhere (40.0% overall coverage). Solver delegation helps some open-weight models more, yet cannot replace validated formulas or reliable variable extraction. arXiv: https://arxiv.org/abs/2509.05821

Paper Overview

Field: ML Authors: Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange Published: 2026-09-11 arXiv: 2509.05821

Summary

Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time.

The authors test an alternative: the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver, and the model's task reduces to deciding how to use it.

Method and Evaluation

Benchmark: MedCalc-Bench Verified (1,100 cases, 55 calculators), after auditing the benchmark's formulas against current clinical guidelines and flagging 16 of 55 with version, use, or coefficient issues.
Baselines: direct model arithmetic and a hand-written 22-calculator library.
Models: Qwen2.5-7B and Qwen2.5-32B-AWQ.

Key Findings

With matched formulas, gold variables, and full record access for both paths, handing off to the solver is not a reliable advantage on the 7B model (75.31% vs 72.02%, paired +3.29 points, 95% cluster CI [-3.49, 10.38]).
On the 32B model it is a reliable advantage (90.53% vs 83.47%, +7.05 [0.47, 14.60], clearly greater than zero).
The hand-written library is fully accurate on its 440 supported cases but abstains elsewhere (40.0% overall).

Conclusion

Adding a deterministic executor helps some open-weight models more than others, even under matched formula, variable, and record access. In any case, it cannot replace validated formulas or reliable variable extraction.

--- *Auto-collected on 2026-09-12*

Towards a Deterministic Math Solver for Clinical Language Models

Summary

Paper Overview

Summary

Method and Evaluation

Key Findings

Conclusion

Tags