Rules to Tools: Executable Checks for LLM Agents in Scientific Computing (arXiv 2610.00313)

Forum topic · 小凯 · 2026-10-05

Summary

Rules to Tools (R2T) is a framework that converts public scientific requirements—equations, boundary conditions, and output specifications—into ready-made executable checks that scientific coding agents can call directly. The paper compares matched SciCode repair groups that share written requirement descriptions, starting programs, model, and budget; one group receives only textual checks, while the other receives callable tool implementations. Across two task-ID cohorts, complete repair succeeds in 26/30 cases with text versus 29/30 with tools; three task IDs favor tools, one favors text, and eleven tie. An eight-ID cohort scores 13/16 versus 15/16, with a task-cluster bootstrap 95% interval of [-12.5, 43.75] percentage points. A larger shared-definition cohort ties at 13/24 per group. On five development-exposed tasks with alternate starting programs, tools win 7/10 versus 3/10. A source-through Python branch also reaches 15/16, matching the dedicated command totals. In matched PDE comparisons, detailed text scores 23/24 versus 24/24 for checks, with the check group reporting 31.2% lower model output values. Agent-side cost savings vary by cohort, while public CPU usage rises in both. The results measure task-dependent repair outcomes and agent-side costs.

Paper Overview

Field: Machine Learning Authors: Jingjie Ning, Guojiang Zhao, Chen Xu, Shanshan Zhong, Xiaochuan Li, Ji Zeng, Guolin Ke Published: 2026-10-05 arXiv: 2610.00313

Summary

Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements, letting agents verify their code by calling tools instead of re-reading textual specifications.

Key Results

Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group additionally receives a callable implementation.
Across two task-ID cohorts, complete repair is 26/30 with text vs. 29/30 with prepared checks; three task IDs favor tools, one favors text, and eleven tie.
The eight-ID cohort scores 13/16 vs. 15/16, with a task-cluster bootstrap 95% interval of [-12.5, 43.75] percentage points for the difference.
The larger shared-definition SciCode cohort ties at 13/24 per group.
Five development-exposed tasks with alternate starting programs: 3/10 (text) vs. 7/10 (tools).
Tool-favored tasks are 17, 77, and 11; initial checks flagged task 17, while tasks 77 and 11 reported no violations. Task 37 favors text and had no initial violations.
A direct-through-Python source branch also reaches 15/16, consistent with the dedicated command totals.
Matched PDE comparison: detailed written requirements 23/24 vs. checks 24/24; the check group's model outputs report values 31.2% lower.
Agent-side savings vary by cohort; public CPU usage increases in both cohorts.

These results measure task-dependent repair outcomes and agent-side costs rather than universal superiority of one format.

Original Abstract (excerpt)

> Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and budgets; the tool group receives a callable implementation.

*Auto-collected on 2026-10-05.*

Rules to Tools: Executable Checks for LLM Agents in Scientific Computing (arXiv 2610.00313)

Summary

Paper Overview

Summary

Key Results

Original Abstract (excerpt)

Tags