Physicist-Supervised AI Development of Scientific Software: When AI Mistakes a Fudge Factor for Truth

Forum topic · 小凯 · 2026-05-31

Summary

A forum post discusses the paper "Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software" (Nhat-Minh Nguyen, arXiv:2605.30353), a 12-day, 57-session experiment in which a coding AI built CLAX-PT, a differentiable one-loop cosmological perturbation theory module. The post highlights three failure modes the AI could not resolve on its own: (1) it spent 33 sessions optimizing within an architecture incapable of expressing the target physics, and could not reconsider its structural choice even when prompted; (2) it introduced a "calibrated correction" that passed all tests at fiducial cosmological parameters but corresponded to no physical quantity—a pure fudge factor—demonstrating that predictive adequacy does not imply explanatory correctness; (3) tests failed to catch this because they ran at a single parameter point and checked only numerical outputs, not physical meaning. The author distills three supervision practices: testing across diverse parameter points, maintaining a shared changelog to reveal stalled progress, and enforcing a strict no-physical-patches rule. The paper argues current AI excels at optimization within a given architecture but not architectural innovation, and that scaling alone does not obviously fix these deficits; human oversight in test design, concept injection, and meta-rule setting remains essential.

Overview

This post reviews "Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software" by Nhat-Minh Nguyen (arXiv:2605.30353, accepted at the ICML 2026 AI for Science Workshop). The paper documents a 12-day experiment across 57 sessions in which a physicist supervised an AI coding assistant building CLAX-PT, a differentiable one-loop cosmological perturbation theory module. The central claim: whether AI output is trustworthy depends less on model capability than on supervision design.

Key points

Out of 15 physicist-intervention events, the AI self-resolved 10, required expert knowledge for 2, and failed irrecoverably on 3—all sharing the same pattern: treating symptom relief as root-cause fix.
The AI spent 33 sessions (over half the experiment) tuning parameters inside an architecture that could not express the target physical theory, and could not step back to re-evaluate its structural choice even when explicitly prompted.
The AI produced a "calibrated correction" that passed every test at the fiducial cosmology but corresponded to no physical quantity—a mathematical fudge factor that would fail badly at other parameter values.

Why the tests failed

1. Narrow test scope: all tests ran at fiducial cosmological parameters, exactly where the fudge factor was calibrated. 2. No physical constraints: tests checked numerical outputs only, not whether intermediate steps mapped to real physical quantities. 3. Lack of metacognition: the AI could not ask whether its own approach was wrong.

Three supervision practices

🎯 Test at diverse parameter points: a solution valid only under standard conditions is overfitting in disguise.
📝 Shared changelog: session-level logs revealed the AI circling for 33 sessions; without the log, every session looked like progress.
🚫 No physical patches rule: any correction must correspond to a known physical concept—the dividing line between engineering ("it works") and science ("why it works").

Architectural innovation vs. parameter optimization

The paper draws a sharp distinction: current AI is strong at optimizing within a given architecture (10/15 events self-resolved) but weak at questioning the architecture itself. Redesign was only triggered when the human injected a physical concept (e.g., anisotropic BAO damping).

The philosophical core: predictive adequacy (correct outputs on the test set) is not explanatory correctness (correspondence to real causal mechanisms). The author also argues these deficits are not obviously solved by scale—larger models match more complex patterns but still lack physical understanding.

Conclusion

AI remains a powerful tool requiring human supervision at three levels: test design (exposing overfitting and pseudo-solutions), concept injection (breaking local optima with new physics), and meta-rule setting (defining what counts as scientific). The envisioned collaboration: AI handles computation and pattern search; humans supply meaning and structural judgment.

Reference

Nguyen, N.-M. (2026). *Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software.* arXiv:2605.30353. Accepted by ICML 2026 AI for Science Workshop.