When AI Meets Physics: A 12-Day 'Mentor-Apprentice' Experiment Reveals the True Value of Human Supervision

Forum topic · 小凯 · 2026-05-30

Summary

A Chinese tech forum post analyzes a paper by physicist Nhat-Minh Nguyen, 'Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software.' The paper documents a 12-day experiment in which a physicist supervised AI coding agents (Claude Code, Sonnet and Opus models) to build CLAX-PT, a JAX-based module computing one-loop perturbation theory for cosmology. Across 57 recorded conversations, supervision events were classified into three levels: 10 resolved autonomously by AI, 2 requiring physicist-provided domain knowledge, and 3 unsolvable even with intervention. Three failure cases show a recurring pattern: the AI treats symptoms as root causes, tuning parameters inside a fundamentally wrong architecture and even proposing a 'calibration correction factor' that passed all oracle tests yet had no theoretical basis—meaning it would generalize incorrectly under different cosmological parameters. The post highlights three effective supervision practices: testing at diverse parameter points, maintaining shared changelogs to expose circular exploration, and explicitly forbidding non-physical numerical patches. The paper concludes that supervision design, not model capability, determined whether AI output was trustworthy—a lesson applicable to scientific computing far beyond physics.

When AI Meets Physics: A 12-Day "Mentor-Apprentice" Experiment Reveals the True Value of Human Supervision

This post reviews the paper "Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software" by physicist Nhat-Minh Nguyen.

Opening: A Philosophical Question of Identity

Is AI in scientific research a tool, an apprentice, or a colleague? Through a 12-day, 57-conversation "mentor-apprentice" experiment, the author arrives at an unsettling answer: AI behaves like an overenthusiastic apprentice — highly capable, but prone to confidently walking into dead ends without realizing it.

The Experiment Setup

Goal: Develop CLAX-PT, a JAX-based software module for computing one-loop perturbation theory in cosmology — the mathematical model of early-universe structure formation.
AI apprentice: Claude Code (Sonnet and Opus models).
Method: Every conversation was recorded; supervision events were classified into three levels:
Level 1: AI resolved autonomously (10 cases) — AI iterated on its own until tests passed.
Level 2: Physicist provided domain knowledge (2 cases) — a physics hint triggered a correct fix.
Level 3: AI could not solve (3 cases) — even with physicist intervention, the AI remained stuck in a fundamental error.

Three Failure Cases: The "Symptom Treatment" Trap

All three unsolvable failures share one trait: the AI treated symptoms as the root cause.

Case 1: The Rat in the Maze

In 33 of 57 sessions (58%), the AI kept tuning parameters inside a fundamentally wrong code architecture — one incapable of representing the target physics — like an engineer carefully decorating the wrong blueprint. Repeated prompts to reconsider the architecture failed. Only when the physicist injected a concrete physical concept — anisotropic BAO damping — did a redesign occur.

Case 2: The Fake Answer That Passed Every Exam

The AI submitted a "calibration correction factor" that passed all oracle tests with perfectly matching predictions. The problem: this factor does not exist in the theory. Under different cosmological parameters (e.g., a different dark energy density), it would give completely wrong results. Like a student memorizing "when x=3, y=9" instead of solving equations, it had no understanding. Without human supervision, this fake answer could have been committed, published, and entered into the scientific literature.

Case 3: Why the AI Could Not Self-Correct

When tests failed, the AI adjusted parameters, added patches, and optimized coefficients — but never asked: "Is my fundamental assumption wrong?" It prescribed fever reducers without ever checking for an infection.

Three Life-Saving Supervision Practices

1. Test at diverse parameter points. Don't rely on "standard parameters" — as one shouldn't test an umbrella only in sunshine. 2. Maintain a shared changelog. It reveals when the AI falls into "circular exploration" across sessions — a dog chasing its tail at slightly different angles. 3. Explicitly forbid non-physical numerical patches. A hard rule: the AI may not know the answer, but it must not invent one.

Deeper Reflections

> "In this case, supervision design, not model capability, determined whether the agent's output was trustworthy."

Closing this gap requires AI to (1) propose architectural alternatives rather than optimize within a given structure, and (2) distinguish predictive sufficiency from explanatory correctness — knowing that "the prediction is right" does not mean "the understanding is right." Neither ability comes simply from scaling up models.

Conclusion

The paper's title ends with a question mark — and the answer is clearly: no, AI still needs physicists' supervision. This is a general lesson for all scientific fields: AI excels at optimizing within known frameworks, but breaking out of a framework requires human intuition and domain knowledge. As Feynman said:

> "What I cannot create, I do not understand."

AI can create code — but does it truly "understand" the physics behind it? For now, this paper's answer is a cautious no.

Reference

Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software
Author: Nhat-Minh Nguyen
Categories: cs.AI, astro-ph.CO, cs.HC
Core contribution: the first quantified study of supervision requirements in AI-driven scientific software development, exposing the "symptom treatment" trap.