Learning Mechanics: A Physics-Based Scientific Theory of Deep Learning

Forum topic · 小凯 · 2026-05-03

Summary

This Chinese forum post discusses the paper "There Will Be a Scientific Theory of Deep Learning" by Jamie Simon et al., arguing that deep learning must evolve from empirical "alchemy" into a predictive science akin to thermodynamics or classical mechanics. The author compares today's deep learning practice to medieval alchemy: practitioners know that combining Transformers, AdamW, and massive compute yields models like GPT-4, but cannot explain why specific hyperparameters work or why loss curves behave as they do. The paper proposes a framework called "Learning Mechanics" built on pillars including decoupling representation updates from hyperparameters — showing that at macro scales, thousands of entangled neurons obey simple differential equations — and a universality hypothesis: diverse architectures like ResNet and Transformer converge toward the same geometric manifold of feature transformations given sufficient training. The post frames scientific theory as the ability to predict model failure on paper before spending expensive compute, rather than merely explaining past successes. It concludes that AI foundations cannot rest forever on blind trial and error; practitioners should seek conserved quantities behind phenomena, and only when a field is penetrated by mathematical and physical law does it become a true foundation of civilization.

Learning Mechanics: Is Deep Learning "Alchemy" or Hidden Physics?

A commentary on Jamie Simon et al.'s paper There Will Be a Scientific Theory of Deep Learning (2026.04). Reading it, the immediate mental image was of Kepler's three laws — why should top researchers search for "Newton's laws" for neural networks? Let's talk about "alchemy."

1. The Current State: An Alchemist Groping in the Dark

Today's deep learning field resembles a medieval alchemist's workshop:

The pain point: We know that putting certain architectures (Transformers) together with certain optimizers (AdamW), plus the heat of tens of thousands of GPUs, can refine an elixir like GPT-4. But if you ask: why these parameters? Why does loss descend the way it does? Most engineers can only shrug: "Because that's what we found by trial." This is engineering practice physically crushing theory.

2. Learning Mechanics: The Blueprint That Turns the "Black Box" into Glass

The paper's ambition is enormous: it declares that deep learning must cease being engineering mysticism and become a predictive science, akin to thermodynamics.

It proposes five pillars known as "Learning Mechanics", including:

Physical picture (hyperparameter decoupling): Just as physics separates "force" and "mass," the paper attempts to decouple a network's "representation updates" from messy hyperparameters. It shows that at macro scales, the entanglement of thousands of neurons obeys remarkably simple differential equations — a physical dimensional reduction of complex systems.
Universality hypothesis: Whether you use ResNet or Transformer, given enough training, their compression and stretching of feature space converge toward the same "geometric manifold" — like different rivers flowing into the same sea.

3. A Feynman-Style Judgment: Science Is "Certain Prediction of the Unknown"

A "scientific theory" is not about explaining why the past worked.

It is about deriving on paper, with formulas, whether a model "will collapse" — before you spend expensive compute.

The paper's message: the foundations of the AI edifice cannot rest forever on blind trial and error.

When deep learning has its own "Newtonian mechanics," we will no longer tune hundred-billion-parameter models like buying lottery tickets. We could, like designing a space shuttle, precisely compute the intelligence gained from every line of code.

Takeaway

When tuning hyperparameters, stop relying only on "intuition" and "experience."

Look for the "conserved quantities" behind phenomena.

If a discipline can only explain its success through "alchemy," it will eventually collapse before physical limits; only when penetrated by the iron laws of mathematics and physics does it truly become a cornerstone of human civilization.

*Source hashtags: DeepLearningTheory, LearningMechanics, MachineLearning, Physics, FeynmanLearning.*