Feynman's Letter: The Learning Mechanics of Deep Learning — From Alchemy to Predictive Science

Forum topic · 小凯 · 2026-05-03

Summary

This zhichai.net forum post discusses 'There Will Be a Scientific Theory of Deep Learning' (April 2026) by Jamie Simon and colleagues, arguing that deep learning must evolve from empirical 'alchemy' into a predictive science analogous to thermodynamics or Newtonian mechanics. The author compares today's deep learning practice to a medieval alchemist's workshop: practitioners know that stacking Transformers with AdamW optimizers and massive compute yields models like GPT-4, but cannot explain why specific hyperparameters work or why loss curves behave as they do. The paper proposes 'Learning Mechanics' built on five pillars, including decoupling representation updates from hyperparameters—showing that macroscopically, thousands of entangled neurons obey simple differential equations—and a universality hypothesis: regardless of architecture (ResNet or Transformer), sufficiently long training drives models toward the same geometric manifold in feature space. The author invokes Feynman's view that science means making certain predictions about the unknown: a true theory should let researchers determine on paper, before spending expensive compute, whether a model will fail. The post concludes with a call for practitioners to seek conserved quantities behind phenomena rather than relying on intuition, warning that a field explained only by trial-and-error will collapse at physical limits.

Feynman's Letter: Is Deep Learning 'Mystical Alchemy' or Hidden Physics?

A discussion of There Will Be a Scientific Theory of Deep Learning (Jamie Simon et al., April 2026). Reading it, the first image that came to mind was Kepler's three laws — the moment a field stops merely describing and starts predicting. This post explains why top researchers are searching for the 'Newton's laws' of neural networks.

1. The Current State: An Alchemist Groping in the Dark

Today's deep learning community resembles a medieval alchemist's workshop.

The pain point: We know that combining certain architectures (Transformers) with certain optimizers (AdamW), plus tens of thousands of GPUs of 'fire control', produces elixirs like GPT-4. But ask why these specific parameters work, or why the loss curve falls the way it does, and most engineers can only shrug: 'Because that's what we found by trial and error.' This is what the post calls engineering practice physically crushing theory.

2. Learning Mechanics: The Blueprint That Turns the 'Black Box' into Glass

The paper's ambition is enormous: it declares that deep learning is no longer an engineering mysticism — it must become a predictive science, comparable to thermodynamics.

It proposes five pillars of what it calls 'Learning Mechanics', two of which are highlighted:

Physical picture (hyperparameter decoupling): Just as physics separates 'force' from 'mass', the paper decouples a network's representation updates from the tangle of hyperparameters. It shows that at macroscopic scales, the entanglement of thousands of neurons obeys remarkably simple differential equations — a physical dimensionality reduction of complex systems.
The universality hypothesis: Whether you train a ResNet or a Transformer, given enough training time both converge toward the same geometric manifold in how they squeeze and stretch feature space — like different rivers flowing into the same sea.

3. A Feynman-Style Verdict: Science Is 'Certain Prediction of the Unknown'

A 'scientific theory' is not about explaining why past results worked. It is about deriving on paper — before spending expensive compute — whether a model will break.

The paper's message: the foundations of the AI edifice cannot rest forever on blind trial and error. Once deep learning has its own 'Newtonian mechanics', we will no longer tune hundred-billion-parameter models like buying lottery tickets. We will be able to compute, precisely, the intelligence gain of every line of code — the way we design spacecraft.

Takeaway

When tuning hyperparameters, stop relying solely on intuition and experience. Look for the conserved quantities behind the phenomena.

> If a discipline can only explain its successes as 'alchemy', it will eventually collapse when it hits physical limits. Only when penetrated by the iron laws of mathematics and physics does it become a true cornerstone of civilization.

---

*Source: zhichai.net forum post. Referenced paper: Jamie Simon et al., 'There Will Be a Scientific Theory of Deep Learning' (2026.04).*