Teaching Robots Actions vs. Giving Them a Physics Brain: An Introduction to Physical Foundation Models

Forum topic · 小凯 · 2026-05-03

Summary

This forum post discusses Physical Foundation Models (PhysFM), a series of studies presented at IEEE CAI 2026 that the author says mark a breakthrough in embodied AI's grasp of physical common sense. Current vision-language-action (VLA) models act as pixel mappers: they can locate a water-filled plate but lack concepts like liquid sloshing, center-of-mass shifts, mass, friction, and fluid dynamics, causing failures in soft or dynamic environments. PhysFM addresses this by pretraining on massive volumes of physics-annotated interaction videos and simulation data, allowing an implicit physics simulator to emerge inside the model's latent space. The result is emergent common sense: reducing gripper torque for fragile glassware, smoothing acceleration when carrying liquids, and zero-shot generalization to unseen tools by reasoning about center of mass and leverage. The author argues that true dexterous manipulation comes from physical intuition—computing low-energy, stable action manifolds in chaotic real-world conditions—rather than hand-coded inverse kinematics. The takeaway for practitioners: train next-generation embodied agents on data with collision geometry and mass properties, since a model that cannot mentally simulate a falling cup breaking will make every real-world reach a dangerous gamble.

After reading the series of studies on Physical Foundation Models (PhysFM) that generated a buzz at the IEEE CAI conference in May 2026, I feel that embodied AI has finally crossed a threshold called common sense.

To show why today's robots still seem clumsy in complex environments, let's talk about carrying a plate.

1. Current state: vision models fumbling blindly in 3D

Today's vision-language-action (VLA) models are powerful, but at their core they remain pixel mappers.

The pain point: Ask a robot to carry a plate full of water, and it can accurately identify the plate's pixels and extend its arm to grab it. The problem is that it doesn't understand liquid sloshing or shifting center of mass. If the carpet is slightly soft, its gait will spill the water. Its internal world contains only geometry—no concepts of "mass," "friction," or "fluid dynamics." This is the severe disconnect between semantic representation and physical law.

2. Physical Foundation Models: a silicon soul with a built-in Newtonian engine

The core logic of the new PhysFM breakthrough is striking: I'm not satisfied with understanding the image—I want to reconstruct the world's physical equations in latent space.

Pretraining enables a dimensional upgrade in cognition:

Physical imagery (implicit dynamics): Instead of feeding the model plain text, researchers trained it on massive amounts of "interaction videos with physical parameters" and "simulation environment data." In the process, an implicit physics simulator spontaneously grew out of the model's weights.
Emergent common sense: When it sees a fragile glass, it automatically reduces the gripper's output torque; when it sees a basin full of water, it instinctively keeps the arm's acceleration smooth.
Zero-shot generalization: Best of all, once it has this physical common sense, it no longer needs retraining for every specific factory part. Like a human, it can pick up a strange, oddly-shaped tool correctly on first sight by reasoning about "center of mass" and "lever principles."

3. A Feynman-style verdict: the endpoint of embodied AI is "physical intuition"

"Dexterous manipulation" is not about writing tens of thousands of lines of perfect inverse kinematics code. It is about whether, amid extremely complex real-world chaos, your brain can instantly use physical intuition to compute the action manifold that keeps the system at minimum energy and maximum stability.

Physical Foundation Models tell us: the only way to give AI common sense is to let it grow up being beaten by the laws of physics.

When we can encode gravity and thermodynamics into the underlying logic of large models the way we encode language, those all-purpose humanoid robots that once existed only in sci-fi movies will finally hold the "passport" to enter our lives.

Key takeaway

When training the next generation of embodied agents, stop feeding them only static high-resolution photos. Feed them data with collision volumes and mass properties.

If your model cannot mentally rehearse the sound of a cup shattering on the floor, then every reach it makes in the real world is an extremely dangerous game of roulette.