Paper Information
- Title: PhysiFormer: Learning to Simulate Mechanics in World Space
- Authors: Yiming Chen, Yushi Lan, Andrea Vedaldi
- Field: Computer Vision (cs.CV)
- Released: June 2026
- Vertices: discrete points on the object surface, each with 3D coordinates (x, y, z)
- Faces: polygons (usually triangles) connecting the vertices
- Trajectory accuracy: significantly lower error than autoregressive baselines
- Rigid-shape preservation: better at keeping relative vertex positions unchanged
- Momentum consistency: trajectories better satisfy conservation laws
- Chen, Y., Lan, Y., & Vedaldi, A. (2026). PhysiFormer: Learning to Simulate Mechanics in World Space. *arXiv preprint*. https://yimingc9.github.io/physiformer
- Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. *NeurIPS 2020*.
- Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. *ICCV 2023*.
- Battaglia, P.W., et al. (2018). Relational inductive biases, deep learning, and graph networks. *arXiv preprint*.
- LeCun, Y. (2022). A path towards autonomous machine intelligence. *Open Review*.
---
Introduction: Why Do Humans Have "Intuitive Physics"?
When you throw a stone into the air, your brain instantly "knows" how it will move—rising, decelerating, pausing, then accelerating downward. You don't compute any equations or know that gravitational acceleration is 9.8 m/s². Your brain completes a full physical prediction in milliseconds.
This ability is called intuitive physics, one of the most fundamental capacities of human cognition. A two-year-old, who knows nothing of Newton's laws, can predict where a ball will roll, which way a block tower will fall, where water will flow. This prediction is not done by rational calculation, but by an implicit, fast "world simulator" in the brain.
Compare this to today's most advanced AI video generation models. When models like Sora or Runway Gen-3 generate a video of a stone thrown into the air, the output may look realistic—but on close inspection, the stone's descent may not follow a parabolic trajectory, or the bounce angle may violate momentum conservation. The model is "painting" a video that resembles the physical world without truly understanding physical laws.
This is the core dilemma of current AI world models: they are good at "looking right," but bad at "actually being right." They learn pixel-level statistical regularities rather than physical causal laws. This paper proposes a fundamentally different approach: instead of learning statistical regularities in pixel space, learn physical laws directly in physical space.
The method is called PhysiFormer, and its core innovation in one sentence: abandon pixels, embrace geometry; abandon views, embrace the world.
---
The Nature of the Problem: Why Pixels Are the "Grave" of Physics
The Tyranny of Pixel Space
Almost all video generation models work in pixel space, predicting the next frame from the previous one. The problem: pixels are the grave of physics—rich physical information is discarded and distorted during pixelation.
Consider a red ball rolling in a white room. Physically, the scene is fully described by the ball's position, velocity, mass, radius, gravity, friction coefficient, and wall boundaries—perhaps dozens of floating-point numbers. But a 10-second 256×256 video at 30 fps requires roughly 59 million numbers, the vast majority of which are irrelevant to physics. Change the camera angle or lighting, and all pixel values change—while the physical laws themselves remain unchanged.
Pixel space entangles physical laws with view-dependent appearance. The model is forced to learn "how the ball moves" and "what the ball looks like under red light" simultaneously—two physically independent things.
The Curse of View Dependence
Video world models learn pixel changes from specific camera viewpoints, not the objects' physical motion. If the camera angle changes at test time, models often fail—like a student who memorized example solutions without understanding the principles.
Humans instead have a view-invariant representation: regardless of viewpoint, the brain represents the ball's position, velocity, and trajectory. This "world space" representation is the basis of intuitive physics.
Why Physics Needs "Geometry"
Physics is fundamentally geometric. Newton's law of gravitation describes how geometric relations (distance) influence force; rigid body dynamics describes how geometric shapes move under force; elasticity describes how shapes deform. Physics cares about shape, mass, position, velocity, elasticity—all geometric quantities—not color, texture, or lighting.
PhysiFormer's solution: put physical simulation directly into geometric space, skipping the pixel intermediate layer.
---
PhysiFormer: "Dreaming" in World Coordinates
PhysiFormer's core: represent objects with 3D meshes, predict motion in world coordinates, and generate the future with a diffusion model.
3D Mesh: From Pixels to Geometry
A 3D mesh consists of:
Advantages:
1. Explicit geometry: the model directly operates on geometric quantities—moving vertices, deforming surfaces, computing collisions. 2. View invariance: the same mesh can be rendered from any angle; the learned physics is in "world space." 3. Direct physical operations: to simulate motion, update vertex positions; to simulate deformation, modify relative positions; to compute collisions, detect geometric intersections.
World Coordinates: Physics' "Universal Language"
All computation happens in world coordinates—an absolute, fixed, observer-independent reference frame—rather than camera coordinates. In pixel space, an object's position depends on camera position, angle, and focal length. In world coordinates, position is absolute and motion objective. A pixel is like a photograph; a world coordinate is like a blueprint.
Diffusion Model + Physics = PhysiFormer
PhysiFormer is a Diffusion Transformer that predicts vertex trajectories directly in world coordinates.
Input: initial vertex positions, initial vertex velocities, material type (rigid or elastic) Output: future vertex position trajectories
The trajectory prediction is formulated as a denoising diffusion process: 1. Start from a random noise trajectory 2. Iteratively denoise toward a physically plausible trajectory 3. Each denoising step infers how to remove noise based on current state, physical laws, and material properties 4. Final result: a smooth, physically consistent vertex trajectory
Advantages:
1. Probabilistic generation: different denoising paths yield different futures, capturing the chaotic nature of physical systems and generating multiple possible futures. 2. No explicit physical constraints needed: with enough training data, the model implicitly learns constraints like momentum and energy conservation. 3. Unified rigid and elastic handling: the same model handles both materials—rigid bodies keep shape, elastic bodies deform.
---
Architecture Deep Dive: Factorized Attention
PhysiFormer embodies the insight that physical motion decomposes across time, space, and objects.
Attention Factorization
Instead of fully connected attention, PhysiFormer factorizes attention into three types:
1. Temporal attention: between the same vertex across timesteps—captures temporal causality and smoothness. 2. Spatial attention: between vertices of the same object—captures geometric structure and constraints. 3. Object attention: between different objects—captures interactions like collision, contact, and gravity.
This factorization is far more computationally efficient than fully connected attention, and mirrors physics' hierarchical structure: physical laws control temporal evolution, geometric constraints control shape, boundary conditions control object interaction.
Permutation Invariance: The Key to Multi-Object Reasoning
When multiple objects exist, the ordering shouldn't affect the simulation result. PhysiFormer ensures permutation invariance through its architecture—object identity is defined implicitly by geometric relations in space, not external labels—so it handles any number of objects without per-object encodings.
---
Experimental Validation
Training Data: 100k+ Simulated Trajectories
PhysiFormer trains on over 100,000 simulated trajectories covering rigid-body motion (collision, rolling, sliding), elastic deformation (stretching, compression, bending), and mixed-material scenes. Training data comes from physics simulators, so every sample is physically correct.
Comparison with Autoregressive Baselines
Autoregressive models predict the future frame by frame, suffering from error accumulation—like the telephone game. PhysiFormer's diffusion approach generates the entire trajectory at once, then refines it via denoising—like a sculptor refining a rough form. Results:
Generalization: Unseen Worlds
1. Unseen geometries: physically plausible motion for unfamiliar shapes, indicating it learned universal physical laws rather than shape-specific patterns. 2. More objects: handles scenes with more objects than training, validating the permutation-invariant design. 3. Real-world data: despite training purely on simulation, it generates reasonable futures from real-world initial states (e.g., 3D meshes reconstructed from video), demonstrating sim-to-real transfer.
---
Why This Work Changes the Game
From "Looks Like" to "Actually Is"
Video generation models produce physically incorrect videos—objects passing through walls, balls flying upward, inelastic collisions violating energy conservation. PhysiFormer doesn't care about "looking like"—only "actually being." Physics and rendering are cleanly separated: the same physical motion can be rendered with different renderers, lighting, and materials.
A Physical Foundation for World Models
Yann LeCun's "world models" vision has focused on semantic-level prediction. PhysiFormer provides the physical foundation—precise physical prediction essential for robot manipulation (grasping, assembly), autonomous driving (collision prediction), and physical design.
The Importance of Coordinate Systems
A deep lesson: the choice of coordinate system determines the ceiling for learning physics. Pixel coordinates are a poor physical coordinate system because they mix physical laws, view dependence, and appearance. World coordinates separate physics from appearance. This may inspire AI research to rethink representation spaces in other domains too—causal reasoning, logical reasoning, knowledge representation.
---
Future Outlook: When AI Truly "Understands" Physics
1. Physical intuition in robot manipulation: robots could infer physical properties from observation and run predictive simulation before acting—"if I push this way, how will the object move?" 2. AI-assisted physical design: PhysiFormer as a fast, approximate simulator giving millisecond-level predictions, complementing slow but accurate FEA solvers in design iteration. 3. Visualized physics education: turning abstract equations into interactive 3D animations where students adjust initial conditions and see results immediately. 4. Film and games: data-driven physical simulation that learns parameters from real data, lowering the barrier compared to hand-tuned engines like PhysX or Havok.
---
Conclusion: Return the World to Itself
PhysiFormer's core contribution: return the world to the world. Video generation models imprison the world in pixels—a beautiful but cruel cage. PhysiFormer frees the world back into its own coordinate system, where physical laws are clear, explicit, and direct. This is not just a technical improvement but a philosophical return: the essence of physics is not pixels but geometry; not views but the world; not appearance but law.
---
References and Further Reading
Core paper:
Related work: