Paper Information
- Title: PhysiFormer: Learning to Simulate Mechanics in World Space
- Authors: Yiming Chen, Yushi Lan, Andrea Vedaldi
- Field: Computer Vision (cs.CV)
- Published: June 2026
- Project page: https://yimingc9.github.io/physiformer
- 3D meshes represent objects: vertices with 3D coordinates plus faces defining the surface. This makes geometry explicit, view-independent, and directly manipulable — simulating motion means updating vertex positions; simulating deformation means moving vertices relative to each other; collision detection becomes geometric intersection.
- World coordinates as the universal frame of reference. Unlike pixel coordinates, which depend on camera position and orientation, world coordinates are absolute and observer-independent. A blueprint of the room is far more useful for understanding motion than a photo of it.
- A diffusion transformer generates future motion. Inputs: initial vertex positions and velocities plus material type (rigid or elastic). Output: a trajectory of future vertex positions. Starting from a random noisy trajectory, the model iteratively denoises it into a physically plausible one.
- Temporal attention: a vertex's future position depends mostly on its own past positions and velocities — capturing causal, smooth time evolution.
- Spatial attention: vertices of the same object interact — capturing geometric constraints like connectivity and shape.
- Object attention: interactions between objects — capturing collisions, contacts, and gravity.
- Trajectory accuracy: significantly lower error versus ground-truth physics.
- Rigidity preservation: rigid bodies keep their shape, while baselines gradually deform.
- Momentum consistency: generated trajectories better respect conservation laws.
- Robotic manipulation: robots could infer physical properties from observation and run predictive simulation before acting — "if I push here, how will it move?" — improving success rates and safety.
- AI-assisted engineering design: PhysiFormer as a fast, approximate simulator for millisecond-level "draft" simulations during design iteration, with precise solvers reserved for final validation.
- Education: turning abstract differential equations into interactive 3D animations where students adjust gravity or mass and instantly see the result.
- Film and games: data-driven alternatives to hand-tuned physics engines like PhysX and Havok, lowering the barrier for cloth, fluid, and rigid-body simulation.
- Chen, Y., Lan, Y., & Vedaldi, A. (2026). PhysiFormer: Learning to Simulate Mechanics in World Space. *arXiv preprint*. https://yimingc9.github.io/physiformer
- Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. *NeurIPS 2020*.
- Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. *ICCV 2023*.
- Battaglia, P.W., et al. (2018). Relational inductive biases, deep learning, and graph networks. *arXiv preprint*.
- LeCun, Y. (2022). A path towards autonomous machine intelligence. *Open Review*.
Why Humans Have "Intuitive Physics"
When you throw a stone into the air, your brain instantly "knows" its trajectory — rising, decelerating, pausing, accelerating downward — without solving any equations. This ability, called intuitive physics, is one of the most fundamental capabilities of human cognition. A two-year-old can predict where a ball will roll or which way a block tower will fall, without knowing Newton's laws.
Compare this with today's most advanced AI video generation models. When asked to generate a thrown stone, models like Sora or Runway Gen-3 may produce visually convincing footage — but the stone's fall may not follow a parabola, or its bounce may violate momentum conservation. Current world models are good at "looking right," not "being right": they learn pixel-level statistics, not physical causality.
This paper proposes a fundamentally different approach: learn physics in physical space, not pixel space. Its guiding idea: abandon pixels, embrace geometry; abandon views, embrace the world.
Why Pixels Are the "Graveyard" of Physics
Nearly all video generation models work in pixel space. A red ball rolling in a white room can be fully described physically with a few dozen floats — position, velocity, mass, radius, gravity, friction, boundaries. But a 10-second 256×256 video contains roughly 59 million numbers, almost all of which are irrelevant to physics. Change the camera angle or lighting, and every pixel changes while the physics stays the same.
Pixel space thus entangles physical laws with view-dependent appearance. Models are forced to learn "how the ball looks under red light" alongside "how the ball moves." Worse, they learn view-dependent pixel dynamics: a model trained on videos from a fixed camera often fails from new viewpoints — like a student who memorized worked examples without understanding the principle.
Humans, by contrast, use a view-invariant representation: the brain infers the position, velocity, and trajectory of objects in an implicit "world space," including surfaces it cannot see.
Finally, physics is inherently geometric: gravity describes how distance affects force; rigid-body dynamics describes how shapes move under forces; elasticity describes how shapes deform. Physics doesn't care about color, texture, or lighting. The solution: put physics simulation directly in geometric space, skipping the pixel layer.
PhysiFormer: Dreaming in World Coordinates
PhysiFormer's core design decisions:
The diffusion formulation offers three advantages:
1. Probabilistic generation. Different denoising paths yield different futures, capturing the chaotic nature of physical systems — multiple plausible futures rather than a single deterministic prediction. 2. No explicit physics constraints. Instead of hand-coding conservation laws and rigid constraints, the model learns them implicitly from data — analogous to how deep networks learn features automatically. 3. Unified rigid and elastic bodies. One model handles both materials, unlike traditional pipelines that use rigid-body dynamics for one and finite element analysis for the other.
Architecture: Factorized Spatiotemporal-Object Attention
Rather than full all-to-all attention, PhysiFormer factorizes attention along three axes that mirror physics' hierarchical structure:
This factorization is far more efficient than full attention and corresponds directly to the physics hierarchy: laws of motion govern time evolution, geometric constraints govern shape, boundary conditions govern interactions.
The architecture is also permutation invariant: relabeling objects doesn't change simulation results. Object identity emerges from geometric relationships in space, not external IDs, so the model handles arbitrary numbers of objects.
Experiments: Simulation to Reality, Rigid to Elastic
Training data: 100,000+ simulated trajectories covering rigid-body motion (collisions, rolling, sliding), elastic deformation (stretching, compression, bending), and mixed-material scenes — all with physically correct ground truth from simulators.
Comparison with autoregressive baselines: Autoregressive models predict frame by frame, accumulating errors like a game of telephone — each step's small errors amplify until long-horizon predictions diverge from reality. PhysiFormer instead generates the entire trajectory at once and refines it via denoising — global optimization rather than local recursion. Results:
Generalization: The model handles geometries never seen in training (e.g., complex animal models), scenes with more objects than during training, and even real-world initial states (3D meshes reconstructed from video) — despite training purely on simulation — demonstrating cross-domain transfer from simulation to reality.
Why This Changes the Game
From "looking like" to "being." Sora-style models excel at plausible-looking video with physically broken dynamics — objects passing through walls, balls defying gravity. PhysiFormer decouples physics from rendering: output is physically correct 3D motion; visual rendering is an optional, swappable downstream step. This reintroduces the classic physics/graphics separation into AI world models.
Physical grounding for world models. Yann LeCun's world-models vision focuses largely on semantic prediction ("if I push the cup, it falls"). PhysiFormer adds precise physical prediction ("it falls at 0.3s, at 45°, and rolls 0.5m") — critical for robotic manipulation, autonomous driving, and physical design.
Representation space matters. As the classic saying goes, feature engineering sets the ceiling of what a model can achieve. Similarly: the choice of coordinate system sets the ceiling for physical learning. Pixel coordinates entangle physics, viewpoint, and appearance; world coordinates separate physics from appearance. This insight may extend beyond simulation to causal reasoning, logic, and knowledge representation.
Outlook: When AI Truly "Understands" Physics
Conclusion: Return the World to the World
Pixel space is a beautiful prison — it lets machines generate stunning visuals but locks physics away, warped and unreachable. PhysiFormer breaks that prison, returning the world to its own coordinate system, where positions are absolute, motion is objective, and deformation is real. When AI learns to dream in world coordinates, it dreams not of beautiful images, but of physical truth — a world that needs only three dimensions and a clock.
References and Further Reading
Core paper:
Related work: