Imagine you are watching a movie. The protagonist throws a rubber ball at the ground — it bounces up in a graceful arc, perfectly matching your intuition of what a "ball" should do. But as the camera zooms in, something strange happens: the ball does not deform at the moment of impact, no energy is transferred or released, and there is no temporal delay at all. It is like a texture being "moved" from the ground back into the air by a program.
This is not a low-budget VFX slip-up. It is a fundamental error being made by today's most advanced video-generation AI: it knows the ball "should" bounce, but it does not understand why the ball bounces.
---
1. The Illusion of Visual Realism
Over the past two years, video generation models have exploded. Sora, Runway, Pika, Kling — their outputs are increasingly hard to distinguish from real footage. Sometimes even professional editors have to squint to tell truth from forgery.
But earlier this year, a benchmark called Physics-IQ poured cold water on the party.
The research team designed 198 real-world physical scenarios — balls rolling downhill, water pouring into cups, magnets attracting nails, candles heating balloons — and used them to test eight mainstream video generation models. The result revealed a troubling rift: visual realism and physical understanding are almost two entirely different things.
Even the best model scored only 24.1% on Physics-IQ (normalized to a 100% scale). In other words, these AIs are barely better than random guessing at "predicting what happens next." More ironically, Sora scored a commanding 55.6% on visual realism — whether it could fool a multimodal LLM — but was as hopeless as everyone else at physics.
This exposes a deeper issue: today's video diffusion models are essentially extremely sophisticated pixel predictors. They do not simulate the world; they imitate the statistical history of pixels. Given a million videos of balls hitting the ground, the AI learns "balls usually bounce after landing" — but it never learns elastic deformation, energy conservation, or momentum transfer. So when a scenario drifts slightly off the training distribution, the cracks show: balls hover in mid-air, collisions pass through like ghosts, and soft clay lands stiff as a steel plate.
---
2. Giving AI a Physics Lesson
The question: can we teach AI genuine physical intuition, instead of just feeding it more videos?
A new arXiv paper, PhyCo, offers an elegant answer. Its approach resembles how humans learn physics — we do not understand friction by watching a hundred thousand real-world videos; we understand it by running controlled experiments.
Step 1: Build a Virtual Laboratory
The team used the PyBullet physics engine and the Blender renderer to build a large-scale virtual lab with six classic scenes: bricks sliding on planes, balls bouncing off walls, soft balls in free fall, objects striking deformable surfaces, billiard collisions, and more. Each scene systematically varies four core physical parameters:
- Friction: from ice to sandpaper
- Restitution: from putty to superballs
- Deformation: from steel to jelly
- Force: from a nudge to a smash
- Narayanan et al. (2026). *PhyCo: Learning Controllable Physical Priors for Generative Motion.* arXiv:2604.28169
- Motamed et al. (2025). *Do generative video models learn physical principles from watching videos?* Physics-IQ benchmark.
- Zhang et al. (2023). *Adding Conditional Control to Text-to-Image Diffusion Models* (ControlNet).
- NVIDIA (2025). *Cosmos World Foundation Model.*
In total, 100,000 videos were generated. The key design: these videos are not merely "realistic-looking" but physically exact and solvable. Every video carries pixel-aligned physical property maps — like annotating every pixel with "the friction coefficient here is 0.3." This lets the model decouple visual appearance from underlying dynamics: the same ball can be red or blue, but its sliding distance on the same surface should depend only on the friction coefficient.
Step 2: Adjusting Physics Like a Palette
The most elegant part: instead of training a video model from scratch — far too expensive — the team took NVIDIA's pretrained Cosmos-Predict2 video diffusion model and attached a ControlNet.
Think of ControlNet as a sustain pedal or a tone knob on a piano. The base model already "plays"; ControlNet lets it adjust specific dimensions on demand. In PhyCo, the "knob" is a physical property map. Feed the model an image where each region's friction and restitution coefficients are color-coded, and it generates video obeying those physical laws.
Better still, the properties are continuously adjustable. Slide friction from 0 to 1, and the generated video smoothly transitions from "curling stone gliding" to "sandpaper scraping." This is not discrete labeling ("rough" or "smooth") but true quantitative control.
Step 3: Let AI Play Physics Teacher
Supervised learning alone is not enough. ControlNet learns to imitate the training data, but it may not truly "understand" physics — it might just memorize visual patterns.
So the team introduced a third component: VLM-guided reward optimization. They fine-tuned a Qwen2.5-VL vision-language model to answer physics questions. Feed it a generated video and ask: "Does the object's motion direction match the direction of the applied force?" Or: "Does the ball's rebound height match what a high-restitution material would predict?"
The VLM's answers are converted into differentiable reward signals that backpropagate to adjust ControlNet's parameters. It is like a student doing experiments, a teacher grading the lab reports, and the student improving based on the feedback — and the teacher needs no real-time human supervision, because it is another AI.
---
3. From Simulation to Reality: A Surprising Leap
What moves me most about PhyCo is not its simulated-data performance, but its generalization.
The model was never trained on real human videos. It only saw simple geometric bodies colliding, sliding, and deforming in a virtual environment. Yet when placed on the real-world Physics-IQ benchmark, it outperformed all baselines. A model trained only on simulated billiard tables understands the dynamics of a real pillow being squashed by a kettle.
This hints at a deep truth: physical laws are domain-invariant. Friction and elasticity do not change with color, texture, or lighting. What PhyCo learned is not pixel-level memory of "how a ball moves" but an abstract representation of "the relation between force and motion." Once extracted correctly, this representation can be transplanted from the greenhouse of synthetic data into the wilderness of the real world.
Even more striking is compositional generalization. The model understands "high friction" and "strong force" individually, and can combine them to generate "an object shoved hard across a rough surface" — even if that exact combination never appeared in training. This is no longer rote memorization, but a degree of causal reasoning.
---
4. "Looking Right" vs. "Moving Right"
PhyCo's significance goes beyond making AI video more lifelike. It touches a more fundamental question: what kind of "world model" do we actually want?
The dominant path is "scale is all you need" — make the model big enough and the data plentiful enough, and physical understanding will emerge. But Physics-IQ results squarely contradict that assumption. Sora's training data may be a million times larger than PhyCo's, yet its physical understanding falls short of a lightweight model trained on structured physics.
It reminds me of Feynman's own experience. Teaching in Brazil, he found students who could recite every formula and solve every textbook problem — but who did not understand physics. They did not know why polarized light splits through calcite, or what really causes tides. They could "calculate correctly" but not "think correctly."
Today's video generation models are, in a sense, those Brazilian students. They produce visually stunning footage but fail physics finals.
PhyCo offers a different path: rather than passively soaking up physical intuition from massive video corpora, actively implant physical laws into the model's representations through carefully designed experiments and feedback mechanisms. This is closer to human science education — controlled experiments, quantitative measurement, and teacher feedback.
---
5. An Unanswered Question
PhyCo has limits. It currently covers only four basic physical properties and is largely confined to solid mechanics. Fluids, thermodynamics, electromagnetism — these more complex domains remain low-scoring items on Physics-IQ.
But the framework itself is general. The same idea could train models to understand surface tension, viscosity, heat conduction, light refraction. Each time you need: a precise simulation environment, a set of decoupled control variables, and a judge capable of semantic feedback.
And that suggests a farther horizon. If video generation models one day truly master Newtonian mechanics, fluid dynamics, and thermodynamics, will they merely become better film tools? Or will they become genuine world simulators — predicting scenes that never happened, testing engineering solutions in virtual worlds, becoming collaborators in scientific discovery?
After all, the core idea of Feynman's path integrals is to treat physical law as a "sum over histories" — all possible paths, weighted by their action. And the process by which diffusion models generate video is itself a kind of "sum over histories" — starting from noise and converging step by step, along a probability gradient, toward the most likely future.
Perhaps the distance between physics and video generation is smaller than we think. PhyCo takes only the first step — but it is the crucial leap from "looking right" to "moving right."
---
References**