Imagine you're watching a movie. The protagonist throws a rubber ball at the ground—it bounces up in a graceful arc, exactly matching your intuition about "balls." But as the camera zooms in, you notice something strange: the moment the ball hits the ground, there's no deformation, no energy transfer or release, not even a temporal lag. It's like a texture sprite being "moved" back into the air by the program.
This isn't a low-budget VFX mistake. This is a fundamental error being made by today's most advanced video generation AIs: it knows the ball "should" bounce, but it doesn't understand why the ball bounces.
---
1. The Illusion of Visual Realism
Over the past two years, video generation models have exploded. Sora, Runway, Pika, Kling... their outputs are increasingly hard to distinguish from real footage. Lighting, textures, cinematography—sometimes even professional editors have to squint to tell real from fake.
But earlier this year, a benchmark called Physics-IQ poured cold water on the party.
A research team designed 198 real-world physical scenarios—balls rolling downhill, water pouring into a cup, magnets attracting nails, candles heating balloons—and used them to test eight mainstream video generation models. The results revealed a troubling rift: a model's visual realism and its physical understanding are almost two entirely different things.
Even the best-performing model scored only 24.1% on Physics-IQ (normalized to 100%). In other words, on the task of "predicting what happens in the next second," these AIs barely beat random guessing. More ironic still: Sora scored 55.6% on the visual realism test of "can it fool multimodal LLMs," far ahead of the pack—but in physical understanding, it was just as hopeless as the rest.
This exposes a deep problem: current video diffusion models are, at their core, extremely sophisticated pixel predictors. They aren't simulating the world; they're imitating the statistical history of pixels. When the training data contains a million videos of balls hitting the ground, the AI learns that "balls usually bounce after landing"—but it never learned elastic deformation, energy conservation, or momentum transfer. So when a scene drifts slightly outside the training distribution, the cracks show: balls hovering in mid-air, collisions where objects ghost through each other, soft clay landing stiff as a steel plate.
---
2. Giving AI a Physics Lesson
The question is: can we teach AI genuine physical intuition, rather than just letting it memorize more videos?
A paper that just appeared on arXiv, PhyCo, offers an extremely clever answer. Its approach resembles how humans learn physics—we don't understand friction by watching a hundred thousand real-world videos; we understand it by running controlled experiments.
Step 1: Build a Virtual Laboratory
The team built a large-scale virtual lab using the PyBullet physics engine and Blender renderer. It contains six classic scenarios: bricks sliding on flat surfaces, balls bouncing off walls, soft balls in free fall, objects striking deformable surfaces, billiard collisions... In each scenario, four core physical parameters are systematically varied:
- Friction: from ice to sandpaper
- Restitution (elasticity): from putty to superballs
- Deformation: from steel to jelly
- Force: from a gentle nudge to a hard strike
- Narayanan et al. (2026). *PhyCo: Learning Controllable Physical Priors for Generative Motion.* arXiv:2604.28169
- Motamed et al. (2025). *Do generative video models learn physical principles from watching videos?* Physics-IQ benchmark.
- Zhang et al. (2023). *Adding Conditional Control to Text-to-Image Diffusion Models* (ControlNet).
- NVIDIA (2025). *Cosmos World Foundation Model.*
In total, 100,000 videos were generated. The key design: these videos aren't just "visually plausible"—they are physically exactly solvable. Every video comes with pixel-aligned physical property maps—like labeling every pixel with "the friction coefficient here is 0.3." This lets the model decouple visual appearance from underlying dynamics: the same ball can be red or blue, but its sliding distance on the same surface should be determined solely by the friction coefficient.
Step 2: Tuning Physics Like a Palette
Next comes the most elegant part. Rather than training a video model from scratch—far too expensive—they took NVIDIA's pretrained Cosmos-Predict2 video diffusion model and attached a ControlNet.
What is ControlNet? Think of it as a sustain pedal or tone knob for a piano. The base model already knows how to "play"; ControlNet lets it adjust specific dimensions on demand. In PhyCo, that "knob" is the physical property map. You feed the model an image with colors encoding the friction and restitution coefficients of each region, and it generates video consistent with those physical laws.
Even better, these properties are continuously adjustable. You can slide friction from 0 to 1, and the generated video transitions smoothly from "curling-stone glide" to "sandpaper drag." This isn't discrete labeling ("rough" or "smooth")—it's genuine quantitative control.
Step 3: Let an AI Be the Physics Teacher
But supervised learning alone isn't enough. ControlNet learns to imitate the training data, but it doesn't necessarily "understand" physics—it might just be memorizing visual patterns.
So the team introduced a third component: VLM-guided reward optimization. They fine-tuned a Qwen2.5-VL vision-language model specifically to answer physics questions. Feed it a generated video and ask: "Is the object's motion direction consistent with the applied force?" or "Does the ball's rebound height match expectations for a high-restitution material?"
The VLM's answers are converted into differentiable reward signals, backpropagated to adjust the ControlNet's parameters. It's like: a student runs an experiment, a teacher grades the lab report, and the student keeps improving based on the feedback. And this teacher requires no real-time human supervision—it's another AI.
---
3. From Simulation to Reality: A Striking Leap
What impressed me most about PhyCo isn't its performance on simulated data—it's its generalization.
The model was never trained on real footage of humans. It has only seen simple geometric bodies colliding, sliding, and deforming in a virtual environment. Yet when tested on the real-world Physics-IQ benchmark, it outperformed all baseline models. A model trained only on simulated billiard tables can understand the dynamics of a real pillow being squashed by a kettle.
This hints at a profound truth: physical laws are domain-invariant. Friction and elasticity don't change based on an object's color, material texture, or lighting. What PhyCo learned isn't pixel-level memory of "how balls should move," but an abstract representation of the relationship between force and motion. Once that representation is correctly extracted, it can be transplanted from the synthetic greenhouse into the real-world wilderness.
Even more surprising is the compositional generalization. The model can individually understand "high friction" and "strong force," and can also combine them—generating "an object shoved hard on a rough surface"—even if that exact combination never appeared in training. This is no longer rote memorization; it's some degree of causal reasoning.
---
4. "Looks Right" vs. "Moves Right"
PhyCo's significance goes far beyond making AI video more realistic. It touches a more fundamental question: what kind of "world model" do we actually want?
The mainstream path today is "scale is all you need"—make the model big enough and the data plentiful enough, and physical understanding will emerge naturally. But the Physics-IQ results hit that assumption hard. Sora's training data may be millions of times larger than PhyCo's, yet on physical understanding it underperforms a lightweight model trained with structured physics.
This reminds me of a story about Feynman himself. While teaching in Brazil, he found that students could recite every physics formula and solve every textbook problem—but they didn't understand physics. They didn't know why polarized light splits through calcite, or what truly causes tides. They could "compute correctly" but not "think correctly."
Today's video generation models are, in a sense, those Brazilian students. They generate visually stunning footage, but they fail the physics final.
PhyCo offers a different path: instead of passively soaking physical intuition out of massive video corpora, actively implant physical laws into the model's representation via carefully designed experiments and feedback mechanisms. This path is closer to human science education—controlled experiments, quantitative measurement, and teacher feedback.
---
5. An Unanswered Question
PhyCo has limits. It currently covers only four basic physical properties and is mostly confined to solid mechanics. Fluids, thermodynamics, electromagnetism—these more complex domains remain low-scoring items on the Physics-IQ benchmark.
But the framework itself is general. You could use the same approach to teach a model surface tension, viscosity, heat conduction, light refraction. Each time, you'd need: a precise simulation environment, a set of decoupled control variables, and a judge capable of semantic feedback.
This points to a more distant vision. If one day our video generation models truly learn Newtonian mechanics, fluid dynamics, and thermodynamics, will they merely become better filmmaking tools? Or will they become true world simulators—able to predict scenarios that never happened, to test engineering designs in virtual worlds, to serve as collaborators in scientific discovery?
After all, the core idea of Feynman's path integrals is to view physical law as a "sum over histories"—all possible paths superposed, weighted by their physical action. And the process by which diffusion models generate video is, in essence, also a "sum over histories"—starting from noise and converging step by step along probability gradients toward the most likely future.
Perhaps the distance between physics and video generation is far smaller than we think. PhyCo has only taken the first step—but it's the crucial leap from "looks right" to "moves right."
---
References