The unanswerable question
A two-second clip shows a ball dropped from a height, hitting the ground, and bouncing back to half its original height. Is the physics correct? You can't tell. You don't know the frame rate, the pixel-to-centimeter scale, the camera setup, or the ball's mass. If the frame rate was altered, fall time changes and gravity becomes impossible to compute accurately.
This is the first wall in evaluating the physical correctness of video generation models.
The workaround: test relationships, not absolute values
A team from the Indian Institute of Science (IISc) and Johns Hopkins University proposed sidestepping the problem: place two objects obeying the same law in one video and check only their relationship.
Two blocks of different masses on identical inclined planes must reach the bottom simultaneously. This criterion depends on neither scale, camera calibration, nor frame rate. It only requires the model to make both objects follow the same rule.
The method is called Principia, described in the paper at arXiv:2609.04200, with a public project page (principiabench.github.io).
The exam content
Eight classes of Newtonian mechanics phenomena are tested:
- Gravity
- Coefficient of restitution
- Friction
- Rotational inertia
- Projectile motion
- Momentum
- Pendulums
- Mass-spring oscillation
- On VBench, a general video quality benchmark, all six models score around 0.8.
- On Principia, none exceeds 0.42.
- Can training fix the scores? Plugging relational invariants into reward or loss functions—how far above 0.42 can it go?
- Coverage: extending from eight Newtonian laws to contact, fluids, and soft bodies—will models fall harder?
- Tracker sensitivity: does swapping the tracker change the scores?
- Robotic relevance: do generators with high physical consistency yield better world-model task success rates? If not, the benchmark's engineering value shrinks.
- VLM-as-judge: the 67% ceiling determines whether automated model-based review remains viable.
The data was recorded by the authors: 401 real controlled scenes, each contributing a first frame plus a text prompt as the generation condition. The first frames pass through an image-editing model to erase the experimenter and equipment, then augmentation brings the total to 529 scenarios, from roughly 750+ raw videos.
How scores are computed
1. First frame + prompt feeds a video generation model. 2. SAM3 tracks object centroids (or bottom edges) frame by frame. 3. Events are detected: landing (centroid crosses a ground threshold while vertical velocity flips sign), bounce apex (vertical velocity flips from negative to positive), pendulum half-periods (horizontal velocity reversal past the anchor line, doubled for full period), and block masses estimated as the cube of edge length assuming constant density. 4. Relational invariants are computed. 5. A directional consistency filter first removes videos with fundamentally wrong motion direction, then a continuous relational consistency score is assigned.
Results: 0.8 vs. 0.42, two parallel lines
The paper's appendix lists six evaluated generative models: Wan2.2-5B, Wan2.2-14B, Cosmos-2.5-2B, Cosmos-2.5-14B, Veo-3.1, and Omni. (Some news aggregators reported five models; the paper appendix is authoritative.)
Per the paper, these two scores measure different capabilities. A striking observation: scaling up model size does not steadily improve physical consistency.
VLMs were also tested as judges—asked to detect physics violations against binary ground truth. The best vision-language model reached only 67% accuracy; most performed near random, and average agreement never exceeded 0.7.
The cost of this benchmark
Generating the full evaluation corpus consumed substantial A100 hours. The appendix lists per-video wall-clock time (A100 80GB): Wan2.2-5B at 7 minutes, Wan2.2-14B at 66 minutes, Cosmos-2.5-2B at 15 minutes, Cosmos-2.5-14B at 74 minutes. Veo-3.1 and Omni use public APIs. The sheer cost of reproducing this evaluation explains why no one had done it before.
Three caveats
1. All 529 scenarios and eight laws are Newtonian mechanics, mostly pairwise objects. Multi-object interaction, contact, fluids, and deformation are out of scope—so the conclusion only covers high-school-level physics. 2. The measurement pipeline uses SAM3; one model judging another means tracking errors get logged as physics violations, and this error is not separately quantified in the paper. 3. Relational invariants block some gaming, but the possibility that models exploit plausible visual priors rather than actual laws is not ruled out.
Why this matters
Video generation models have been cast as world-model candidates. Robotics, autonomous driving, and simulation environments all bank on the same bet: models trained on massive video learn how objects move and can predict post-action frames. This benchmark tests exactly that foundation. Between visual realism and physical consistency, the current picture is two nearly parallel lines.
In the same week, a related effort called TrackEverything—from Carnegie Mellon and Meta—ties computation to unique 3D content in a scene, tracking all points in world coordinates across clips exceeding 1,000 frames. Evaluation tooling is getting denser, a prerequisite for seriously discussing what world models actually learn. (Based on news aggregation; the primary paper was not reviewed.)
Open questions
References
1. Principia: Relational Physics Tests for Video Models, arXiv:2609.04200 2. principiabench.github.io (project page) 3. Anand Bhattad's five-post project thread on Bluesky, 2026-09-08 4. AGI Hunt entry on Principia, 2026-09 5. FutureX / AGI Hunt paraphrase of TrackEverything, 2026-09-26 (second-hand, primary paper not reviewed)