Paper Overview
- Research area: Computer Vision (CV)
- Authors: Mingju Gao, Qingle Liu, Yuzhao Peng, Xinjie Lin, Ziming Qin, Zheng Jiang, Wenyi Li, Calvin Xiao, Youjie Zheng, Kaisen Yang, Qinhuai Na
- Published: 2026-10-06
- arXiv: 2610.08791
- 40 controlled tasks spanning:
- Mechanics
- Optics
- Fluids
- Thermal and phase-change phenomena
- Electromagnetism
- Surface tension
- Each task pairs an initial image and a generation prompt with predefined physical criteria, enabling interpretable tests of observable physical relationships without requiring reference videos.
- The evaluator combines task observability screening with task-specific quantitative physical measurements.
- Experiments on 1,280 videos from 8 video generation models revealed persistent physical inconsistencies and significant variation across tasks; the best model achieved an overall score of 57.76/100.
- Evaluation on synthetic videos with known physical relationships provides evidence of the measurement modules' effectiveness under controlled conditions.
- The evaluator's agreement with human judgments outperforms direct vision-language model baselines.
Summary
Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and planning in embodied AI systems. Existing evaluations often rely on model-based judgments or reference videos, while direct physical tests have largely focused on mechanics.
The authors introduce World Models' Last Exam in Physics, a measurement-based benchmark for evaluating physical consistency in video world models.
Benchmark Design
Findings
*Auto-collected on 2026-10-08*