Latent Dynamics Reasoning: A Video World Model That Extrapolates Learned Physics Beyond Training Distribution

Forum topic · 小凯 · 2026-08-11

Summary

This paper introduces Latent Dynamics Reasoning (LDR), a video world model designed to learn motion dynamics purely from pixels rather than merely fitting visual appearance. LDR reformulates latent transitions as explicit kinematic integration, numerically integrating lower-order dynamics while letting the neural network regress only the third- and higher-order residuals that drive rollout. Crucially, integration is performed on structured latents instead of dense convolutional features, improving out-of-distribution extrapolation. Evaluated on the PhyWorld white-box benchmark across five physics tasks (uniform motion, projectile, collision, bouncing, and approximation) at 256x256 resolution, LDR reduces the gap between in-distribution and out-of-distribution error by more than 20x relative to video diffusion baselines, while using 26x fewer parameters and running 143x faster. The authors demonstrate a striking generalization: trained only on red balls moving left-to-right, LDR correctly predicts right-to-left motion of blue squares, suggesting it is the first video world model to extrapolate learned dynamics outside the training distribution.

Overview

Area: Computer Vision
Authors: Haodong Li, Shaoteng Liu, Tianyu Wang
Released: 2026-08-11
arXiv: 2508.03804
Project page: https://lat-dyn-reason.github.io/

Problem

The world evolves according to its laws of motion. Leading video diffusion models primarily fit pixel distributions without explicitly modeling how pixels transition over time. As a result, they can render visually plausible frames while silently violating physical laws, and they tend to fail under distribution shift.

Method: Latent Dynamics Reasoning (LDR)

LDR captures dynamics purely from pixels by casting the latent transition as an explicit kinematic integration:

Lower-order dynamics (velocity, acceleration) are integrated numerically.
The neural network regresses only the third- and higher-order residual that drives the rollout.
Integration runs on a structured latent rather than dense convolutional features, so the integrator state is compact and extrapolates better.

Evaluation: PhyWorld Benchmark

Following the PhyWorld controlled white-box benchmark, LDR is tested on five physics tasks:

1. Uniform motion 2. Projectile motion 3. Collision 4. Bouncing 5. Approximation

The focus is on out-of-distribution (OOD) scenarios, probing whether the model has truly learned the underlying dynamics.

Key Results

At 256^2 resolution, on both single-task and joint-task training:
The gap between in-distribution and OOD error is more than 20x smaller than that of video diffusion baselines.
Parameters are 26x fewer, and runtime is 143x faster.
Cross-object generalization: Trained only on red balls moving left-to-right, LDR correctly predicts the right-to-left motion of blue squares under severe distribution shift.

Significance

To the authors' knowledge, this is the first video world model that extrapolates learned dynamics outside the training distribution, instead of memorizing visual patterns. The approach reframes video prediction as physics-aware latent integration rather than pixel-level synthesis.

Source

Original abstract (excerpt):

> The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (unifo...