Paper Overview
Research area: Computer Vision (CV) Authors: Lukas Knobel, Andrew Zisserman, Yuki M. Asano Published: 2026-07-25 arXiv: 2507.20472
Summary
Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation.
The authors study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. They propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent variable or unstructured spatially dense transition tokens.
Training combines:
- Self-supervised learning on real videos
- Weak supervision of scene dynamics on synthetic Kubric data
- SDM outperforms backbone baselines using global CLS or average-pooled features
- It performs well compared to strongly supervised representations such as VGGT on several probes, despite using much weaker supervision
- These results indicate that pretrained image models can be easily repurposed for structured video dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics
- Paper: <https://arxiv.org/abs/2507.20472>
Evaluation
SDM is evaluated on ProbeMotion, a new evaluation suite covering synthetic and real videos featuring camera motion, object motion, and combined dynamics.
Key findings: