Self-Supervised Learning of Structured Dynamics from Videos: Structured Dynamics Model (SDM)

Forum topic · 小凯 · 2026-07-25

Summary

This arXiv paper (2507.19312) by Lukas Knobel, Andrew Zisserman, and Yuki M. Asano introduces the Structured Dynamics Model (SDM), a self-supervised approach for disentangling camera motion from object motion in video. Frame-to-frame changes entangle two sources of dynamics, which are hard to supervise separately in natural videos. SDM recovers structured motion representations from frozen features of a pretrained image vision transformer, explicitly separating the dominant source of temporal change from residual dynamics via future-feature prediction, instead of using entangled latent variables or dense transition tokens. Training combines self-supervised learning on real videos with weak supervision of scene dynamics on synthetic Kubric data. Evaluation uses ProbeMotion, a new benchmark covering synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or mean-pooled features and competes with strongly supervised representations like VGGT despite weaker supervision, showing pretrained image models can be repurposed for structured video dynamics.

Paper Overview

Field: Computer Vision (CV)
Authors: Lukas Knobel, Andrew Zisserman, Yuki M. Asano
Published: 2026-07-24
arXiv: 2507.19312

Abstract

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation.

The authors study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. They propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent variable or unstructured, spatially dense transition tokens.

Method

Builds on frozen features from a pretrained image ViT — no end-to-end video pretraining required.
Explicitly factorizes temporal change into a dominant source (typically camera motion) and residual dynamics (object motion).
Training combines self-supervised learning on real videos with weak supervision of scene dynamics on synthetic Kubric data.

Evaluation: ProbeMotion

The paper introduces ProbeMotion, a new evaluation suite covering synthetic and real videos with camera motion, object motion, and combined dynamics. Findings:

SDM outperforms backbone baselines using global CLS or mean-pooled features.
It performs competitively with strongly supervised representations such as VGGT on several probes, despite using far weaker supervision.

Conclusion

These results show that pretrained image models can be readily repurposed for structured video dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

---

*Auto-collected on 2026-07-25.*