Self-Supervised Learning of Structured Dynamics from Videos (SDM, arXiv 2507.21738)

Forum topic · 小凯 · 2026-07-27

Summary

This paper introduces the Structured Dynamics Model (SDM), a method for disentangling camera motion from object motion in videos. The authors — Lukas Knobel, Andrew Zisserman, and Yuki M. Asano — show that structured motion representations can be recovered from frozen features of a pretrained image Vision Transformer. Instead of encoding video change with a single entangled latent variable or unstructured spatially dense transition tokens, SDM explicitly separates the dominant source of temporal change from residual dynamics via future-feature prediction. Training combines self-supervised learning on real videos with weak supervision on scene dynamics using synthetic Kubric data. Evaluation is performed on ProbeMotion, a new benchmark suite covering synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features and competes with strongly supervised representations such as VGGT on several probes despite much weaker supervision. Results indicate pretrained image models can be readily repurposed for structured video dynamics representation. arXiv: 2507.21738.

Paper Overview

Field: Computer Vision Authors: Lukas Knobel, Andrew Zisserman, Yuki M. Asano Published: 2025-07-27 arXiv: 2507.21738

Introduction

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation.

Method: Structured Dynamics Model (SDM)

The authors study whether structured motion representations can be recovered from frozen features of a pretrained image Vision Transformer. The proposed Structured Dynamics Model (SDM):

Explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction.
Avoids representing video change with a single entangled latent variable or unstructured spatially dense transition tokens.
Trains with self-supervised learning on real videos, combined with weak supervision on scene dynamics using synthetic Kubric data.

Evaluation: ProbeMotion Benchmark

SDM is evaluated on ProbeMotion, a new evaluation suite covering synthetic and real videos featuring camera motion, object motion, and combined dynamics.

Key Findings

SDM outperforms backbone baselines that use global CLS or average-pooled features.
It performs competitively with strongly supervised representations such as VGGT on several probes, despite using far weaker supervision.

Conclusion

These results suggest that pretrained image models can be readily repurposed for structured video dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

---

*Auto-collected on 2026-07-27*