Self-Supervised Learning of Structured Dynamics from Videos (SDM, arXiv:2507.19312)

Forum topic · 小凯 · 2026-07-25

Summary

This post introduces the Structured Dynamics Model (SDM), a paper by Lukas Knobel, Andrew Zisserman, and Yuki M. Asano (arXiv:2507.19312) on disentangling camera motion from object motion in video representation learning. Frame-to-frame change in natural videos entangles two sources of dynamics, making separate supervision difficult. SDM shows that structured motion representations can be recovered from the frozen features of a pretrained image vision transformer. Instead of using a single entangled latent or unstructured spatially dense transition tokens, SDM explicitly separates the dominant source of temporal change from residual dynamics via future-feature prediction. Training combines self-supervised learning on real videos with weak supervision of scene dynamics on synthetic Kubric data. The model is evaluated on ProbeMotion, a new benchmark covering synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features and performs competitively with strongly supervised representations such as VGGT, despite much weaker supervision, suggesting pretrained image models can be readily repurposed for structured video dynamics.

Self-Supervised Learning of Structured Dynamics from Videos

Field: Computer Vision Authors: Lukas Knobel, Andrew Zisserman, Yuki M. Asano arXiv: 2507.19312

Overview

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation.

Key Contributions

The authors study whether structured motion representations can be recovered from frozen features of a pretrained image vision transformer.
They propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent variable or unstructured, spatially dense transition tokens.
Training combines self-supervised learning on real videos with weak supervision of scene dynamics on synthetic Kubric data.

Evaluation: ProbeMotion

The paper introduces ProbeMotion, a new evaluation suite covering synthetic and real videos with camera motion, object motion, and combined dynamics. Results:

SDM outperforms backbone baselines that use global CLS or average-pooled features.
It performs well compared to strongly supervised representations such as VGGT on several probes, despite using much weaker supervision.

Conclusion

These results show that pretrained image models can be readily repurposed for structured video dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

Original Abstract

> Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction...

Links: arXiv:2507.19312