SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

Forum topic · 小凯 · 2026-07-28

Summary

SM4RT is a Structured Motion 4D Reconstruction Transformer for end-to-end monocular 3D reconstruction and structured motion perception, proposed by researchers including Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo, Yuqi Wu, Jie Zhou, and Jiwen Lu (arXiv:2607.22534). While Geometry Foundation Models (GFMs) have advanced monocular 3D reconstruction, extending them to 4D dynamic understanding remains challenging. Existing motion perception methods, such as sparse tracking and dense point-wise flow, treat motion as independent point-wise displacements, ignoring that real-world objects follow rigid-body kinematics and points move collectively. SM4RT represents scene motion as a set of compact motion bases, each expressed as a time series of 6D twists in SE(3). Dense scene motion is recovered via sparse, temporally shared per-pixel assignment weights over these bases, ensuring points on the same object share common rigid-body motion trajectories. With parallel motion geometry encoders and decoders, SM4RT jointly infers 3D geometry, world-coordinate motion, and scene kinematic structure from monocular RGB video in a single forward pass, achieving strong motion reconstruction while preserving geometric structure.

Paper Overview

Field: Computer Vision Authors: Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo, Yuqi Wu, Jie Zhou, Jiwen Lu Published: 2026-07-24 arXiv: 2607.22534

Motivation

Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. Real-world objects usually obey rigid-body kinematics, so points move collectively rather than in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by SE(3), rather than unstructured point-wise displacements.

Method

Building on this insight, the authors propose SM4RT, a Structured Motion 4D Reconstruction Transformer for end-to-end 3D reconstruction and structured motion perception:

Motion bases: Scene motion is decomposed into a compact set of motion bases, each represented as a time series of 6D twists in SE(3).
Dense motion recovery: Dense scene motion is recovered via sparse, temporally shared per-pixel assignment weights over these bases, ensuring points on the same object share common rigid-body motion trajectories.
Architecture: Parallel motion geometry encoders and decoders jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure from monocular RGB video in a single forward pass.

Results

SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.

---

*Originally posted on zhichai.net, auto-collected 2026-07-28.*