SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

Forum topic · 小凯 · 2026-07-28

Summary

SM4RT is a Structured Motion 4D Reconstruction Transformer for end-to-end monocular 3D reconstruction and structured motion perception, presented in arXiv paper 2607.22534 by Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo, Yuqi Wu, Jie Zhou, and Jiwen Lu. While Geometry Foundation Models (GFMs) have greatly advanced monocular 3D reconstruction, extending them to 4D dynamic understanding remains challenging. Existing motion perception methods such as sparse tracking and dense point-wise flow treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. SM4RT instead models scene motion as a set of compact motion bases, each represented as a time series of 6D screws in SE(3), reflecting the rigid-body kinematics of real-world objects. Dense scene motion is recovered via sparse, temporally shared per-pixel assignment weights over these bases, so points on the same object share common rigid-body motion trajectories. With parallel motion geometry encoders and decoders, SM4RT jointly infers 3D geometry, world-coordinate motion, and scene kinematic structure from a single forward pass on monocular RGB video, achieving strong motion reconstruction while preserving geometric structure.

Paper Overview

Field: Computer Vision Authors: Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo, Yuqi Wu, Jie Zhou, Jiwen Lu Published: 2026-07-24 arXiv: 2607.22534

Abstract

Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by SE(3), rather than unstructured point-wise displacements.

Building on this insight, the authors propose SM4RT, a Structured Motion 4D Reconstruction Transformer for end-to-end 3D reconstruction and structured motion perception.

Key Ideas

Structured motion representation: Scene motion is decomposed into a set of compact motion bases, where each basis is expressed as a time series of 6D screws in SE(3).
Dense motion recovery: Dense scene motion is restored through sparse, temporally shared per-pixel assignment weights over these bases, ensuring that points on the same object share common rigid-body motion trajectories.
Architecture: SM4RT introduces parallel motion geometry encoders and decoders that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure from a single forward pass on monocular RGB video.
Results: SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.

---

*Source: arXiv:2607.22534, auto-collected on 2026-07-28.*