MoSE3: Learning World-Space SE(3) Motion at Every Pixel from Monocular RGB Video

Forum topic · 小凯 · 2026-10-06

Summary

MoSE3 is the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, outputting full 6-DoF rigid transforms at every pixel in world space. Unlike dense 3D point tracking, which only captures 3-DoF translation curves per pixel, per-pixel SE(3) motion simultaneously provides rotation, translation, and rigid grouping information. Since rotations lie on a curved manifold unsuited to Euclidean regression and SE(3) annotations are scarce, MoSE3 jointly learns two intermediate quantities—3D point tracks and rigidity embeddings—then recovers SE(3) via differentiable transform fitting within soft rigid clusters, enabling end-to-end prediction and supervision. The authors also introduce Art-Kubric, a large-scale synthetic dataset of articulated objects with rich physical interactions, dense SE(3), and rigidity labels. Trained only on synthetic motion data, MoSE3 achieves state-of-the-art pixel-, part-, and object-level SE(3) estimation on rigid and articulated benchmarks, best 3D point tracking accuracy across three datasets, and strong generalization to real-world videos. Paper: arXiv 2610.03716.

Paper Overview

Research Area: Computer Vision Authors: Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du, Qianqian Wang Published: 2026-10-02 arXiv: 2610.03716

Abstract

Dense 3D point tracking has been a prominent paradigm for modeling motion in dynamic scenes, but a point track is just a 3-DoF translation curve per pixel: it captures where pixels go, not the rotation of the underlying part, nor which pixels move together as one body.

We propose MoSE3, the first feed-forward model that predicts dense SE(3) motion from monocular RGB video, producing full 6-DoF rigid transforms at every pixel in world space. Per-pixel SE(3) motion offers a richer view of how a scene moves: rotation, translation, and grouping all at once.

Directly predicting SE(3) is challenging: rotations lie on a curved manifold that is ill-suited to Euclidean regression, and annotations for SE(3) are particularly difficult to acquire.

To address these challenges, MoSE3 predicts per-pixel SE(3) through two jointly learned intermediate quantities—3D point tracks and rigidity embeddings—and recovers SE(3) via differentiable transform fitting within each soft rigid cluster, enabling end-to-end prediction and supervision.

To fill the data gap, the authors introduce Art-Kubric, a large-scale synthetic dataset of articulated objects with rich physical interactions, providing dense SE(3) and rigidity labels.

Key Contributions

First feed-forward model for dense world-space SE(3) motion estimation from monocular RGB video
Predicts SE(3) via jointly learned 3D point tracks and rigidity embeddings, with differentiable transform fitting inside soft rigid clusters
Introduces Art-Kubric, a large-scale synthetic dataset with dense SE(3) and rigidity annotations for articulated objects
State-of-the-art results on pixel-, part-, and object-level SE(3) estimation for rigid and articulated object benchmarks
Best average 3D point tracking accuracy across three datasets, with strong generalization to real-world videos despite training only on synthetic motion data