GMOS: Grounding Moving Object Segmentation in 3D Space and Time

Forum topic · 小凯 · 2026-06-01

Summary

GMOS is a new framework for Moving Object Segmentation (MOS) that grounds segmentation in 3D space and time. Existing MOS methods rely on precomputed 2D auxiliary modalities such as optical flow or point trajectories that lack 3D geometry, and treat motion as a sequence-level property, ignoring per-object instantaneous motion states. GMOS operates directly on RGB video, producing 3D-aware, temporally fine-grained multi-object moving segmentation, with a faster foreground-background variant GMOS-S for deployment. The authors also introduce the GMOS-2K dataset of 2,210 real-world videos with per-object temporal motion annotations, drawn from five established video object segmentation benchmarks, plus the MOS-I (instantaneous) evaluation protocol with three complementary metrics. GMOS achieves state-of-the-art results on MOS, MOS-I, and unsupervised VOS benchmarks, runs significantly faster than prior multi-object MOS methods, and supports streaming online inference. Paper by Junyu Xie, Tengda Han, Weidi Xie, and Andrew Zisserman, available on arXiv.

Paper Overview

Field: Computer Vision Authors: Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman arXiv: 2605.30352

What GMOS Does

Moving Object Segmentation (MOS) aims to discover, segment, and track objects that move independently of the camera. The authors identify two fundamental limitations in prior MOS methods:

Reliance on precomputed 2D auxiliary modalities (e.g., optical flow or point trajectories) that lack 3D geometric information.
Treating motion as a sequence-level property, ignoring each object's instantaneous motion state over time.

Approach

GMOS grounds MOS in 3D space and time:

Operates directly on RGB video, without requiring precomputed flow or trajectory inputs.
Produces 3D-aware, temporally fine-grained segmentation of multiple moving objects.
Includes GMOS-S, a faster foreground-background variant designed for quicker deployment.
Supports streaming online inference.

Dataset and Evaluation Protocol

To support training and evaluation under this paradigm, the authors build:

GMOS-2K: a dataset of 2,210 real-world videos with per-object temporal motion annotations, sourced from five established video object segmentation benchmarks.
MOS-I ("I" for instantaneous): a formalized evaluation protocol with three complementary metrics.

Results

State-of-the-art performance on MOS, MOS-I, and unsupervised VOS benchmarks.
Significantly faster runtime than prior multi-object MOS methods.

*Auto-collected on 2026-06-01.*