Detailed Geometry and Appearance from Opportunistic Motion: Joint Pose and Shape Optimization with 2D Gaussian Splatting

Research field: Computer Vision Authors: Ryosuke Hirai, Kohei Yamashita, Antoine Guédon Published: 2025-03-30 arXiv: 2503.23761

Summary

Reconstructing 3D geometry and appearance from a sparse set of fixed cameras is a foundational task with broad applications, yet it remains fundamentally constrained by limited viewpoints. This paper shows that this bound can be broken by exploiting opportunistic object motion: as a person manipulates an object (e.g., moving a chair or lifting a mug), the static cameras effectively "orbit" the object in its local coordinate frame, providing additional virtual viewpoints.

Challenges

• Tight coupling between object pose and geometry estimation.
• Complex appearance variations of a moving object under static illumination.

Method

• A joint pose and shape optimization formulated with 2D Gaussian splatting, using alternating minimization over the 6-DoF object trajectory and the primitive parameters.
• A novel appearance model that decomposes diffuse and specular components in spherical harmonics space via reflection-direction probing.

Results

Extensive experiments on synthetic and real datasets demonstrate that the proposed method recovers significantly more accurate geometry and appearance than state-of-the-art baselines under extremely sparse viewpoints.

Original abstract (excerpt):

> Reconstructing 3D geometry and appearance from a sparse set of fixed cameras is a foundational task with broad applications, yet it remains fundamentally constrained by the limited viewpoints. We show that this bound can be broken by exploiting opportunistic object motion: as a person manipulates an object (e.g., moving a chair or lifting a mug), the static cameras effectively "orbit" the object in its local coordinate frame, providing additional virtual viewpoints. Harnessing this object motion, however, poses two challenges: the tight coupling of object pose and geometry estimation and the complex appearance variations of a moving object under static illumination. We address these by formulating a joint pose and shape optimization using 2D Gaussian splatting with alternating minimization...

Paper link: https://arxiv.org/abs/2503.23761