Overview
Research area: Computer Vision Authors: Ruihang Zhang, Felix Taubner, Pooja Ravi Published: 2025-07-09 arXiv: 2507.06829
Abstract
Tracking the six-degree-of-freedom (6-DoF) pose of objects and surfaces from monocular video is a long-standing problem in computer vision. To tackle this problem, existing methods require inputs beyond the video itself—such as 3D models, depth maps, object masks, or task-specific learned features—and they struggle with textureless, transparent, reflective, or deformable surfaces.
ProxyPose recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video—a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Because the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classic pose estimation with off-the-shelf solvers.
Key points
- Leverages large-scale video pretraining to absorb the hardest aspects of pose tracking—challenging materials, occlusion, and deformation—into the translation step.
- Operates at the pixel level, requiring no assumptions about object identity, boundaries, or global rigidity.
- Achieves state-of-the-art 6-DoF pose tracking accuracy without the additional inputs required by competing methods, fine-tuning the video model only on synthetic data.
- Scales to facial tracking, camera pose estimation, and challenging in-the-wild scenes that existing methods cannot handle.