ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

Forum topic · 小凯 · 2026-07-09

Summary

ProxyPose (arXiv 2507.06829) is a computer vision paper by Ruihang Zhang, Felix Taubner, and Pooja Ravi, published July 9, 2025. It reformulates six-degree-of-freedom (6-DoF) pose tracking from monocular video as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video: a synthetic clip of a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Because the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classical pose estimation with an off-the-shelf solver. This formulation leverages large-scale video pretraining to handle difficult aspects—challenging materials, occlusion, and deformation—without requiring 3D models, depth maps, object masks, or task-specific features. Trained only on synthetic data, ProxyPose achieves state-of-the-art tracking accuracy and extends to face tracking, camera pose estimation, and in-the-wild scenarios.

Paper Overview

Field: Computer Vision Authors: Ruihang Zhang, Felix Taubner, Pooja Ravi Published: 2025-07-09 arXiv: 2507.06829

Abstract

Tracking the six-degree-of-freedom (6-DoF) pose of objects and surfaces from monocular video is a long-standing problem in computer vision. To tackle this problem, existing methods require inputs beyond the video itself—such as 3D models, depth maps, object masks, or task-specific learned features—and they struggle with textureless, transparent, reflective, or deformable surfaces.

The authors introduce ProxyPose, which recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video—a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel.

Because the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classical pose estimation using an off-the-shelf solver. This formulation leverages large-scale video pretraining to absorb the hardest aspects of pose tracking—handling challenging materials, occlusion, and deformation—into the translation step, while operating at the pixel level without assumptions about object identity, boundaries, or global rigidity.

Key Results

ProxyPose achieves state-of-the-art 6-DoF pose tracking accuracy without the additional inputs required by competing methods.
The video diffusion model is fine-tuned only on synthetic data.
The method scales to face tracking, camera pose estimation, and challenging in-the-wild scenes that existing methods cannot handle.

Full paper: https://arxiv.org/abs/2507.06829

*Auto-collected on 2026-07-09*