ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

Forum topic · 小凯 · 2026-07-09

Summary

ProxyPose is a new computer vision method from arXiv paper 2507.06829 (Ruihang Zhang, Felix Taubner, Pooja Ravi, July 2025) that reframes 6-DoF pose tracking as video-to-video translation. Given only a monocular video and a single marked pixel in the first frame, a fine-tuned video diffusion model generates a proxy video: a synthetic clip of a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Since the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classical pose estimation with off-the-shelf solvers. This approach leverages large-scale video pretraining to handle difficult aspects such as challenging materials, occlusion, and deformation, operating at the pixel level without assumptions about object identity, boundaries, or global rigidity. ProxyPose achieves state-of-the-art 6-DoF pose tracking accuracy without the extra inputs (3D models, depth maps, masks, or learned features) required by competing methods, and is fine-tuned only on synthetic data. The authors also demonstrate extensions to face tracking, camera pose estimation, and challenging in-the-wild scenes.

Paper Overview

Field: Computer Vision Authors: Ruihang Zhang, Felix Taubner, Pooja Ravi Published: 2025-07-09 arXiv: 2507.06829

Abstract

Tracking the six-degree-of-freedom (6-DoF) pose of objects and surfaces from monocular video is a long-standing problem in computer vision. Existing methods require inputs beyond the video itself — such as 3D models, depth maps, object masks, or task-specific learned features — and they struggle with textureless, transparent, reflective, or deformable surfaces.

ProxyPose recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video: a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel.

Key Ideas

Video-to-video translation: A video diffusion model converts the input video into a proxy video featuring a known synthetic object (colored polyhedron).
Simplified pose recovery: Because the proxy's geometry and appearance are known by construction, recovering its 6-DoF trajectory reduces to classical pose estimation with off-the-shelf solvers.
Pretraining advantage: The formulation leverages large-scale video pretraining to absorb the hardest aspects of pose tracking — challenging materials, occlusion, and deformation — into the translation step.
Minimal assumptions: Operates at the pixel level with no assumptions about object identity, boundaries, or global rigidity.

Results

State-of-the-art 6-DoF pose tracking accuracy without the additional inputs required by competing methods.
Fine-tuned only on synthetic data.
Extends to face tracking, camera pose estimation, and challenging in-the-wild scenes that existing methods cannot handle.

--- *Auto-collected 2026-07-09*