ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

Forum topic · 小凯 · 2026-07-09

Summary

ProxyPose is a new approach to six-degree-of-freedom (6-DoF) pose tracking from monocular video, presented by Ruihang Zhang, Felix Taubner, and Pooja Ravi in arXiv paper 2507.06829 (July 2025). Instead of requiring extra inputs like 3D models, depth maps, object masks, or task-specific learned features, the method reformulates pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video: a synthetic clip of a colored polyhedron undergoing the same local rigid-body motion as the marked surface region. Since the proxy's geometry and appearance are known by construction, recovering its 6-DoF trajectory reduces to classic pose estimation with off-the-shelf solvers. This leverages large-scale video pretraining to handle textureless, transparent, reflective, and deformable surfaces, occlusions, and deformation, without assumptions about object identity, boundaries, or global rigidity. ProxyPose achieves state-of-the-art accuracy while fine-tuned only on synthetic data, and extends to face tracking, camera pose estimation, and challenging in-the-wild scenarios.

Overview

Field: Computer Vision Authors: Ruihang Zhang, Felix Taubner, Pooja Ravi Published: 2025-07-09 arXiv: 2507.06829

Abstract

Tracking the six-degree-of-freedom (6-DoF) pose of objects and surfaces from monocular video is a long-standing problem in computer vision. To tackle this problem, existing methods require inputs beyond the video itself — such as 3D models, depth maps, object masks, or task-specific learned features — and they struggle with textureless, transparent, reflective, or deformable surfaces.

ProxyPose recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel in the first frame, a fine-tuned video diffusion model translates the input into a proxy video — a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Because the proxy's geometry and appearance are known by construction, recovering its full 6-DoF trajectory reduces to classic pose estimation with an off-the-shelf solver.

Key Points

Problem reformulation: 6-DoF pose tracking is treated as a video-to-video translation task rather than direct pose regression or correspondence matching.
Diffusion-based translation: A fine-tuned video diffusion model converts the input video into a proxy video of a known colored polyhedron mimicking the local surface motion at the marked pixel.
Classic estimation on top: With the proxy's geometry and appearance known by construction, full 6-DoF trajectory recovery uses standard off-the-shelf pose estimation solvers.
Leverages video pretraining: Large-scale video pretraining absorbs the hardest aspects of pose tracking — challenging materials, occlusion, and deformation — into the translation step.
Minimal assumptions: Operates at the pixel level without assumptions about object identity, boundaries, or global rigidity.
No extra inputs needed: Achieves state-of-the-art 6-DoF tracking accuracy without 3D models, depth maps, or masks, and the video model is fine-tuned only on synthetic data.
Extensibility: Scales to face tracking, camera pose estimation, and challenging in-the-wild scenes that existing methods cannot handle.