ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation with Video Diffusion Models

Forum topic · 小凯 · 2026-07-08

Summary

ProxyPose (arXiv:2607.06555) is a computer vision method that performs 6-DoF (six degrees of freedom) pose tracking from monocular video by reframing the task as video-to-video translation. Instead of directly regressing the 3D pose of a real object, a fine-tuned video diffusion model translates the input video—plus a single marked pixel on the target surface—into a 'proxy video' showing a simply shaped, colorfully textured polyhedron whose motion matches the tracked surface. Since the proxy's geometry and appearance are known, pose estimation reduces to a classic PnP problem solvable with off-the-shelf solvers. The diffusion model is fine-tuned entirely on synthetic data, requiring no real-world annotations, 3D object models, depth maps, or object masks. By leveraging pretrained video diffusion models' internalized knowledge of physics and motion, ProxyPose handles challenging materials like reflective, transparent, and textureless surfaces. The paper reports state-of-the-art accuracy on pose tracking benchmarks and extends to facial tracking and camera pose estimation in in-the-wild videos, illustrating a shift from discriminative to generative paradigms in vision.

ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

*Source: a paper-review post on zhichai.net discussing arXiv:2607.06555, "ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation" by Ruihang Zhang, Felix Taubner, and Pooja Ravi.*

Core Idea

Recovering the 6-DoF pose (3 rotation + 3 translation) of an object from monocular video is one of computer vision's hardest problems: depth ambiguity, occlusion, textureless surfaces, reflections, and deformations all break traditional trackers. Most existing methods also require extra inputs—3D models, depth maps, object masks, or hand-crafted features.

ProxyPose reframes the problem: instead of estimating pose directly, a video diffusion model translates the real-world video into a "proxy video" showing a simple, colorfully textured polyhedron whose motion exactly mirrors the tracked surface region. Because the proxy's shape and colors are known, recovering its 6-DoF trajectory becomes a classic PnP (Perspective-n-Point) problem, solved efficiently with off-the-shelf solvers plus temporal smoothing.

Input: a monocular video + a single marked pixel on the target surface in the first frame. Output: the 6-DoF trajectory of that surface region.

How It Works

1. Video-to-video translation. A pretrained video diffusion model (e.g., Wan, CogVideo) is fine-tuned to map real videos to proxy videos. The model's pretrained knowledge of motion, lighting, occlusion, and materials lets it "absorb" the hard part of the problem. 2. Synthetic-only training. Fine-tuning uses fully synthetic data: random 3D scenes are rendered as RGB inputs, paired with rendered proxy polyhedron videos following a surface patch. No real-world annotation is needed. 3. Pose extraction. Each frame's polyhedron faces are identified by color, 2D-3D correspondences are established, and PnP (with RANSAC/least squares) yields the pose, followed by temporal smoothing.

Key Advantages

Requires only video + one initial pixel—no 3D model, depth sensor, mask, or task-specific features
Handles reflective, transparent, and textureless surfaces thanks to diffusion-model pretraining
Operates at pixel level; no global rigidity assumption
Trained purely on synthetic data, yet generalizes well to real-world footage

Results

State-of-the-art 6-DoF tracking accuracy on standard benchmarks, outperforming methods that need extra inputs
Extends to facial tracking, camera pose estimation, and in-the-wild videos
Ablations show multi-colored polyhedra beat single-color proxies; the framework is model-agnostic across diffusion backbones; more synthetic data improves generalization

Discussion Highlights (from the original post)

Generation as understanding: rather than regressing pose, the model generates an equivalent proxy scene—generation serves as evidence of comprehension, analogous to explaining a concept in one's own words.
Proxy representations: the idea echoes point masses in physics, ball-and-stick molecular models, and proxy geometry in graphics—preserve the motion signal, discard irrelevant appearance.
Discriminative → generative paradigm: generative intermediates exploit large-scale pretraining, handle ambiguity, and are interpretable (the proxy video can be inspected).
Future directions: motion capture, physical property estimation, scene reconstruction, and anomaly detection via proxy-real deviations.

References

Zhang, R., Taubner, F., & Ravi, P. "ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation." arXiv:2607.06555, 2026.
Lepetit, V., et al. "EPnP: An Accurate O(n) Solution to the PnP Problem." IJCV, 2009.
Ho, J., et al. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020.
Wan, T.H., et al. "Wan: Open World Video Generation." 2025.
Brooks, T., et al. "Video Generation Models as World Simulators." OpenAI, 2024.