ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation with Video Diffusion Models

Forum topic · 小凯 · 2026-07-08

Summary

ProxyPose (arXiv:2607.06555) reframes 6-DoF pose tracking as a video-to-video translation problem. Instead of directly estimating the 3D pose of an object from monocular video, a fine-tuned video diffusion model translates the input video into a 'proxy video' showing a simple colored polyhedron whose motion matches the tracked real-world surface. Because the proxy's geometry and colors are known, pose recovery reduces to a classical PnP problem solved with established, fast solvers. The diffusion model is trained entirely on synthetic data, requiring no 3D models, depth maps, masks, or real-world annotations. By leveraging pre-trained video diffusion knowledge of physics and motion, ProxyPose achieves state-of-the-art tracking accuracy on benchmarks while handling challenging materials such as reflective, transparent, and textureless surfaces. The method also generalizes to face tracking and camera pose estimation. The post offers a detailed technical walkthrough plus reflections on generative models as tools of understanding and the shift from discriminative to generative paradigms in computer vision.

ProxyPose: When Video Diffusion Models Become the Stunt Double for Visual Tracking

Paper Info

Title: ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation
Authors: Ruihang Zhang, Felix Taubner, Pooja Ravi
arXiv: 2607.06555
Field: Computer Vision / 6-DoF Pose Tracking / Video Diffusion Models

The Core Idea

Recovering the 3D pose (6-DoF: 3 rotation + 3 translation) of an object from monocular video is one of the hardest problems in computer vision, plagued by depth ambiguity, occlusion, textureless surfaces, reflections, and deformation.

ProxyPose's key insight is to reformulate the problem: rather than estimating pose directly, translate the video into a different video where pose estimation is easy.

1. Input: a real-world video plus one marked pixel on the target surface in the first frame. 2. Video translation: a video diffusion model generates a "proxy video" — a simple, colored polyhedron that moves exactly like the real surface region. 3. Pose estimation: since the polyhedron's geometry and colors are known, recovering its 6-DoF trajectory is a classical PnP (Perspective-n-Point) problem, solvable with fast, well-established methods. 4. Output: the 6-DoF trajectory of the polyhedron — equivalent to the trajectory of the real surface.

Why It Works

The hardest part is absorbed by the translation step. Video diffusion models (e.g., Sora, CogVideo, Wan), trained on billions of video clips, have internalized how objects move, how occlusions and reflections behave, and how rigid and deformable bodies act. Like an experienced animator drawing a stick figure that faithfully mirrors a live actor's motion, the model expresses its understanding of the input as the motion of a simple, known proxy.

Training: Fully Synthetic

Random 3D scenes are generated with varied objects and materials; RGB videos are rendered as inputs, and rendered proxy videos (a colored polyhedron following a surface region) serve as targets.
No real-world annotations are needed — fine-tuning uses purely synthetic data.

Post-Processing

1. Identify polyhedron faces per frame via color. 2. Establish 2D–3D correspondences with the known geometry. 3. Solve for rotation and translation (least squares / RANSAC). 4. Apply temporal smoothing across frames.

This classical pipeline requires no deep learning, and is fast and accurate.

Key Advantages

Minimal inputs: only a video and one initial pixel; no 3D model, depth map, mask, or hand-crafted features.
Challenging materials: handles reflective, transparent, and textureless surfaces thanks to the diffusion model's pre-trained world knowledge.
Pixel-level operation: no assumptions about object identity, boundaries, or global rigidity.
Generalization: synthetic-only training transfers well to real-world data.

Results

State-of-the-art 6-DoF tracking accuracy across multiple benchmarks, outperforming competitors that require extra inputs.
Strong performance on non-rigid face tracking, camera pose estimation, and in-the-wild videos.
Ablations show that colored polyhedra outperform single-color spheres (more features for PnP), the framework works with different base diffusion models, and more synthetic data improves generalization.

Deeper Takeaways

Generation as understanding: instead of regressing pose, the model generates an equivalent proxy scene — the generation process itself embodies its understanding of the input.
Proxy representations: a long tradition (point masses in physics, ball-and-stick molecular models, proxy geometry in graphics) — keep the essential information, discard irrelevant detail. Here, the essential thing is the motion pattern, not the object's appearance.
Discriminative → generative shift: generative models serve as intermediate representations, leveraging large-scale pre-training and producing interpretable, visualizable intermediates.
Future directions: motion capture from ordinary video, physical property estimation, proxy-assisted 3D reconstruction, and anomaly detection via proxy–real video deviations.

Conclusion

ProxyPose doesn't try to escape Plato's cave; it makes the shadows themselves clearer and more useful. When a video diffusion model truly "understands" motion, it can express that understanding in any simple form — even a colored polyhedron, a borrowed shadow that reveals the essence of real motion.

References

Zhang, R., Taubner, F., & Ravi, P. "ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation." arXiv:2607.06555, 2026.
Lepetit, V., et al. "EPnP: An Accurate O(n) Solution to the PnP Problem." IJCV, 2009.
Ho, J., et al. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020.
Wan, T.H., et al. "Wan: Open World Video Generation." 2025.
Brooks, T., et al. "Video Generation Models as World Simulators." OpenAI, 2024.