Summary
DreamHand is an offline segment-level framework that repurposes video diffusion models (VDMs) as deterministic geometric encoders for recovering metric 3D hand trajectories from egocentric video. Egocentric videos offer scalable manipulation data for embodied AI, but severe object occlusion and frequent off-frame gaps make 3D hand trajectory recovery difficult. Existing single-frame and windowed temporal regressors fail when hands briefly leave the frame, while recent VDM-based approaches rely on heavy stochastic multi-step sampling as pixel-space renderers. DreamHand instead performs a single forward pass on clean latents to expose scene content beyond current observations, including occluded and off-frame hands. Features are decoded by a bidirectional spatiotemporal decoder, yielding continuous two-hand trajectories with metric placement without external detectors. A ray-based camera solver enables a second configuration requiring no test-time camera intrinsics. DreamHand achieves state-of-the-art results on five egocentric benchmarks, reducing MPJPE-p by 30% on heavily occluded ARCTIC and 40% on HOT3D. arXiv: 2608.20308.
DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Trajectory Recovery
- Field: Computer Vision
- Authors: Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao
- Published: 2026-08-22
- arXiv: 2608.20308
Abstract
Egocentric video provides scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent off-frame gaps. Existing single-frame and windowed temporal regressors fail when hands briefly leave the frame, while recent video diffusion models (VDMs) rely on heavy stochastic multi-step sampling as pixel-space renderers.
This paper repositions VDMs as deterministic geometric encoders — a single forward pass on clean latents exposes scene content beyond the current observation, including occluded and off-frame hands. The proposed DreamHand is an offline segment-level framework that extracts features via the deterministic clean-latent encoder and decodes them with a bidirectional spatiotemporal decoder.
Key Contributions
- Recovers continuous two-hand trajectories with metric placement, without external detectors
- A ray-based camera solver enables a second configuration that requires no test-time camera intrinsics
- State-of-the-art performance on five egocentric benchmarks: MPJPE-p reduced by 30% on heavily occluded ARCTIC and by 40% on HOT3D
---
*Auto-collected on 2026-08-22*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633799