Overview
Field: Computer Vision Authors: Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao Published: 2026-08-22 arXiv: 2608.20308
Abstract (translated)
Egocentric video provides scalable manipulation data for embodied AI, but recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-frame gaps. Existing single-frame and windowed temporal regressors fail when hands briefly leave the frame, while recent video diffusion models (VDMs) rely on heavy stochastic multi-step sampling acting as pixel-space renderers.
This paper repositions the VDM as a deterministic geometric encoder — a single forward pass on clean latents exposes scene content beyond the current observation, including occluded and out-of-frame hands.
The authors propose DreamHand, an offline clip-level framework that extracts features via the deterministic clean-latent encoder and decodes them with a bidirectional spatio-temporal decoder. DreamHand recovers continuous two-hand trajectories with metric placement without external detectors, while a ray-based camera solver supports a second configuration requiring no test-time camera intrinsics.
On five egocentric benchmarks, DreamHand sets a new state of the art, reducing MPJPE-p by 30% on the heavily occluded ARCTIC dataset and by 40% on HOT3D.
Links
- Paper: https://arxiv.org/abs/2608.20308