DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Trajectory Recovery

Forum topic · 小凯 · 2026-08-22

Summary

DreamHand is a new framework for recovering metric 3D hand trajectories from egocentric video, addressing severe object occlusion and frequent out-of-frame gaps. Instead of using video diffusion models (VDMs) as pixel-space renderers with heavy stochastic multi-step sampling, DreamHand repurposes the VDM as a deterministic geometric encoder: a single forward pass on clean latents reveals scene content beyond current observations, including occluded and off-frame hands. The framework operates offline at the segment level, extracting features with the deterministic clean-latent encoder and decoding them with a bidirectional spatio-temporal decoder. It recovers continuous two-hand trajectories with metric placement without external detectors, and a ray-based camera solver enables a second configuration requiring no test-time camera intrinsics. Across five egocentric benchmarks DreamHand sets new state-of-the-art results, reducing MPJPE-p by 30% on heavily occluded ARCTIC and by 40% on HOT3D. Paper: arXiv 2608.20308.

Paper Overview

Field: Computer Vision Authors: Yufei Liu, Xixi Wang, Hao Li, Ganlong Zhao Published: 2026-08-22 arXiv: 2608.20308

Abstract

Egocentric video provides scalable manipulation data for embodied AI, but recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-frame gaps. Existing single-frame and window-based temporal regressors fail when hands briefly leave the frame, while recent video diffusion models (VDMs) rely on heavy stochastic multi-step sampling as pixel-space renderers.

This paper repositions the VDM as a deterministic geometric encoder — a single forward pass on clean latents exposes scene content beyond the current observation, including occluded and off-frame hands.

Method

DreamHand is an offline, segment-level framework.
Features are extracted by a deterministic clean-latent encoder and decoded by a bidirectional spatio-temporal decoder.
The model recovers continuous two-hand trajectories with metric placement, without external detectors.
A ray-based camera solver supports a second configuration that requires no test-time camera intrinsics.

Results

On five egocentric benchmarks, DreamHand sets new state-of-the-art performance:

30% reduction in MPJPE-p on the heavily occluded ARCTIC dataset.
40% reduction in MPJPE-p on HOT3D.

---

*Auto-collected on 2026-08-22*