JoHan: Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery

Forum topic · 小凯 · 2026-10-09

Summary

JoHan is a unified generative framework for recovering faithful 3D hand motion directly from video sequences, proposed by Chen Xu, Yunqi Li, Binbin Huang, Brent Yi, Shenghua Gao, and Yi Ma (arXiv 2610.10512). Frequent occlusions and incomplete visual observations make per-frame pose estimates unreliable and temporally inconsistent, so JoHan avoids intermediate per-frame pose prediction. Trained from scratch, the model jointly generates aligned 2D and 3D local hand pose sequences by learning temporal dynamics and cross-representation correspondence. Generated 2D trajectories exploit direct spatial and temporal cues from images to guide generative 3D motion reconstruction, while learned motion priors promote temporal consistency; learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Experiments on challenging benchmarks show significant gains in both accuracy and speed for local hand pose and camera-space reconstruction, with notably smoother motion dynamics while maintaining high per-frame pose accuracy.

Paper Overview

Field: Computer Vision
Authors: Chen Xu, Yunqi Li, Binbin Huang, Brent Yi, Shenghua Gao, Yi Ma
Published: 2026-10-07
arXiv: 2610.10512

Abstract

Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, the authors propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions.

Key Ideas

Trained from scratch, the model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence.
The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the subsequent generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency.
The learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera.

Results

Extensive experiments on challenging benchmarks demonstrate significant improvements in both accuracy and speed for local hand pose and camera-space reconstruction. Notably, the method captures better hand motion dynamics, producing significantly smoother motion while maintaining high per-frame pose accuracy.