Paper Overview
Field: Computer Vision (CV) Authors: Hyeonwoo Kim, Jeonghwan Kim, Kyungwon Cho arXiv: 2604.20841
Summary
Recent advances in video generative models enable the synthesis of realistic human-object interaction (HOI) videos across a wide range of scenarios and object categories, including complex dexterous manipulations that are difficult to capture with motion capture systems. While the rich interaction knowledge embedded in these synthetic videos holds strong potential for motion planning in dexterous robotic manipulation, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets in physics-based character control.
The authors present DeVI (Dexterous Video Imitation), a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects.
Key Contributions
- Hybrid tracking reward: To overcome the imprecision of generative 2D cues, DeVI integrates 3D human tracking with robust 2D object tracking.
- Zero-shot generalization: Unlike approaches that rely on high-quality 3D kinematic demonstrations, DeVI requires only generated videos to generalize across diverse objects and interaction types.
- Strong empirical results: Extensive experiments show DeVI outperforms existing methods that imitate 3D HOI demonstrations, particularly in modeling dexterous hand-object interactions.
- Broader validation: DeVI is further validated in multi-object scenes and with text-driven action diversity, highlighting the advantage of using videos as HOI-aware motion planners.