DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

Forum topic · 小凯 · 2026-04-24

Summary

DeVI (Dexterous Video Imitation) is a framework presented by Hyeonwoo Kim, Jeonghwan Kim, and Kyungwon Cho (arXiv:2604.20841) that uses text-conditioned synthetic videos from generative models to control physically plausible dexterous agents interacting with unseen objects. Because generated videos have limited physical fidelity and are purely 2D, they cannot directly serve as imitation targets for physics-based character control. DeVI addresses this with a hybrid tracking reward that combines 3D human tracking with robust 2D object tracking, compensating for imprecise generative 2D cues. Unlike methods that require high-quality 3D kinematic demonstrations, DeVI achieves zero-shot generalization across diverse objects and interaction types using only generated videos. Experiments show it outperforms prior methods that imitate 3D human-object interaction demonstrations, particularly in modeling dexterous hand-object interactions. The authors also validate effectiveness in multi-object scenarios and text-driven action diversity, demonstrating the value of video as an HOI-aware motion planner for dexterous robotic manipulation.

Paper Overview

Field: Computer Vision (CV)
Authors: Hyeonwoo Kim, Jeonghwan Kim, Kyungwon Cho
Published: 2026-04-22
arXiv: 2604.20841

Key Points

Recent advances in video generative models enable synthesis of realistic human-object interaction (HOI) videos across broad scenarios and object categories, including complex dexterous manipulations that are hard to capture with motion capture systems.
Despite the rich interaction knowledge in these synthetic videos, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets for physics-based character control.
DeVI (Dexterous Video Imitation) leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects.
To overcome the imprecision of generative 2D cues, the authors introduce a hybrid tracking reward that integrates 3D human tracking with robust 2D object tracking.
Unlike prior approaches requiring high-quality 3D kinematic demonstrations, DeVI needs only generated videos, achieving zero-shot generalization across diverse objects and interaction types.

Results

Extensive experiments show DeVI outperforms existing methods that imitate 3D HOI demonstrations, especially in modeling dexterous hand-object interaction.
The framework is further validated in multi-object scenarios and text-driven action diversity, highlighting the advantage of using video as an HOI-aware motion planner.

---

*Auto-collected on 2026-04-24.*