DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

Forum topic · 小凯 · 2026-04-24

Summary

DeVI (Dexterous Video Imitation) is a framework by Hyeonwoo Kim, Jeonghwan Kim, and Kyungwon Cho (arXiv:2604.20841) that uses text-conditioned synthetic videos from generative models to control physically plausible dexterous agents interacting with unseen objects. Because generated videos have limited physical fidelity and are purely 2D, they cannot directly serve as imitation targets for physics-based character control. DeVI addresses this with a hybrid tracking reward combining 3D human tracking with robust 2D object tracking, compensating for imprecise generative 2D cues. Unlike methods requiring high-quality 3D kinematic demonstrations, DeVI achieves zero-shot generalization across diverse objects and interaction types using only generated videos. Experiments show it outperforms approaches that imitate 3D human-object interaction demonstrations, particularly in modeling dexterous hand-object interactions, and remains effective in multi-object scenes with text-driven action diversity, positioning synthetic video as a valuable source for HOI-aware motion planning in robotics.

Paper Overview

Field: Computer Vision (CV)
Authors: Hyeonwoo Kim, Jeonghwan Kim, Kyungwon Cho
Published: 2026-04-22
arXiv: 2604.20841

Abstract (Translation)

Recent advances in video generative models enable the synthesis of realistic human-object interaction videos across a wide range of scenarios and object categories, including complex dexterous manipulations that are difficult to capture with motion capture systems. While the rich interaction knowledge embedded in these synthetic videos holds strong potential for motion planning in dexterous robotic manipulation, their limited physical fidelity and purely 2D nature make them difficult to use directly as imitation targets in physics-based character control.

The authors present DeVI (Dexterous Video Imitation), a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects. To overcome the imprecision of generative 2D cues, DeVI introduces a hybrid tracking reward that integrates 3D human tracking with robust 2D object tracking.

Unlike approaches that depend on high-quality 3D kinematic demonstrations, DeVI achieves zero-shot generalization across diverse objects and interaction types using only generated videos. Extensive experiments show that DeVI outperforms existing methods that imitate 3D human-object interaction demonstrations, especially in modeling dexterous hand-object interactions. The authors further validate DeVI in multi-object scenarios and text-driven action diversity, demonstrating the advantage of using video as an HOI-aware motion planner.

Key Contributions

Physically plausible dexterous control driven purely by text-conditioned synthetic videos
Hybrid tracking reward: 3D human tracking + robust 2D object tracking to compensate for imprecise generative 2D cues
Zero-shot generalization to unseen objects and interaction types without 3D motion capture data
Superior performance on dexterous hand-object interaction versus 3D-demonstration-based baselines

---

*Auto-collected on 2026-04-24. Source: arXiv:2604.20841*