DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation

Summary

DUET-DINO is a simultaneous cross-view latent world model for robot manipulation introduced by researchers including Nisarga Nilavadi and Rudolf Lioutikov (arXiv:2609.10506). Action-conditioned latent world models enable zero-shot goal-conditioned robot planning, but their predictions of fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. DUET-DINO addresses this by jointly learning action-conditioned predictions from static side-camera and wrist-camera observations through cross-view conditioning, exploiting complementary global scene and gripper-centric information to enable latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, it consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. Trained from scratch on DROID and RoboArena datasets, DUET-DINO generalizes robustly under visual distribution shifts. The authors also show that DINOv3 predictions capture action-conditioned scene changes better than V-JEPA 2, whose wrist-view predictions underestimate fine-grained visual dynamics. Code and checkpoints will be open-sourced.