DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation

Forum topic · 小凯 · 2026-09-11

Summary

DUET-DINO is a simultaneous cross-view latent world model for robot manipulation that jointly learns action-conditioned predictions from static side-camera and wrist-camera observations via cross-view conditioning. By combining global scene context with gripper-centric information, it enables zero-shot, goal-conditioned latent planning over the full 7-DoF end-effector action space, addressing the unreliable fine-grained spatial and rotational predictions of prior single-view latent world models. Across spatially diverse reach tasks (92% success), orientation-intensive angled-reach tasks (72.5%), and multi-goal grasp-and-lift tasks (60.0%), DUET-DINO consistently outperforms single-view and independent dual-view baselines. The model is trained from scratch on the DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. The authors also show that V-JEPA 2 wrist-view predictions underestimate visual dynamics from fine-grained actions, while DINOv3-based predictions better capture action-conditioned scene changes, yielding stronger downstream planning. Code and checkpoints will be open-sourced.

Paper Overview

Field: Computer Vision / Robotics Authors: Nisarga Nilavadi, Ralf Römer, Moritz Reuss, Michael Krawez, Tobias Jülg, Angela P. Schoellig, Rudolf Lioutikov, Wolfram Burgard Published: 2026-09-09 arXiv: 2609.10506

Abstract

Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, the authors introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning.

Key Contributions

Cross-view conditioning: Jointly models static side-view and wrist-view observations, exploiting complementary global scene and gripper-centric information.
Full 7-DoF latent planning: Enables reliable latent planning over the complete 7-DoF end-effector action space, where prior models struggled with fine-grained spatial and rotational actions.
Strong empirical results: Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines:
Reach: 92% success rate
Angled-reach: 72.5% success rate
Lift: 60.0% success rate
Robust generalization: Trained from scratch on the DROID and RoboArena datasets, DUET-DINO generalizes robustly under visual distribution shifts.
Representation comparison: The authors show that V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, while DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning.

The code and model checkpoints will be open-sourced.

Project page: https://utn-air.github.io/DUET-DINO