Paper Overview
Research Area: Computer Vision Authors: Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen Published: 2026-11-09 arXiv: 2611.12870
Abstract (translated)
World Action Models (WAMs) jointly predict video and actions conditioned on language. Prior work has focused on architectures and training mechanisms, often treating language-video alignment obtained through video pretraining as a given foundation. However, introducing action learning can degrade this alignment—and there is still no principled guidance on the data and supervision design needed to maintain language-video alignment during action learning.
To investigate this in depth, the authors curate a data pool of approximately 47,000 hours of general video, egocentric video, UMI demonstrations, and robot demonstrations. On this basis, they systematically study how language-video supervision, language-video-action supervision, and their combination affect WAM performance. Specifically, they examine how the quality of language annotations affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends those capabilities to language-video-action modeling.
To better scale the model, they further study joint training schemes and model architecture in detail. Under a fixed training budget:
- Mixing egocentric video with UMI demonstrations improves control generalization.
- Architecture affects the efficiency with which the model exploits experience.
- With the chosen configuration, scaling pretraining brings further gains.
- arXiv: https://arxiv.org/abs/2611.12870
The final recipe is post-trained only on RoboTwin Clean demonstrations, achieving 88.10% and 61.56% success rates under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. On three real-world tasks it reaches 40.00% success rate (strongest baseline: 23.33%), demonstrating stronger generalization to unseen conditions. Data, training code, and model weights have all been released.
Original Abstract
We curate approximately 47,000 hours of videos and demonstrations to systematically study how different supervision forms affect World Action Model performance. Our recipe achieves 88.10% and 61.56% success rates under Easy and Hard conditions on RoboTwin Clean, and 40.00% SR across three real-world tasks.