English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

SWAM: Scaling World Action Models with Cross-Modal Alignment

Forum topic · 小凯 · 2026-11-10

Summary

SWAM is a research paper (arXiv:2611.12870) studying how to train World Action Models (WAMs)—models that jointly predict video and actions conditioned on language—without degrading the language-video alignment acquired during pretraining. The authors curate roughly 47,000 hours of general video, egocentric video, UMI demonstrations, and robot demonstrations to systematically evaluate how language-video supervision, language-video-action supervision, and their combination affect WAM performance, along with the impact of annotation quality, joint training schemes, and model architecture. With a fixed training budget, mixing egocentric video with UMI demonstrations improves control generalization; architecture affects sample efficiency; and scaling pretraining with the chosen configuration yields further gains. After post-training only on RoboTwin Clean demonstrations, the model reaches 88.10% and 61.56% success rates under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, and achieves 40.00% success on three real-world tasks versus 23.33% for the strongest baseline, showing stronger generalization to unseen conditions. Data, training code, and model weights are publicly released.

Paper Overview

Research Area: Computer Vision Authors: Yu Zhou, Xinyuan Xia, Xuankun Yang, Yinghao Yang, Wendi Chen Published: 2026-11-09 arXiv: 2611.12870

Abstract (translated)

World Action Models (WAMs) jointly predict video and actions conditioned on language. Prior work has focused on architectures and training mechanisms, often treating language-video alignment obtained through video pretraining as a given foundation. However, introducing action learning can degrade this alignment—and there is still no principled guidance on the data and supervision design needed to maintain language-video alignment during action learning.

To investigate this in depth, the authors curate a data pool of approximately 47,000 hours of general video, egocentric video, UMI demonstrations, and robot demonstrations. On this basis, they systematically study how language-video supervision, language-video-action supervision, and their combination affect WAM performance. Specifically, they examine how the quality of language annotations affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends those capabilities to language-video-action modeling.

To better scale the model, they further study joint training schemes and model architecture in detail. Under a fixed training budget:

  • Mixing egocentric video with UMI demonstrations improves control generalization.
  • Architecture affects the efficiency with which the model exploits experience.
  • With the chosen configuration, scaling pretraining brings further gains.
  • The final recipe is post-trained only on RoboTwin Clean demonstrations, achieving 88.10% and 61.56% success rates under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. On three real-world tasks it reaches 40.00% success rate (strongest baseline: 23.33%), demonstrating stronger generalization to unseen conditions. Data, training code, and model weights have all been released.

    Original Abstract

    We curate approximately 47,000 hours of videos and demonstrations to systematically study how different supervision forms affect World Action Model performance. Our recipe achieves 88.10% and 61.56% success rates under Easy and Hard conditions on RoboTwin Clean, and 40.00% SR across three real-world tasks.

    Links

  • arXiv: https://arxiv.org/abs/2611.12870

Tags

#world-action-models#robotics#computer-vision#video-pretraining#cross-modal-alignment#umi#embodied-ai#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637018