CLAP: Cross-Embodiment Action-Conditioned Video World Models for Zero-Shot Physical Simulation

Forum topic · 小凯 · 2026-08-30

Summary

CLAP is a cross-embodiment, action-conditioned video generation framework developed by Kechen Liu and Ola Shorinwa (arXiv:2608.27406) that can be trained on internet-scale videos spanning both human and robotic agents. The key insight is that universal physical laws govern spatiotemporal dynamics regardless of the actor, enabling generalizable physics learning from heterogeneous video data. Because action representations vary sharply across robot platforms and are usually missing in human videos, CLAP unifies diverse action spaces using end-effector poses, language instructions, and latent actions. It further introduces a curriculum-based cross-embodiment learning recipe: latent-action learning on unlabeled videos builds foundational physics priors, which are then anchored to end-effector action spaces for zero-shot deployment on real-world tasks. CLAP approaches or exceeds state-of-the-art single-embodiment video models such as DROID on challenging benchmarks.

论文概要

研究领域: CV 作者: Kechen Liu, Ola Shorinwa 发布时间: 2026-08-27 arXiv: 2608.27406

Overview

State-of-the-art action-conditioned video models are typically restricted to a single robot embodiment, preventing them from leveraging the vast corpus of heterogeneous video data that contains rich signals for learning generalizable physics. To bridge this gap, the authors introduce CLAP, a framework for cross-embodiment action-conditioned video generation capable of being trained on diverse, internet-scale videos across human and robotic agents.

CLAP is grounded in the insight that universal physical laws govern spatiotemporal dynamics regardless of the actor. However, cross-embodiment learning is non-trivial because action representations vary sharply across robot platforms and are typically absent in human videos.

Key Contributions

1. Unified action spaces: CLAP harmonizes different action representations using end-effector poses, language instructions, and latent actions. 2. Curriculum-based cross-embodiment learning: To address the respective limitations of each action representation, CLAP first learns foundational physics priors via latent actions across unlabeled video data, then anchors them in the end-effector action space for zero-shot deployment to real-world tasks.

Results

CLAP approaches or exceeds state-of-the-art single-embodiment video models, such as DROID, in challenging environments.