RepWAM: World Action Modeling with Representation Visual-Action Tokenization

Forum topic · 小凯 · 2026-06-14

Summary

RepWAM is a representation-centric world action model (WAM) built on a novel representation visual-action tokenizer, proposed by Junke Wang, Qihang Zhang, and Shuai Yang (arXiv 2506.10666, June 2025). Existing WAMs typically inherit reconstruction-oriented video tokenizers from pretrained video generation models; while these preserve visual fidelity, pixel reconstruction alone offers limited guidance for learning instruction-following dynamics that link future prediction with robot control. To address this, the authors explore a semantic visual-action latent space: they train a tokenizer that maps visual inputs into aligned visual and latent action tokens, then pretrain the WAM to jointly model future visual states and connecting latent actions under language instructions, before adapting it to real robot trajectories for closed-loop manipulation. Experiments on real-world manipulation tasks and simulation benchmarks show strong performance across diverse settings, with ablations highlighting the value of semantic visual-action tokenization over reconstruction-oriented alternatives. Code and weights are planned at github.com/wdrink/RepWAM.

Overview

Field: Computer Vision (CV) Authors: Junke Wang, Qihang Zhang, Shuai Yang Published: 2025-06-13 arXiv: 2506.10666

Abstract

This work presents RepWAM, a representation-centric world action model (WAM) built on representation visual-action tokenizers. Existing WAMs typically inherit reconstruction-oriented video tokenizers from pretrained video generation models. Although these tokenizers preserve visual fidelity, pixel reconstruction alone provides limited guidance for learning instruction-following dynamics that connect future prediction with robot control. To address this, the authors explore a semantic visual-action latent space for representation-centric world action modeling.

Key points

Train a representation visual-action tokenizer that maps visual inputs into aligned visual and latent action tokens.
Pretrain the WAM to jointly model future visual states and the latent actions that connect them, under language instructions.
Adapt the pretrained model to real robot trajectories for closed-loop manipulation.
Experiments on real-world manipulation tasks and simulation benchmarks show strong performance across various manipulation settings.
Ablation studies highlight the value of semantic visual-action tokenization over reconstruction-oriented alternatives.
The results establish representation visual-action tokenization as a promising foundation for world action models, moving toward generalist robot policies.

Code and weights will be available at: https://github.com/wdrink/RepWAM

*Automatically collected on 2026-06-14.*