LeWAM: A JEPA World Action Model with Diffusion-Steering-Based MPC

Forum topic · 小凯 · 2026-10-11

Summary

LeWAM is a new world action model introduced by Shashank Hegde, Alexander Popov, Elie Aljalbout, and Nikolai Smolyanskiy in an arXiv paper (2610.12407, published 2026-10-08). World action models (WAMs) predict both actions and future observations, but conventional reconstruction-based representations carry noisy, redundant information that complicates downstream prediction. LeWAM instead uses a decoder-free JEPA latent space trained end-to-end, with a bidirectional transformer handling forward, backward, inverse dynamics, and policy prediction jointly. The authors report three benefits: (1) Alignment - linear probes read robot and object state from LeWAM's latent better than from a forward-only JEPA world model (LeWM), while ignoring visual distractors comparably to LeWM and far better than reconstruction-based WAMs; (2) Acting - closed-loop performance matches a flow-matching policy trained on the same encoder at equal scale, while additionally providing a world model; (3) Planning - instead of sampling raw actions for MPC, planning in the noise space of the policy head improves closed-loop performance by avoiding exploitation of dynamics-model inaccuracies.

Paper Overview

Field: Computer Vision (CV) Authors: Shashank Hegde, Alexander Popov, Elie Aljalbout, Nikolai Smolyanskiy Published: 2026-10-08 arXiv: 2610.12407

Abstract (Original)

World action models (WAMs) predict actions and future observations, typically from a reconstruction-based representation that carries noisy, redundant information which can complicate downstream predictions. We introduce LeWAM, a bidirectional transformer for forward, backward, inverse dynamics and policy prediction, on a decoder-free JEPA latent trained end-to-end through all four modes. We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at equal scale, while additionally providing a world model. 3) Planning: When planning with WAMs, sampling raw actions lets MPC exploit inaccuracies of the dynamics model; instead, planning in the noise space of the policy head improves closed-loop performance of these WAMs.

Key Takeaways

Architecture: LeWAM is a bidirectional transformer covering four prediction modes (forward, backward, inverse dynamics, policy) over a decoder-free JEPA latent, trained end-to-end across all modes.
Representation quality: Latents encode robot/object state more linearly readable than a forward-only JEPA world model (LeWM), and ignore visual distractors comparably to LeWM and far better than reconstruction-based WAMs.
Control performance: Matches a regular flow-matching policy trained on the same encoder, while also offering a world model.
Planning insight: MPC over raw actions can exploit dynamics-model inaccuracies; steering via the policy head's noise space (diffusion-steering-based MPC) yields better closed-loop results.

*Auto-collected on 2026-10-11.*