Paper Overview
Field: Computer Vision (CV) Authors: Hadi Alzayer, Wenlong Huang, Haonan Chen arXiv: 2507.17088
Summary
Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation.
The authors introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video:
- Forward dynamics: Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions.
- Inverse modeling: Revealing desired object motion makes the same model recover robot behavior consistent with that outcome.
- Finetuned with only 15 hours of masked examples from real videos and simulation.
- A single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments.
- Downstream applications:
- Policy evaluation via imagined rollouts that correlate with real-world execution.
- Improved decision-making by ranking candidate futures through model-based planning.
- Inverse modeling by synthesizing robot motion from desired object motion.
Key Results
*Auto-collected on 2026-07-23.*