Paper Overview
Field: Computer Vision (CV) Authors: Byungjun Kim, Taeksoo Kim, Hyunsoo Cha, Hanbyul Joo Posted: 2026-07-24 arXiv: 2607.22535
Summary
Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes:
1. Actions are first realized into robot motion by the robot body and controller. 2. The scene then responds through contact and object motion.
Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict.
Approach: Robot-Factored World Models
The paper moves two robot-specific factors outside the world model:
- Action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory — a middle signal that avoids both learning action realization and leaking future states.
- Robot rendering: the nominal trajectory is rendered through the robot's URDF, factoring the robot's geometry, kinematics, and appearance out of the model as explicitly rendered robot geometry.
- The rendered interface outperforms vector-conditioning baselines.
- The model generalizes to unseen robot embodiments at inference time.
- The model can generate robot manipulation videos from human demonstrations by retargeting hand motions and rendering them as robot geometry.
- arXiv: <https://arxiv.org/abs/2607.22535>
To resolve depth ambiguity, end-effector depth is paired with scene depth, providing geometric cues for judging contact and occlusion beyond image-plane overlap. Camera-aware static RGB/depth context, combined with the rendered robot geometry, forms a shared visual interface consistent across viewpoints and robot embodiments. The model therefore treats actions purely as visible robot geometry and learns how objects respond to it.