Robot-Factored World Models via Robot Rendering

Forum topic · 小凯 · 2026-07-28

Summary

This paper introduces robot-factored world models, a new approach for action-conditioned video world models in robotics. Instead of conditioning directly on raw action commands (which forces the model to learn robot control dynamics) or on logged future states (which leaks prediction outcomes), the method factors out two robot-specific processes. First, action realization: commands are rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory. Second, robot rendering: this trajectory is rendered via the robot's URDF into explicit robot geometry, decoupling the robot's geometry, kinematics, and appearance from the learned world model. To resolve depth ambiguity, end-effector depth is paired with scene depth, giving geometric cues about contact and occlusion. Camera-aware static RGB/depth context combined with rendered robot geometry forms a shared visual interface that stays consistent across viewpoints and embodiments. Experiments show the rendering interface outperforms vector-conditioning baselines and generalizes at inference to unseen robot embodiments. The authors also demonstrate generating robot manipulation videos from human demonstrations by retargeting hand motions and rendering them as robot geometry. Authored by Byungjun Kim, Taeksoo Kim, Hyunsoo Cha, and Hanbyul Joo; arXiv:2607.22535.

Paper Overview

Research area: Computer Vision Authors: Byungjun Kim, Taeksoo Kim, Hyunsoo Cha, Hanbyul Joo Published: 2026-07-24 arXiv: 2607.22535

Abstract

Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into robot motion by the robot body and controller, and the scene then responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict.

The authors propose robot-factored world models, which move two robot-specific factors outside the world model:

1. Action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory — an intermediate signal that avoids both learning action realization and leaking future states. 2. Robot rendering: the nominal trajectory is rendered through the robot's URDF into explicitly rendered robot geometry, decoupling the robot's geometry, kinematics, and appearance from the model.

To resolve depth ambiguity, end-effector depth is paired with scene depth, providing geometric cues for judging contact and occlusion beyond image-plane overlap. Camera-aware static RGB/depth context combined with the rendered robot geometry forms a shared visual world-model interface that remains consistent across viewpoints and robot embodiments. As a result, the model treats actions purely as visible robot geometry and learns how objects respond to it.

Key Findings

The rendering-based interface outperforms vector-conditioning baselines.
The model generalizes at inference time to unseen robot embodiments.
The model can generate robot manipulation videos from human demonstrations by retargeting hand motions and rendering them as robot geometry.

--- *Auto-collected on 2026-07-28.*