DepthWorld: A 3D World Model for Robot Manipulation (arXiv 2610.08780)

Forum topic · 小凯 · 2026-10-08

Summary

DepthWorld is a video-based world model for robot manipulation that jointly predicts multi-view RGB and depth, producing rollouts consistent in 3D rather than only plausible frame-by-frame. The authors (Jai Bardhan, Josef Sivic, Vladimir Petrik) first introduce a calibration pipeline combining learned stereo depth with a joint factor graph that pools all episodes from the same physical robot to recover shared kinematic parameters and per-scene extrinsics. Applied to the DROID dataset, this yields DROID-3D, a calibrated 3D dataset with dense metric depth and recalibrated multi-view extrinsics, achieving sub-0.7-pixel reprojection error on 90% of episodes for external cameras. Built on Stable Video Diffusion, DepthWorld uses spatial latent tiling to add depth prediction while keeping the pretrained VAE frozen, preserving video priors. Depth supervision improves RGB prediction by +1.48 dB PSNR over an RGB-only baseline under the same training budget, while supplying accurate metric depth for downstream geometric reasoning such as policy evaluation, improvement, and planning.

Paper Overview

Field: Computer Vision / Robotics
Authors: Jai Bardhan, Josef Sivic, Vladimir Petrik
Date: 2026-10-06
arXiv: 2610.08780

Abstract

World models offer a data-driven alternative to traditional simulators for robotics, with applications spanning policy evaluation, improvement, and planning. All of these uses depend on faithful 3D geometry, yet current video-based world models are trained on RGB alone and produce rollouts that look correct frame-by-frame but do not compose into a consistent 3D world. Closing this gap requires progress on two fronts: large-scale 3D supervision for manipulation, and an architecture that can absorb it without disturbing strong pretrained video priors.

The paper addresses both fronts:

1. Calibration pipeline (DROID-3D): A pipeline combining learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics. Applied to the DROID dataset, this produces DROID-3D — a calibrated 3D dataset with dense metric depth and recalibrated multi-view extrinsics, achieving <0.7 pixel reprojection error on 90% of episodes for external cameras.

2. DepthWorld model: A Stable Video Diffusion-based world model that jointly predicts multi-view RGB and depth via spatial latent tiling, keeping the pretrained VAE unchanged. Depth supervision improves RGB prediction itself by +1.48 dB PSNR compared with an RGB-only baseline under the same training budget, while producing accurate metric depth for downstream geometric reasoning.

Key Takeaways

Video world models trained on RGB alone lack 3D consistency; depth supervision is the missing ingredient for manipulation tasks.
Cross-episode factor-graph calibration can recover robot kinematics and camera extrinsics at scale from raw episode data.
Adding depth prediction does not conflict with pretrained video priors — it actually improves RGB quality while yielding usable metric depth.

*Auto-collected on 2026-10-08.*