论文概要
研究领域: CV 作者: Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung, Bowen Wen, Stan Birchfield, Yunzhu Li, Deva Ramanan, Jeffrey Ichnowski 发布时间: 2026-09-16 arXiv: 2609.19142
English Translation
World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse volumes of data, to instill a rich prior into downstream applications. Existing methods typically require robot action labels to learn action-conditioned 3D dynamics, which excludes web video data from the training pool.
We study 3D point track completion as a pre-training objective for learning transferable 3D dynamics without robot data. Given a single RGB-D observation and sparse partial 3D trajectories (tracks), we predict future 3D tracks of all observed points. We show this objective produces a rich 3D dynamics prior, without requiring robot action labels.
Key contributions:
- A diverse synthetic dataset of 2.9 million frames spanning deformable, articulated, and rigid objects, used to train PointZero.
- A flexible, expressive transformer that outperforms prior methods on the same data.
- Two downstream applications demonstrating the utility of the pre-training objective: 1. Action-conditioned 3D dynamics prediction — when fine-tuned to condition on end-effector poses, PointZero outperforms baselines on the recent PGND 3D dynamics benchmark. 2. Imitation learning — when fine-tuned to jointly predict robot actions and 3D tracks, PointZero outperforms or matches baselines on 6 of 7 simulated and real-world robot manipulation tasks.
- Ablation studies training PointZero from scratch, to disentangle the contributions of the proposed architecture versus the pre-training objective and dataset.
Source: arXiv:2609.19142