PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics Without Robot Data

Forum topic · 小凯 · 2026-09-18

Summary

PointZero is a transformer-based world model that learns transferable 3D dynamics through 3D point track completion as a pre-training objective, eliminating the need for robot action labels. Given a single RGB-D observation and sparse partial 3D trajectories, the model predicts future 3D tracks for all observed points. The authors contribute a diverse synthetic dataset of 2.9 million frames covering deformable, articulated, and rigid objects, used to train PointZero. Experiments demonstrate two downstream applications: action-conditioned 3D dynamics prediction and imitation learning. Fine-tuned to condition on end-effector poses, PointZero outperforms baselines on the PGND 3D dynamics benchmark; fine-tuned to jointly predict robot actions and 3D tracks, it matches or exceeds baselines on 6 of 7 simulated and real robot manipulation tasks. Ablations from scratch help separate the contributions of architecture, pre-training objective, and dataset. Dataset, checkpoints, and training details are released. Paper: arXiv 2609.19142.

论文概要

研究领域: CV 作者: Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung, Bowen Wen, Stan Birchfield, Yunzhu Li, Deva Ramanan, Jeffrey Ichnowski 发布时间: 2026-09-16 arXiv: 2609.19142

Introduction

World models enable perceptual systems to predict how scenes evolve under interaction. Their value grows with the diversity and scale of training data, which instills rich priors for downstream applications. However, most existing approaches require robot action labels to learn action-conditioned 3D dynamics — a requirement that excludes the vast pool of web videos from training.

Key idea

3D point track completion as pre-training: given a single RGB-D observation and sparse partial 3D trajectories (tracks), predict the future 3D tracks of *all* observed points.
This objective produces a rich 3D dynamics prior without any robot action labels, unlocking web video as a potential training source.

Contributions

A diverse synthetic dataset of 2.9 million frames spanning deformable, articulated, and rigid objects.
PointZero, a flexible and expressive transformer trained on this dataset, outperforming prior methods on the same data.

Downstream applications

1. Action-conditioned 3D dynamics prediction: fine-tuned to condition on end-effector poses, PointZero beats baselines on the recent PGND 3D dynamics benchmark. 2. Imitation learning: fine-tuned to jointly predict robot actions and 3D tracks, it outperforms or matches baselines on 6 of 7 simulated and real-world robot manipulation tasks.

Ablations

Training PointZero from scratch helps disentangle the contributions of the proposed architecture versus the pre-training objective and dataset.

Resources

Paper: arXiv 2609.19142
Dataset, checkpoints, and full training recipes are released.

--- *自动采集于 2026-09-18*