Puffin-World: Scaling a Unified Multimodal Model for Native 3D World Generation

Forum topic · 小凯 · 2026-09-05

Summary

Puffin-World is a unified multimodal architecture for physical understanding, spatial simulation, and 3D world generation and reconstruction, proposed by Kang Liao, Yihang Luo, and Xiao-Ming Wu (arXiv:2609.04196). Unlike pipelines that rely on external offline modules, the framework jointly models three native world states—physics (gravity and latitude fields), geometry (depth), and appearance (images)—alongside a unified Omni-Camera representation supporting diverse tasks and flexible motion. A strategy propagates physical dynamics across future frames, and by anchoring absolute camera properties in the real world, Puffin-World achieves physically consistent and visually stable world generation. Appearance and geometry are coupled in a single generative process, jointly synthesizing each future view while reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications such as imitation and self-calibrating world exploration. To scale to complex scenes, the authors built Puffin-16M, a dataset of 15 million vision-language-camera triplets and 1 million trajectories with challenging motions. Code, models, and the dataset have been publicly released.

Paper Overview

Field: Computer Vision Authors: Kang Liao, Yihang Luo, Xiao-Ming Wu Published: 2026-09-03 arXiv: 2609.04196

Abstract (Translated)

We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules.

To reliably construct and interact with 3D worlds, the framework jointly models three native world states:

Physics: gravity fields and latitude
Geometry: depth
Appearance: images

Together with a unified Omni-Camera representation that supports diverse tasks and flexible motion.

Key Contributions

Joint modeling of physics, geometry, and appearance states
A strategy for propagating physical dynamics across future frames
Anchoring absolute camera properties in the real world, enabling physically consistent and visually stable world generation
Coupling appearance and geometry within a single generative process: each future view is synthesized jointly with the reconstruction of its underlying geometry
Support for interleaved closed-loop applications requiring cross-task coordination, including imitation and self-calibrating world exploration

Puffin-16M Dataset

To scale Puffin-World to complex scenes, the authors built Puffin-16M, containing:

15 million vision-language-camera triplets
1 million trajectories with various challenging motions

Resources

Code, models, and the dataset have been released to facilitate further research in this area.

---

*Auto-collected on 2026-09-05*