Puffin-World: A Unified Multimodal Model for Native 3D World Generation and Reconstruction

Forum topic · 小凯 · 2026-09-05

Summary

Puffin-World is a unified multimodal architecture proposed by Kang Liao, Yihang Luo, and Xiao-Ming Wu (arXiv:2609.04196) that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. The framework jointly models three native world states—physics (gravity fields and latitude), geometry (depth), and appearance (images)—together with a unified Omni-Camera representation that supports diverse tasks and flexible motion. A novel strategy propagates physical dynamics across future frames, and by anchoring absolute camera attributes to the real world, Puffin-World achieves physically consistent and visually stable world generation. Appearance and geometry are coupled in a single generative process, jointly synthesizing each future view while reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications such as imitation and self-calibrating world exploration. For scaling to complex scenes, the authors built Puffin-16M, a dataset with 15 million vision-language-camera triplets and 1 million trajectories covering challenging motions. Code, models, and datasets are released.

Paper Overview

Field: Computer Vision (CV)
Authors: Kang Liao, Yihang Luo, Xiao-Ming Wu
Published: 2026-09-03
arXiv: 2609.04196

Abstract

We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules.

To reliably construct and interact with 3D worlds, the framework jointly models three native world states:

Physics (gravity fields and latitude)
Geometry (depth)
Appearance (images)

along with a unified Omni-Camera representation that supports diverse tasks and flexible motion.

Beyond modeling these states, the authors introduce a strategy for propagating physical dynamics across future frames. By anchoring absolute camera attributes to the real world, Puffin-World achieves physically consistent and visually stable world generation.

Key Contributions

Unified world states: Physics, geometry, and appearance are jointly modeled in a single multimodal architecture.
Omni-Camera representation: A unified camera formulation enabling diverse tasks and flexible motion control.
Physical dynamics propagation: Physics is propagated across future frames for consistency.
Coupled generation and reconstruction: Appearance and geometry are synthesized together in one generative process, producing each future view alongside its underlying 3D geometry.
Interleaved closed-loop applications: The paradigm supports tasks requiring multi-task synergy, including imitation and self-calibrating world exploration.

Dataset: Puffin-16M

To scale Puffin-World to complex scenes, the team built Puffin-16M, consisting of:

15 million vision-language-camera triplets
1 million trajectories with various challenging motions

Resources

Code, models, and the dataset have been publicly released to facilitate further research in this area. See the paper at arXiv:2609.04196.

--- *Auto-collected on 2026-09-05.*