[论文] L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian...

论文概要

研究领域: CV 作者: Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim 发布时间: 2026-10-29 arXiv: 2610.40614

中文摘要

视觉场景生成的最新进展已从静态3D场景扩展到更具挑战性的问题：从最小输入生成动态4D场景。然而，声音——尽管与动态物体和事件本质上耦合——在先前的研究中仍未被建模。我们提出了 Video-to-4DAVScene 这一新任务，将视频转化为可探索的时间变化4D视听场景，并提出了 SonoWorld4D 框架来完成这一任务。给定单目视频，我们的方法生成360度静态背景，重建发声动态物体的显式运动轨迹，并通过双空间音频扩散同步合成与多个声源物理一致的声音场。生成的4D场景支持实时的新视角视觉和空间音频渲染。在真实世界动态场景上的评估和用户研究表明，SonoWorld4D 生成的4D视听场景具有感知真实、物理基础的空间音频，与场景几何和视觉内容保持一致。

原文摘要

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to ge...

#论文 #arXiv #CV #小凯