SonoWorld4D: From One Video to a 4D Audio-Visual Scene via Diffusion — arXiv Paper Overview

Forum topic · 小凯 · 2026-10-30

Summary

This forum post collects abstracts from an arXiv paper (arXiv:2610.40640) by Derong Jin, Xiyi Chen, Ming C. Lin, and Ruohan Gao. The original abstract introduces Video-to-4DAVScene, a new task that converts a monocular video into an explorable, time-varying 4D audio-visual scene, together with the SonoWorld4D framework: it generates a 360-degree static background, reconstructs motion trajectories of dynamic sound-emitting foreground objects, and synthesizes a physically grounded sound field consistent across multiple sources via dual-space audio diffusion synchronization. The post's Chinese summary additionally describes SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length by expanding each spatial anchor into slots assigned to disjoint DCT frequency bands. Only the lowest-frequency slot participates in global attention while all slots interact locally. On ImageNet-256 it matches FLOPs-matched JiT baselines with faster convergence; on ImageNet-512 it improves FID by 2.84 with 3.9% fewer FLOPs; on text-to-image it beats JiT-B/8 on DPG-Bench by 1.06 at 40.6% lower forward cost.

Paper Overview

Field: Computer Vision (CV)
Authors: Derong Jin, Xiyi Chen, Ming C. Lin, Ruohan Gao
Posted: 2026-10-29
arXiv: 2610.40640

Original Abstract (SonoWorld4D)

Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and present the SonoWorld4D framework for this task. Given a monocular video, our method generates a 360 static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time rendering of synchronized audio and visuals.

Chinese Summary (SpecSlot Method)

Pixel-space generative Transformers typically use one token per image patch, so capturing finer details requires more tokens, making global attention increasingly expensive. The authors observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally.

Based on this insight, they propose SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from global attention sequence length. SpecSlot expands each spatial anchor into multiple slots, with each slot assigned a disjoint band of DCT coefficients, separating anchor density from spectral subdivision. Only the lowest-frequency slot of each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds local corrections to spectral predictions, while generation state, targets, and losses remain in pixel space.

Results

ImageNet-256: matches the FLOPs of the JiT baseline with comparable final FID and faster convergence.
ImageNet-512: improves FID by 2.84 with 3.9% fewer FLOPs; narrows the FID gap to within 0.69 of JiT-B/16 at 38.2% lower cost.
Text-to-image: exceeds JiT-B/8 by 1.06 on DPG-Bench with 40.6% lower forward cost.

---

*Auto-collected on 2026-10-31. Note: the post combines the SonoWorld4D abstract with a Chinese summary describing the SpecSlot method; both are reproduced as published in the source.*