WorldSonus: Real-Time Spatial Sound Generation for World Models

Forum topic · 小凯 · 2026-10-08

Summary

WorldSonus is an interactive video-to-audio framework designed to bring real-time spatial sound to world models, which currently produce largely silent generated environments. The system addresses three core challenges: real-time generation, interactive control, and spatially aligned stereo audio. It uses a streaming causal autoregressive diffusion architecture that synthesizes audio chunks with a low real-time factor (RTF) of 0.41, enabling it to keep pace with interactive video streams. For interactive control, it combines an audio-centric captioning pipeline with chunk-indexed prompt scheduling, allowing dynamic manipulation of sound events during generation. Spatial alignment is achieved through high-quality stereo supervision curated from diverse stereo and ambisonic data, reflecting scene geometry and camera motion. Experiments show that although tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or exceeding state-of-the-art bidirectional models in both acoustic quality and spatial alignment. Paper: arXiv 2610.08760.

Paper Overview

Research Area: Computer Vision (CV) arXiv: 2610.08760

Abstract

Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion.

To address these demands, the authors introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models.

Key Contributions

Real-time generation: WorldSonus employs a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41.
Interactive control: It incorporates an audio-centric captioning pipeline and chunk-indexed prompt scheduling, enabling dynamic manipulation of sound events during generation.
Spatial alignment: The model leverages high-quality stereo supervision curated from diverse stereo and ambisonic data.

Results

Extensive experiments demonstrate that, although tailored for world models, WorldSonus generalizes effectively to open-domain video-to-audio benchmarks, matching or surpassing state-of-the-art bidirectional models in both acoustic quality and spatial alignment.