WavFlow: Audio Generation Directly in Waveform Space

Forum topic · 小凯 · 2026-05-20

Summary

WavFlow is a framework that generates high-fidelity audio directly in raw waveform space, challenging the dominant paradigm of latent-space audio generation that adds complexity and potential information loss. The authors address the difficulty of modeling high-dimensional, low-energy signals by reshaping audio into 2D token grids via waveform patchify and introducing amplitude lifting to align signal scales, enabling stable optimization through direct x-prediction in flow matching. To capture semantic alignment and temporal synchronization, an automated pipeline curated 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. WavFlow achieves competitive results on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding latent-based methods. The work shows intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler, more scalable path for multimodal audio generation.

Overview

Research area: CV
Authors: Feiyan Zhou, Luyuan Wang, Shoufa Chen
Published: 2026-05-19
arXiv: 2505.14308

Abstract (translated)

Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations.

To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching.

To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch.

Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or even surpassing existing latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.

Key Results

| Benchmark | Task | Metrics | |---|---|---| | VGGSound | Video-to-audio | FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44 | | AudioCaps | Text-to-audio | FD_PANNs: 10.63, IS_PANNs: 12.62 |

Key Takeaways

Generates audio directly in raw waveform space, avoiding latent compression losses.
Waveform patchify + amplitude lifting enable stable flow matching optimization.
Trained on 5M curated video-text-audio triplets from an automated data pipeline.
Competitive with or better than latent-space approaches on VGGSound and AudioCaps.

--- *Auto-collected on 2026-05-20*