[论文] WavFlow: Audio Generation in Waveform Space

论文概要

研究领域: CV 作者: Feiyan Zhou, Luyuan Wang, Shoufa Chen 发布时间: 2026-05-19 arXiv: 2505.14308

中文摘要

现代音频生成主要依赖隐空间压缩，引入了额外的复杂性和潜在的信息损失。本文用WavFlow挑战这一范式——一种直接在原始波形空间中生成高保真音频而无需中间表示的框架。为克服建模高维低能量信号的固有困难，我们通过波形分块（waveform patchify）将音频重塑为2D token网格，并引入幅度提升（amplitude lifting）以对齐信号尺度，从而通过流匹配中的直接x预测实现稳定优化。为捕捉复杂的语义对齐和时间同步，我们利用自动化数据管道整理了500万高质量视频-文本-音频三元组，使模型能够从零开始学习细粒度声学模式。实验结果表明，WavFlow在视频到音频基准VGGSound（FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44）和文本到音频基准AudioCaps（FD_PANNs: 10.63, IS_PANNs: 12.62）上达到具有竞争力的性能，与已有的基于隐空间的方法相当甚至超越。我们的工作证明中间压缩并非高质量合成的先决条件，为多模态音频生成提供了更简单、更可扩展的替代方案。

原文摘要

Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained ...

自动采集于 2026-05-20

#论文 #arXiv #CV #小凯