FlatSounds: Benchmarking Single-Factor Physical Video-to-Audio Generation

Forum topic · 小凯 · 2026-06-01

Summary

This post summarizes an arXiv paper (2605.30339) introducing FlatSounds, a benchmark for auditing physical reasoning in generative video-to-audio (V2A) models. Existing evaluations emphasize perceptual realism but ignore physical correctness under controlled interventions. FlatSounds uses two evaluation settings: controlled counterexample pairs that vary a single physical factor, and single-video pattern tests probing internal consistency and directional trends, checking whether generated audio correctly reflects specific physical attributes and timing. Evaluations of state-of-the-art models reveal a consistent trade-off: models rely more on text captions than visual streams to infer physics and semantics. Captions generally improve physical and semantic accuracy but paradoxically reduce temporal alignment. The authors argue the field should shift from audio quality toward learning physical processes directly from pixels, and show that physics-based metrics correlate strongly with human preference tests on their data. The paper is authored by Tingle Li, Siddharth Gururani, Kevin J. Shih, and colleagues from the CV research area.

Paper Overview

Research Area: Computer Vision (CV) Authors: Tingle Li, Siddharth Gururani, Kevin J. Shih, Gantavya Bhatt, Sang-gil Lee, Zhifeng Kong, Arushi Goel, Gopala Anumanchipalli, Ming-Yu Liu Release Date: 2026-05-28 arXiv: 2605.30339

Abstract (Translated)

Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emphasize perceptual realism and overlook physical correctness under controlled interventions. This paper introduces the FlatSounds benchmark, which audits physical reasoning in V2A models through:

1. Controlled counterfactual pairs where a single physical factor is varied; 2. Single-video pattern tests probing internal consistency and directional trends.

These settings test whether generated audio correctly reflects specific physical properties and their timing. Evaluations of state-of-the-art models reveal a consistent trade-off: models rely more on text captions than on the visual stream to infer physics and semantics. Captions generally improve physical and semantic accuracy, but paradoxically degrade temporal alignment.

The results highlight the need to move from audio quality toward learning physical processes directly from pixels. Furthermore, the authors find that physics-based metrics correlate highly with human preference tests conducted on their own data.

--- *Auto-collected on 2026-06-01*

FlatSounds: Benchmarking Single-Factor Physical Video-to-Audio Generation

Summary

Paper Overview

Abstract (Translated)

Tags