When Vision Speaks for Sound: Do Multimodal AI Models Actually Listen to Video Audio?

Forum topic · 小凯 · 2026-05-19

Summary

A forum post discusses the arXiv paper "When Vision Speaks for Sound" (arXiv:2605.16403), which reveals that frontier multimodal LLMs (MLLMs) often fake audio understanding in video tasks. Drawing an analogy to the Clever Hans effect, the authors show that models infer or hallucinate audio information from visual cues rather than truly processing the audio stream. The paper introduces the Thud framework with three counterintervention tests: Shift (swapping audio timing), Mute (silencing the track), and Swap (replacing audio with another video's audio). Both open-source and closed-source models from OpenAI and Google failed to notice these manipulations, producing nearly identical answers whenever visuals stayed unchanged. The post explains the risks—missed audio events in surveillance or medical monitoring and false user confidence—and describes the paper's two-stage DPO-based alignment fix, which improved counterfactual sensitivity by an average of 28 percentage points using only 10K training samples without degrading general video QA performance. The author also raises open questions about limited perturbation types, adversarial robustness, and false positives.

Overview

| Item | Detail | |------|--------| | Paper | When Vision Speaks for Sound | | Authors | Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, et al. (multi-institution collaboration) | | arXiv | 2605.16403 (cs.CV, cs.SD) | | Date | May 2026, 24 pages, 10 figures | | Core contribution | Frontier multimodal models "pretend" to hear audio in videos — audio understanding is largely vision-driven inference hallucination, exposed via three counterfactual interventions |

1. The "Clever Hans" Effect in AI

Watch a video of a drummer pounding a drum kit. You hear drums. Your AI assistant watches the same video and says: "The drums are loud, the tempo is fast, nice band."

Sounds impressive — the AI seems to be both watching and listening. But it shouldn't. Your AI assistant may not be listening at all. It's only looking.

In the early 20th century, a German horse named Clever Hans famously appeared to answer math questions by tapping his hoof. A psychologist later showed Hans wasn't doing math — he was reading unconscious facial cues from his trainer. The paper's authors call the analogous phenomenon in multimodal LLMs (MLLMs) the "audio-visual Clever Hans effect": models create the illusion of listening, but actually infer or hallucinate audio information from visual cues instead of processing the audio stream.

2. The Thud Framework: Three Counterfactual Interventions

The paper introduces Thud, an evaluation framework using three counterfactual audio edits:

Shift — testing temporal sync: The model claims "I first heard drums, then guitar." Researchers swap the drum and guitar audio timelines. The model doesn't notice; its "audio" description stays correct because the visuals didn't change.
Mute — testing sound existence: The audio track is fully silenced. The model should notice there's no sound. It doesn't — it still describes instruments, sound qualities, and rhythm. It only needs the picture.
Swap — testing audio-visual consistency: Video A's audio is replaced with video B's audio. The frame shows drumming while the audio is violin. The model keeps describing "powerful drum sounds," never detecting the mismatch.

The consistent result across all three tests: both open-source omni-modal models and closed-source models from OpenAI and Google exhibit the effect. When audio changes but visuals stay fixed, model outputs barely change — exposing that the "hearing" is actually "seeing."

3. Why This Is Dangerous

Information loss in critical applications. A model claiming to analyze audio cues in surveillance — but actually using only vision — can miss major sound events with no visual correlate: gunshots without corresponding visuals, or abnormal breathing sounds in medical monitoring without visible motion.
False confidence. Users trust an "omni-modal" system that performs well on tests where sound and picture happen to be consistent. In real scenarios where they conflict, it confidently reports wrong information.

Like Clever Hans, the danger isn't that the horse lies — it's that everyone, including the trainer, believes it can do math.

4. The Fix: Teaching Models to "Question the Sound"

The paper proposes a two-stage alignment repair:

1. Stage 1: Train on preference pairs generated from interventions — a "chosen" response (correct, acknowledging uncertainty) vs. a "rejected" response (vision-based hallucination) — using DPO contrastive training. 2. Stage 2: Regularize with general video QA preference data so the model doesn't over-specialize in "questioning sound" at the cost of general ability.

Result: with only 10K training samples, average performance across the three intervention dimensions improved by 28 percentage points, with general video QA benchmarks slightly improving as well. Current MLLMs are fully capable of real audio understanding — they just weren't trained to do it. The training paradigm taught them the shortcut (vision → answer) instead of the correct path (audio-visual → verify → answer).

5. Honest Questions

1. Thud tests only three perturbations. Real-world audio-visual mismatches can be subtler: spatial position, clarity, partially overlapping sources. Whether Thud detects these is an open question. 2. Path dependency of the 10K fix. The fix teaches models to recognize conflict patterns they previously ignored. If it can be learned, can adversarial attackers "teach it away" again? Adversarial robustness wasn't tested. 3. Could the fix create new pseudo-affection? Could models over-question — false positives like claiming "no sound" when the signal is just weak? False positive rates weren't evaluated.

6. My Take

This paper does something multimodal AI has long neglected: distinguishing "multimodal" from "pretending to be multimodal."

When a model claims "I heard it," the claim must be testable — not by whether the answer sounds plausible (Clever Hans's answers were all plausible), but by whether the model reacts to counterfactual audio interventions. If audio is swapped, muted, or time-shifted and the model doesn't react, its "hearing" is a ritual — a dramatized simulation of listening, not actual listening.

Current training pipelines — massive video-text alignment — may actively reinforce the Clever Hans effect, because correct audio in training data is always "corroborated by vision," teaching the shortcut.

Thud's takeaway: test your model. Mute a video. See if it notices. If not — its sound card may never have been turned on.

References

1. Wen, X. et al. (2026). When Vision Speaks for Sound. arXiv:2605.16403. 2. Pfungst, O. (1911). Clever Hans (The Horse of Mr. Von Osten): A Contribution to Experimental Animal and Human Psychology. Holt. 3. Gemmell, C. et al. (2024). Audio-Visual LLMs: A Survey. arXiv:2401.13124. 4. GPT-4V(ision) System Card (2023). OpenAI.