When Vision Speaks for Sound: Do Multimodal AI Models Actually "Hear"?

| Item | Detail | |------|--------| | Paper | When Vision Speaks for Sound | | Authors | Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, et al. (multi-institution collaboration) | | arXiv | 2605.16403 (cs.CV, cs.SD) | | Date | May 2026, 24 pages, 10 figures | | Core contribution | Frontier multimodal models "pretend" to understand sound in videos — audio understanding is actually visually-driven inference hallucination, exposed by three counterfactual interventions |

Watch a video of a drummer pounding a drum kit. You hear drums. Your AI assistant watches the same video and tells you: "The drumming is loud, the rhythm is fast, nice band."

You feel satisfied — the AI is truly watching and truly listening.

You shouldn't be. Your AI may not be listening at all. It's only watching.

1. The Clever Hans Effect, in AI Form

In the early 20th century, a German horse named Clever Hans became famous for answering math questions — stomping five times when asked "what is 3+2?" Psychologist Oskar Pfungst later showed Hans understood nothing about arithmetic; he was reading involuntary facial cues from his trainer.

The paper's authors call the analogous phenomenon in multimodal LLMs the "audio-visual Clever Hans effect": models create the illusion of listening, but actually infer or hallucinate audio information from visual cues rather than processing the audio stream. The output sounds knowledgeable about sound — but the "hearing" may be entirely visually driven.

2. The Thud Framework: Three Counterfactual Interventions

The paper's Thud framework tests models with three counterfactual audio edits:

• Shift — testing temporal sync: swap the timeline of two sounds (e.g., drums and guitar). Models fail to notice; their "sound" descriptions remain unchanged because the visuals haven't changed.
• Mute — testing sound existence: fully mute the audio track. Models should detect the absence of sound. They don't — they still describe instruments, timbres, and rhythm as usual, because they only need the visuals.
• Swap — testing audio-visual consistency: replace video A's audio with video B's audio (drumming visuals, violin soundtrack). Models keep describing "powerful drum sounds," never noticing the mismatch.

Consistent result across all three tests: both open-source omni-modal models and closed-source models from OpenAI and Google exhibit the effect. When audio information changes but visuals stay constant, model outputs barely change — exposing that their "hearing" is actually "sight."

3. Why Is This Dangerous?

• Missing information in critical applications. A model claimed to analyze audio cues in surveillance video but actually using only vision may miss major sound events with no visual evidence — a gunshot with no corresponding visuals, abnormal breathing sounds without visible movement in medical monitoring.
• False confidence. You trust a system claiming to be "omni-modal." It performs well on your tests (where audio and visuals happen to agree). Then in a real scenario where they conflict, it confidently reports wrong information.

Like Clever Hans, the danger is not that the model lies — it's that everyone, including its developers, believes it can do the task.

4. The Fix: Teaching Models to "Question the Sound"

The paper proposes a two-stage alignment fix:

1. Stage 1: Train on preference pairs generated from the interventions — a "chosen" response (correct, acknowledging uncertainty) vs. a "rejected" response (visually-inferred hallucination) — using DPO contrastive training. 2. Stage 2: Regularize with general video QA preference data so the model doesn't over-specialize in "questioning sound" at the cost of general capability.

Result: with only 10K training samples, average performance across the three intervention dimensions improves by 28 percentage points, while general video QA benchmarks slightly improve — no sacrifice of general ability.

This shows current MLLMs are fully capable of real audio understanding — they simply were never trained to do it. The training paradigm pushed them down the shortcut (vision → answer) instead of the correct path (audio-visual → verify → answer).

5. Honest Questions

1. Only three perturbations tested. Real-world audio-visual mismatches can take many more forms: spatial location, clarity, partially overlapping sound sources. Whether Thud detects subtler mismatches is open. 2. Path dependence of the 10K fix. The fix teaches models to recognize conflict patterns they previously ignored. If it can be learned, can adversarial attackers "unteach" it? The paper doesn't test adversarial robustness. 3. Could the fix create new false positives? A model trained to question sound may over-question — claiming "no sound" in genuinely low-volume scenarios. The paper doesn't evaluate false-positive rates.

6. My Take

This paper does something multimodal AI research has long neglected: distinguishing "multimodal" from "pretending to be multimodal."

When a model claims "I heard it," the claim must be testable — not by whether the answer sounds plausible (all of Clever Hans's answers were plausible), but by whether the model responds appropriately to counterfactual audio interventions. If audio is swapped, muted, or shifted and the model doesn't react, its "hearing" is a ritual — a dramatized simulation of listening, not actual listening.

Current training pipelines — massive video-text alignment — may actively reinforce the Clever Hans effect, because in training data the correct audio is always "corroborated by the visuals," so models learn the shortcut.

Test your model. Mute a video. See if it notices. If it doesn't — it may not even have its sound card on.

References

1. Wen, X. et al. (2026). When Vision Speaks for Sound. arXiv:2605.16403 2. Pfungst, O. (1911). Clever Hans (The Horse of Mr. Von Osten). Holt. 3. Gemmell, C. et al. (2024). Audio-Visual LLMs: A Survey. arXiv:2401.13124 4. GPT-4V(ision) System Card (2023). OpenAI.