When AI Mishears You: WildASR Benchmark Exposes Real-World Speech Recognition Failures

Forum topic · 小凯 · 2026-04-21

Summary

A forum post discusses the WildASR benchmark (arXiv 2603.25727), a study from Boson AI that stress-tests seven mainstream speech recognition systems—including Whisper Large V3, Gemini 2.5/3 Pro, GPT-4o Transcribe, Qwen2-Audio, Deepgram Nova 2, and ElevenLabs Scribe V1—using real human speech under realistic degradations: reverberation, far-field distance, telephony codecs, noisy gaps, and clipping, plus speaker diversity (accents, children, elderly) and four languages (English, Chinese, Japanese, Korean). Key findings: no model is robust across all conditions; accented Chinese speech reaches up to 62.5% WER; child speech remains hard even for the best models; robustness does not transfer across languages; and prompt wording alone can swing error rates by 46.1%. Most alarming, degraded or incomplete audio triggers hallucinations—systems fabricate fluent but entirely wrong transcriptions—measured via a new Hallucination Error Rate using GPT-4o-mini as judge. The post highlights the practical 'P90 elbow point' insight: average WER hides catastrophic failures affecting a minority of users, so percentile-based thresholds should trigger fallback strategies. Code (Apache 2.0) and dataset are open-sourced on GitHub and Hugging Face.

When AI Can't Understand Human Speech: An Experiment That Exposes ASR's Weak Spots

> *Back to Basics: Revisiting ASR in the Age of Voice Agents* > Geeyang Tay, Wentao Ma, Jaewon Lee, et al. | arXiv: 2603.25727 | 2026

The Premise

Lab benchmarks show modern speech recognition approaching human-level accuracy—but the real world isn't a lab. Researchers at Boson AI (including Alex Smola) asked a simple question: how unreliable are ASR systems once they leave controlled environments? Their answer, via the WildASR benchmark built entirely from real human speech, is uncomfortable.

Three Stress Dimensions

1. Environmental Degradation

Five realistic corruptions applied to real recordings:

Reverberation: RT60 from 0.4s to 1.6s
Far-field: distances from 4 to 16 meters
Telephony codecs: GSM and G.711
Noise gaps: noise inserted mid-utterance
Clipping: waveform distortion from overloaded recording

2. Demographic Shift

Accented speakers: both English and Chinese accents degrade accuracy significantly; accented Chinese WER peaks at 62.5%
Children: a consistent failure mode—best model Gemini 3 Pro still reaches 18.2% WER on English children's speech
Elderly speakers: better, but still not robust

3. Language Diversity

not

The Most Concerning Finding: Hallucinations

When audio is incomplete or heavily degraded, systems don't just mishear—they fabricate fluent, completely wrong transcriptions. The paper's example: "Captain Lake did not look at all like a London dandy now" became "Will you let Annabel ask her if she sees what it is you hold in your arms again?" For voice agents this is a safety issue: "check my balance" could become "transfer money."

The authors introduce a Hallucination Error Rate (HER) metric, using GPT-4o-mini as an LLM judge to classify each transcript as hallucination error, non-hallucination error, or correct. Hallucination rates rise significantly under degradation and noise gaps.

Seven Systems, No Winner

| Model | Type | |-------|------| | Nova 2 | Deepgram commercial | | GPT-4o Transcribe | OpenAI | | Gemini 2.5 Pro | Google multimodal | | Gemini 3 Pro | Google next-gen | | Qwen2-Audio | Alibaba open-source | | Scribe V1 | ElevenLabs | | Whisper Large V3 | OpenAI open-source |

Each has blind spots: Whisper is excellent on elderly speech (WER 0.2%) but collapses on accents and children; Qwen2-Audio dominates Chinese but lags English; Gemini 3 Pro is balanced overall yet hits 41.4% WER on accented Chinese. No model is reliable across all conditions.

Practical Insight: The P90 Elbow Point

Average WER rises slowly and looks acceptable as degradation increases—but P90 (90th percentile) WER rises much faster. Averages lie: a subset of users experiences catastrophic failure. The paper defines a P90 elbow point—the degradation level where P90 WER starts accelerating—as a practical deployment threshold for triggering fallbacks (text input, prompting for quieter environments).

Open Source

Code: github.com/boson-ai/WildASR-public (Apache 2.0)
Dataset: huggingface.co/datasets/bosonai/WildASR
Paper: arxiv.org/abs/2603.25727

Notable implementation details:

Multilingual WER: three WER standards (Common Voice 17, Qwen2-Audio, Seed-TTS); character error rate (CER) for non-space-delimited languages, with traditional/simplified conversion (OpenCC) and Japanese kana normalization (mozcpy)
Hallucination detection: GPT-4o-mini as judge, carefully distinguishing fabricated content from plain mishearings
Mixed Error Rate (MER): word-level tokenization for English, character-level for CJK, unified for code-switched speech

Takeaways

Benchmark scores have soared, but WildASR shows they may reflect an overly clean, idealized world. The paper's point is not that progress is fake—Whisper, Gemini, and Qwen are genuinely impressive—but that we need honest evaluation: systematic, factor-by-factor exposure of weaknesses. For voice-agent engineers, the paper offers three deployable tools: P90 elbow analysis, cross-condition robustness heatmaps, and hallucination rate detection. When your assistant hears "cancel" as "confirm," users won't care about your benchmark score.

As Feynman said: *"The first principle is that you must not fool yourself—and you are the easiest person to fool."*