Paper Overview
Field: AI Authors: Geeyang Tay, Wentao Ma, Jaewon Lee, Yuzhi Tang, Daniel Lee, Weisu Yin, Dongming Shen, Silin Meng, Yi Zhu, Mu Li, Alex Smola Published: 2026-03-26 arXiv: 2603.25727
Summary
Automatic speech recognition (ASR) systems have achieved near-human accuracy on carefully curated benchmarks, yet they still fail in real-world voice agent applications under conditions that current evaluation suites do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of performance degradation.
Key Points
- WildASR benchmark: A multilingual (four-language) diagnostic benchmark sourced entirely from real human speech.
- Three robustness axes: It factorizes ASR robustness into environmental degradation, demographic shift, and linguistic diversity.
- Evaluation results: Seven widely used ASR systems show severe and uneven performance degradation; model robustness does not transfer across languages or conditions.
- Hallucination risk: On partial or degraded input, models often produce plausible content that was never actually spoken — a concrete safety risk for downstream agent behavior.
- Takeaway: Targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production voice agents.
- Tooling: Beyond the benchmark itself, the authors release three analysis tools to help practitioners guide deployment decisions.
Original Abstract (excerpts)
> Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions...
---
*Auto-collected on 2026-03-28.*