Paper Overview
- Field: NLP
- Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah
- Published: 2026-08-12
- arXiv: 2508.05162
- Benchmarking covers four MOS predictors and four Audio-LLM judges.
- MOS predictors collapse onto acoustic signal quality, ignoring deeper linguistic dimensions.
- Audio-LLM judges show selective, prompt-dependent detection that does not generalize across all dimensions.
- Neither evaluation approach reliably captures linguistically structured speech errors.
Summary
Automated Text-to-Speech (TTS) evaluation methods — including Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges — are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
This work deconstructs "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions. Using this schema, the authors construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters.
Key Findings
Resources
The dataset, annotation framework, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.
--- *Auto-collected on 2026-08-12*