Paper Overview
Field: NLP Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah Published: 2026-08-11 arXiv: 2508.03806
Abstract
Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dimensions.
Key Takeaways
- "Naturalness" is decomposed into 10 linguistically grounded perceptual dimensions.
- First dimension-level meta-evaluation benchmark for TTS: 860 utterances annotated by trained linguist raters.
- MOS predictors largely collapse onto acoustic signal quality rather than linguistic perception.
- Audio-LLM judges exhibit selective, prompt-dependent detection that does not generalize across dimensions.
- Neither family of evaluators reliably captures broad linguistically structured speech errors.
- Dataset, annotation schema, and evaluation code are publicly released for more targeted, interpretable TTS evaluation.
*Auto-collected on 2026-08-12*