Overview
- Field: NLP
- Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah
- Published: 2026-08-11
- arXiv: 2508.03806
- Automated TTS evaluation methods (MOS predictors and Audio-LLM judges) are expected to reflect human perception, but how well they capture the distinct aspects of speech listeners actually perceive remains unclear.
- The authors deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions.
- Using this schema, they construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters.
- Benchmarking four MOS predictors and four Audio-LLM judges reveals that MOS predictors collapse onto acoustic signal quality.
- Audio-LLM judges show selective, prompt-dependent detection that does not generalize across all dimensions.
- Neither family of methods reliably captures a broad range of linguistically structured speech errors.
- The dataset, annotation framework, and evaluation code are publicly released to enable more targeted and interpretable TTS evaluation.
Key points
Original abstract (excerpt)
> Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dim...
---
*Auto-collected on 2026-08-12*