Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Research Field: NLP Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah Published: 2026-08-11 arXiv: 2508.03806
Summary
Automated Text-to-Speech (TTS) evaluation methods—Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges—are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
The authors deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters.
Key Findings
- MOS predictors collapse onto acoustic signal quality. They do not reliably capture broader, linguistically structured speech errors.
- Audio-LLM judges show selective, prompt-dependent detection. Their ability to identify issues does not generalize consistently across all 10 perceptual dimensions.
- Neither class of automated evaluator reliably captures the full range of linguistically structured speech errors that trained human listeners perceive.
- The annotated dataset (860 utterances across 10 perceptual dimensions)
- The linguistically grounded annotation framework/schema
- The evaluation code
- arxiv:2508.03806
- text-to-speech
- tts-evaluation
- mean-opinion-score
- audio-llm
- naturalness
- speech-synthesis
- meta-evaluation
Contributions
The authors publicly release:
These resources are intended to support more targeted and interpretable TTS evaluation going forward.
Original Abstract
> Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dim…