Summary
This post introduces an arXiv paper (2508.03806) that examines whether automated text-to-speech (TTS) evaluators—Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges—actually capture the distinct aspects of speech quality that human listeners perceive. The authors deconstruct the notion of 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions and use it to build the first dimension-level meta-evaluation benchmark for TTS, consisting of 860 utterances annotated by trained linguist raters. Benchmarking four MOS predictors and four Audio-LLM judges shows that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges exhibit selective, prompt-dependent detection that fails to generalize across all dimensions. Neither approach reliably captures the wide range of linguistically structured speech errors. The authors release the dataset, annotation schema, and evaluation code to support more targeted and interpretable TTS evaluation.
Paper Overview
Field: NLP
Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah
Published: 2026-08-11
arXiv: 2508.03806
Summary
Automated Text-to-Speech (TTS) evaluation methods—Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges—are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.
The authors deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters.
Key Findings
- MOS predictors collapse onto acoustic signal quality, missing linguistically grounded aspects of speech quality.
- Audio-LLM judges show selective, prompt-dependent detection that does not generalize across all dimensions.
- Neither approach reliably captures the wide range of linguistically structured speech errors.
Resources
The dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.
arXiv link: https://arxiv.org/abs/2508.03806
---
*Auto-collected on 2026-08-12*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178633334