English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Beyond Naturalness: Probing Automated TTS Evaluators with a Linguistically Grounded Benchmark

Forum topic · 小凯 · 2026-08-12

Summary

This paper, posted on zhichai.net and available as arXiv 2508.05162, examines how well automated Text-to-Speech (TTS) evaluation methods reflect what human listeners actually perceive. The authors deconstruct 'naturalness' into a linguistically grounded annotation schema covering 10 distinct perceptual dimensions, and build the first dimension-level meta-evaluation benchmark for TTS, consisting of 860 utterances annotated by trained linguist raters. Benchmarking four Mean Opinion Score (MOS) predictors and four Audio Large Language Model (Audio-LLM) judges shows that MOS predictors essentially collapse onto acoustic signal quality, while Audio-LLM judges exhibit selective, prompt-dependent detection abilities that fail to generalize across all dimensions. Neither approach reliably captures linguistically structured speech errors. The team publicly releases the dataset, annotation framework, and evaluation code to support more targeted and interpretable TTS evaluation. Field: NLP; posted 2026-08-12.

Paper Overview

Field: NLP Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah Posted: 2026-08-12 arXiv: 2508.05162

Summary

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive.

The authors deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters.

Key Findings

  • Benchmarking four MOS predictors and four Audio-LLM judges reveals that MOS predictors collapse onto acoustic signal quality.
  • Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all perceptual dimensions.
  • Neither family of evaluators reliably captures linguistically structured speech errors.
  • The dataset, annotation framework, and evaluation code are publicly released to enable more targeted and interpretable TTS evaluation.
--- *Auto-collected on 2026-08-12*

Tags

#tts#speech-evaluation#nlp#audio-llm#mos-predictors#benchmark#arxiv

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633370