English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Beyond Naturalness: Probing Automated TTS Evaluators on Linguistically Grounded Dimensions

Forum topic · 小凯 · 2026-08-11

Summary

This arXiv paper (2508.03806) examines how well automated text-to-speech (TTS) evaluation methods reflect human speech perception. The authors deconstruct the vague notion of 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and build the first dimension-level meta-evaluation benchmark for TTS, consisting of 860 utterances annotated by trained linguist raters. Benchmarking four Mean Opinion Score (MOS) predictors and four Audio-LLM judges reveals significant weaknesses: MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show only selective, prompt-dependent detection that fails to generalize across all dimensions. Neither approach reliably captures broad linguistically structured speech errors. The team releases the dataset, annotation schema, and evaluation code to support more targeted and interpretable TTS evaluation. The work is relevant to NLP and speech synthesis research.

Paper Overview

Field: NLP Authors: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah Published: 2026-08-11 arXiv: 2508.03806

Abstract

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dimensions.

Key Takeaways

  • "Naturalness" is decomposed into 10 linguistically grounded perceptual dimensions.
  • First dimension-level meta-evaluation benchmark for TTS: 860 utterances annotated by trained linguist raters.
  • MOS predictors largely collapse onto acoustic signal quality rather than linguistic perception.
  • Audio-LLM judges exhibit selective, prompt-dependent detection that does not generalize across dimensions.
  • Neither family of evaluators reliably captures broad linguistically structured speech errors.
  • Dataset, annotation schema, and evaluation code are publicly released for more targeted, interpretable TTS evaluation.
---

*Auto-collected on 2026-08-12*

Tags

#tts#speech-synthesis#nlp#audio-llm#evaluation-benchmark#mos-prediction#naturalness

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633347