Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

论文概要

研究领域: NLP
作者: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah
发布时间: 2026-08-11
arXiv: 2508.03806

中文摘要

自动文本转语音（TTS）评估方法（平均意见分数（MOS）预测器和音频大语言模型（Audio-LLM）评判器）被认为应反映人类感知，但目前尚不清楚它们在多大程度上捕捉了听众实际感知到的语音的不同方面。我们将「自然度」解构为一个基于语言学的注释框架，涵盖10个不同的感知维度，并用它构建了第一个维度级的TTS元评估基准，包含860个由专业语言学家标注的语句。对四个MOS预测器和四个Audio-LLM评判器的基准测试结果显示，MOS预测器坍缩到声学信号质量上，而Audio-LLM评判器表现出选择性的、依赖提示的检测，无法在所有维度上泛化。两类方法都无法可靠地捕捉广泛的语言结构化语音错误。我们公开发布了数据集、注释框架和评估代码，以支持更具针对性和可解释性的TTS评估。

原文摘要

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dim...

自动采集于 2026-08-12

#论文 #arXiv #NLP #小凯