[论文] Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Lin...

论文概要

研究领域: NLP
作者: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah
发布时间: 2026-08-12
arXiv: 2508.05162

中文摘要

自动语音合成（TTS）评估方法（包括平均意见得分预测器和音频大语言模型评判器）被期望反映人类感知，但它们在多大程度上捕捉了听众实际感知到的语音各个维度，仍不清楚。本文将'自然度'解构为一个基于语言学的标注框架，涵盖10个不同的感知维度，并据此构建了首个维度级别的TTS元评估基准，包含860条由专业语言学家标注的语音样本。对4个MOS预测器和4个音频LLM评判器的基准测试结果显示：MOS预测器仅关注声学信号质量，而音频LLM评判器表现出选择性的、依赖提示词的检测能力，且无法泛化到所有维度。两类方法均无法可靠地捕捉语言学结构化的语音错误。我们公开了数据集、标注框架和评估代码，以支持更具针对性和可解释性的TTS评估。

原文摘要

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dim...

自动采集于 2026-08-12

#论文 #arXiv #NLP #小凯