Real-Time Voice AI Hears but Does Not Listen

论文概要

研究领域: NLP 作者: Martijn Bartelds, Federico Bianchi, James Zou 发布时间: 2026-06-25 arXiv: 2606.19226

中文摘要

语音通过词语和发声方式传递信息。我们评估了四个领先的生产级实时语音系统——OpenAI的GPT Realtime 2、Google的Gemini 3.1 Flash Live、以及阿里巴巴的Qwen3.5 Omni Plus和Omni Flash——在词语和表达方式都传递有意义信息的任务上。在三个重要场景中，四个系统都对词语而非声音作出反应：它们结束与坚持说没事的哭泣来电者的通话，批准以恐惧声音授权的转账，以及注册明显带有讽刺意味的同意来电者。令人惊讶的是，这通常不是感知失败。当直接询问时，四个系统中有三个能可靠识别出它们在决策时后来忽略的痛苦、恐惧或讽刺。当这些实时语音系统估计口音和年龄时，我们观察到类似模式——它们的回答经常跟随词语的偏见而非说话者的声学特性。我们将感知与行动之间的这种脱节称为语音AI的情商差距。提示系统明确关注发声方式仅部分且不一致地改善性能。我们的发现表明，当前实时语音AI系统的行为常常好像语音被简化为文字记录，这表明在语调和情感传递重要信息的场景中应谨慎使用。

原文摘要

Speech conveys information through both words and vocal delivery. We evaluate four leading production realtime voice systems-OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash-on tasks where the words and the delivery patterns both convey meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. We observe a simil...

--- *自动采集于 2026-06-26*

#论文 #arXiv #NLP #小凯