Real-Time Voice AI Hears but Does Not Listen: The Emotional Intelligence Gap

Forum topic · 小凯 · 2026-06-26

Summary

A new arXiv paper (2606.19226) by researchers from Stanford evaluates four leading production real-time voice AI systems—OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash—on tasks where both words and vocal delivery carry meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice: they end calls with crying callers who insist they are fine, approve wire transfers authorized in frightened voices, and enroll callers whose consent is clearly sarcastic. Surprisingly, this is often not a perception failure—three of the four systems reliably identify distress, fear, or sarcasm when asked directly, yet ignore it during decisions. Similar patterns appear when estimating accent and age, where responses follow word-based biases over acoustic cues. The authors call this perception-action disconnect the "emotional intelligence gap" of speech AI. Prompting systems to attend to vocal delivery only partially and inconsistently improves performance, suggesting current voice AI effectively reduces speech to transcripts and should be used cautiously where tone and emotion matter.

Paper Overview

Field: NLP
Authors: Martijn Bartelds, Federico Bianchi, James Zou
Published: 2026-06-25
arXiv: 2606.19226

Abstract (English)

Speech conveys information through both words and vocal delivery. The authors evaluate four leading production real-time voice systems—OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash—on tasks where words and delivery patterns both convey meaningful information.

Key Findings

Across three consequential scenarios, all four systems act on the words rather than the voice:
They end calls with crying callers who insist nothing is wrong.
They approve wire transfers authorized in frightened voices.
They enroll callers whose agreement is clearly sarcastic.
Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions.
A similar pattern appears when the systems estimate accent and age: their answers often follow word-based biases rather than the speaker's acoustic characteristics.
The authors name this disconnect between perception and action the emotional intelligence gap of speech AI.
Prompting systems to explicitly attend to vocal delivery only partially and inconsistently improves performance.

Implications

Current real-time voice AI systems often behave as if speech is reduced to a transcript. The authors caution that these systems should be used carefully in scenarios where tone and emotional delivery carry important information.

---

*Auto-collected on 2026-06-26*