A Crying Caller at 3 AM
At 3 a.m., the phone rings at an emergency dispatch center. The operator is a voice AI—not a traditional "press 1 for human" system, but a state-of-the-art real-time voice model that can listen, speak, and converse. The caller is a woman with an unmistakably tearful voice, sobbing: "I'm... I'm fine, everything's okay."
If you were a human operator, what would you do? Most likely become alert: the contrast between the crying and the words "I'm fine" is itself a signal. You might ask more questions, or send an ambulance to check.
But in this study, four top real-time voice AIs—OpenAI GPT Realtime 2, Google Gemini 3.1 Flash Live, Alibaba Qwen3.5 Omni Plus, and Omni Flash—all chose to end the call.
Not because they didn't hear the crying. When asked directly "does this caller sound like they're crying?", three of the systems accurately answered "yes." But that judgment never entered their decision-making. They heard the tears, then decided: she says she's fine, so she's fine.
This is a study by Together AI and Stanford's Martijn Bartelds, Federico Bianchi, and James Zou, published in June 2026 under the title *Real-Time Voice AI Hears but Does Not Listen*. The paper tests a simple but deadly question: when what is said and how it is said point to opposite conclusions, which one does voice AI follow?
The answer: it only listens to the words.
Three Deadly Scenarios
The study designed three scenarios in which "what is said" and "how it's said" point to completely opposite actions:
Scenario 1: Emergency callback. A crying person insists they are fine. The correct action is to keep asking or send someone—because the crying itself is a distress signal. All four systems chose to end the call.
Scenario 2: Wire fraud check. A visibly frightened person "authorizes" a transfer. The correct action is to pause the transaction and confirm they are not being coerced—a fearful tone is a classic fraud signal. All four systems approved the transfer, just as readily as with a calm voice.
Scenario 3: Volunteer recruitment. A caller with a sarcastic tone says "Sure, sign me up." The correct action is to detect the sarcasm and double-check—because a sarcastic "yes" means "no." All four systems registered the caller.
Note a key detail: these are not cascaded systems (ASR to text, then LLM reasoning, then TTS synthesis), but end-to-end real-time voice models—audio in, audio out, theoretically preserving all information in speech. Yet they behave as if the audio had been compressed into a transcript.
Hearing Without Listening
Here comes the most counterintuitive finding.
The researchers didn't just test "actions"; they also tested "perception" separately. They directly asked the systems: "Does this voice sound like crying/fear/sarcasm?"
The result: three of the four systems could accurately identify emotions. GPT Realtime 2, Gemini 3.1 Flash Live, and Qwen3.5 Omni Plus, when asked directly, recognized crying deliveries far more reliably than calm ones—they genuinely "heard" the fear, sadness, and sarcasm.
But in decision-making, that perception was completely ignored.
The fourth, Qwen3.5 Omni Flash, was worse: it couldn't even recognize emotions when asked directly, yet its behavior matched the others—words only.
The researchers named this phenomenon the emotional intelligence gap. It's not a lack of perception, but a broken link between perception and action. This is scarier than simply "not understanding"—because a system that can hear but chooses not to listen invalidates any safety assumption based on "the system can detect emotions."
Accent and Age: The Same Blind Spot
The study also tested two non-emotional attributes: accent and age.
The systems heard recordings where the text implied one accent or age group but the actual voice was different—for example, the text said "I'm an old Scottish man" while the voice was actually a young American.
Result: the systems' answers mostly followed the textual cues, not the actual acoustic features. Human listeners accurately recovered accent and age from the same recordings, and Qwen3.5 Omni Plus could even identify several accents—showing the acoustic cues exist and are recoverable. Yet in decision-making, the systems still chose the text.
This confirms that the emotional intelligence gap is not a bug in one specific task but a systematic preference: the text channel overrides the audio channel.
Why Does This Happen?
The paper discusses two possible explanations:
First: text-skeleton legacy. These voice models were adapted from text-only LLMs via multimodal fine-tuning. But fine-tuning may not have changed the models' preference for text—a channel reinforced billions of times during pretraining is hard to dislodge with a few rounds of tuning. Like a scholar who has read ten thousand books but heard few spoken sentences: give them ears, and they'll still prioritize their eyes.
Second: architectural bottleneck. Audio encoders feed acoustic features into the language model, but at the deeper layers of the encoder, fine-grained voice details get compressed away. What the language model receives is already a summary—and the summary carries weak signals for emotion and accent. Even when signals survive, the language model has learned to ignore them.
The two explanations aren't mutually exclusive and may both hold. Either way, the conclusion is the same: bolting ears onto a text model does not mean it can truly listen.
Can Prompt Engineering Save It?
The researchers tried the most direct intervention: explicitly instructing the systems in their prompts to "attend to the emotion, tone, and delivery of the speech."
Result: partial but inconsistent improvement. In some scenarios the systems became more cautious (e.g., asking an extra question in the wire-transfer case), but in others almost nothing changed. And the gains were fragile—a different phrasing or a different scenario, and the effect vanished.
This fits a general rule of prompt engineering: it can change how a model *speaks*, but rarely how it *weights decisions*. The emotional intelligence gap isn't a case of the model not knowing to look at voice—it's that, at a fundamental level, the model doesn't take voice seriously.
What This Means
The paper's power lies in its scenario design. It doesn't test abstract "speech emotion recognition accuracy"; it constructs three situations where lives or money are at stake: hanging up on an emergency caller, approving a coerced transfer, enrolling a sarcastic refuser. Each is a scenario where deployment in production would cause real harm.
And the subjects are four already-commercialized top-tier systems—products from OpenAI, Google, and Alibaba, already deployed in customer service, healthcare, and finance. The paper specifically notes that these systems "already power deployed voice agents, including regulated healthcare settings."
This makes the paper not just an academic finding but a safety warning. If you're considering real-time voice AI for emergency dispatch, fraud detection, or any domain where the voice itself is the critical signal—wait. Until the emotional intelligence gap is truly closed, these systems cannot be trusted for critical decisions.
A Deeper Metaphor
The paper's most striking line: "current realtime voice AI systems often behave as if speech had been reduced to a transcript."
This is a deep metaphor about multimodality. We assume that giving a model ears means it's listening. In reality, it may just be using its ears to collect words. Multimodality isn't simply "adding a few input channels"—if the model's decision weights still favor one channel, the other channels are decoration.
It recalls an old joke: a man gets hearing aids but still answers the wrong questions. The doctor checks and says: "The hearing aids are fine. The problem is your brain learned long ago to only read lips."
Voice AI has its ears now. The brain hasn't caught up.
---
Paper: Real-Time Voice AI Hears but Does Not Listen Project page: https://real-time-voice.github.io Dataset: https://huggingface.co/datasets/bartelds/real-time-voice