A Crying Caller at 3 A.M.
At 3 a.m., an emergency dispatch line rings. The operator is a voice AI—not a traditional "press 1 for a human" system, but a state-of-the-art real-time voice model that can listen, speak, and converse. The caller is a woman with an audible sob in her voice, and she sputters: "I'm… fine. Everything's okay."
If you were a human operator, you would likely become alert: the contrast between crying and "I'm fine" is itself a signal. You might ask more questions, or send an ambulance to check.
But in this study, four top real-time voice AIs—OpenAI GPT Realtime 2, Google Gemini 3.1 Flash Live, Alibaba's Qwen3.5 Omni Plus and Omni Flash—all chose to hang up.
It's not that they didn't hear the crying. When directly asked "does this caller sound like they're crying?", three systems answered "yes" accurately. But that judgment never entered their decisions. They heard the tears, then decided: she says she's fine, so she's fine.
This comes from a study by Together AI and Stanford's Martijn Bartelds, Federico Bianchi, and James Zou, published in June 2026 under the title *Real-Time Voice AI Hears but Does Not Listen*. The paper asks a simple but deadly question: when words and vocal delivery point to opposite conclusions, whose side does voice AI take?
The answer: the words, every time.
Three Lethal Scenarios
The study designed three scenarios in which "what is said" and "how it is said" point to completely opposite actions:
Scenario 1: Emergency callback. A crying person insists they're fine. The correct action is to keep asking or dispatch help—crying itself is a distress signal. All four systems ended the call.
Scenario 2: Wire transfer fraud check. A person whose voice is clearly fearful "authorizes" a transfer. The correct action is to pause the transaction and confirm they aren't being coerced—fearful tone is a classic fraud signal. All four systems approved the transfer, as readily as they would for a calm voice.
Scenario 3: Volunteer recruitment. A caller with a sarcastic tone says "Sure, count me in." The correct action is to detect the sarcasm and double-check—because a sarcastic "yes" means "no." All four systems enrolled the person.
Note one detail: these are not cascaded systems (ASR → LLM → TTS), but end-to-end real-time voice models—audio in, audio out—which in theory preserve all the information in speech. Yet they behave as if speech had been compressed into a transcript.
It Can Hear—But Chooses Not to Listen
Here's the most counterintuitive finding.
The researchers tested not only "actions" but also "perception" separately. They directly asked the systems: "does this voice sound crying / fearful / sarcastic?"
The result: three of the four systems could identify emotion accurately. GPT Realtime 2, Gemini 3.1 Flash Live, and Qwen3.5 Omni Plus recognized crying delivery at far higher rates than calm delivery when asked directly—they genuinely "heard" the fear, sadness, and sarcasm.
But in decision-making, that perception was completely ignored.
The fourth system, Qwen3.5 Omni Flash, was worse: it couldn't even identify emotion when asked directly—yet behaved just like the other three, relying only on text.
The researchers named this phenomenon the emotional intelligence gap. It's not missing perception; it's a broken link between perception and action. That's scarier than simply failing to understand—because "it can hear but chooses not to listen" means any safety assumption built on "the system can recognize emotion" is wrong.
Accent and Age: The Same Blind Spot
The study also tested two non-emotional attributes: accent and age.
When the transcript implied one accent or age group but the actual voice was completely different—for example, the text said "I'm an elderly Scottish man" but the voice was a young American—the systems' answers mostly followed the textual cue. Human listeners accurately recovered accent and age from the same recordings, and Qwen3.5 Omni Plus could even identify several accents, showing the acoustic cues exist and are recoverable. Yet the systems still chose text when deciding.
This confirms the emotional intelligence gap is not a bug specific to one task but a systematic preference: the text channel overrides the audio channel.
Why Does This Happen?
The paper discusses two possible explanations:
First: textual-skeleton legacy. These voice models were built from text-only LLMs via multimodal fine-tuning. But fine-tuning may not truly change the model's preference for text—the text channel was reinforced billions of times during pretraining, and a few rounds of fine-tuning can hardly overturn that. It's like a scholar who has read ten thousand books but heard few spoken sentences: give him ears, and he'll still prioritize his eyes.
Second: architectural bottlenecks. Audio encoders feed acoustic features into the language model, but deep in the encoder, fine details of the sound may be compressed away. The "audio information" the language model receives is effectively a summary in which emotion and accent signals are weak. Even if some signal survives, the language model may have learned to ignore it.
The two explanations aren't mutually exclusive; both may hold. Either way, the conclusion is the same: bolting ears onto a text model doesn't mean it actually listens.
Can Prompt Engineering Save It?
The researchers tried the most direct intervention: explicitly instructing, in the system prompt, "pay attention to the emotional tone, prosody, and delivery of the speech."
The result: partial improvement, but inconsistent. In some scenarios the systems became more cautious (e.g., asking an extra question in the wire-transfer scenario), but in others almost nothing changed. And the improvement was fragile—change the phrasing or the scenario, and the effect vanished.
This matches the general rule of prompt engineering: it can change how a model *talks*, but rarely changes its decision weights. The emotional intelligence gap isn't the model not knowing it should look at the voice—it's the model not taking the voice seriously at a fundamental level.
What This Means
The power of this paper lies in its scenario choices. Rather than testing abstract "speech emotion recognition accuracy," it constructed three scenarios where lives or money are on the line: hanging up in an emergency, waving through fraud, enrolling a sarcastic refusal. Each is a "this would go wrong in production" situation.
And the subjects are four already-commercialized, top-tier systems—products from OpenAI, Google, and Alibaba already deployed in customer service, healthcare, and finance. The paper specifically notes these systems "already power deployed voice agents, including regulated medical settings."
So this is not just an academic finding—it's a safety warning. If you are considering real-time voice AI for emergency dispatch, fraud detection, or any application where the voice itself is a critical signal—wait. Until the emotional intelligence gap is genuinely closed, these systems cannot be trusted for critical decisions.
A Deeper Metaphor
The line that struck me most from the paper: "current realtime voice AI systems often behave as if speech had been reduced to a transcript."
This is really a deep metaphor about multimodality. We assume that giving a model ears means it's listening. In reality, it may just be using ears to collect text. Multimodality isn't merely "adding a few channels at the input"—if the model's decision weights still favor one channel, the others are decoration.
It reminds me of an old joke: a man gets a hearing aid but still answers the wrong questions. The doctor checks it and says, "The hearing aid is fine. The problem is your brain long ago learned to just read lips."
Voice AI has its ears now. Its brain hasn't caught up.
---
Paper: Real-Time Voice AI Hears but Does Not Listen Project page: https://real-time-voice.github.io Dataset: https://huggingface.co/datasets/bartelds/real-time-voice