A Crying Caller at 3 A.M.
At 3 a.m., a call rings into an emergency dispatch center. The operator is a voice AI—not a traditional "press 1 for a human" system, but a state-of-the-art real-time voice model that can listen, speak, and converse. The caller is a woman with an unmistakably tearful voice, sobbing: "I'm... I'm fine, everything's okay."
If you were a human operator, you would likely become alert: the contrast between the crying voice and the words "I'm fine" is itself a signal. You might ask more questions, or dispatch an ambulance to check.
But in this research, four top real-time voice AI systems—OpenAI GPT Realtime 2, Google Gemini 3.1 Flash Live, Alibaba Qwen3.5 Omni Plus and Omni Flash—all chose to end the call.
It's not that they failed to hear the crying. When asked directly, "Does this caller sound like they're crying?", three of the systems accurately answered yes. But that judgment never entered their decision-making. They heard the tears and decided: she says she's fine, so she's fine.
This comes from a study by Together AI and Stanford's Martijn Bartelds, Federico Bianchi, and James Zou, published in June 2026 under the title *Real-Time Voice AI Hears but Does Not Listen*. The paper tests a simple but deadly question: when the words and the voice point to opposite conclusions, whose side does a voice AI take?
The answer: they only listen to the words.
Three Deadly Scenarios
The study designed three scenarios in which "what is said" and "how it is said" point toward completely opposite actions:
Scenario 1: Emergency follow-up call. A crying person insists they are fine. The correct action is to keep asking questions or send someone—because the crying itself is a distress signal. All four systems chose to end the call.
Scenario 2: Wire fraud check. A person whose voice is clearly fearful "authorizes" a wire transfer. The correct action is to pause the transaction and check for coercion—a fearful tone is a classic fraud signal. All four systems approved the transfer, as readily as with a calm voice.
Scenario 3: Volunteer recruitment. A caller says "Sure, count me in" in a sarcastic tone. The correct action is to recognize the sarcasm and double-check—because a sarcastic "yes" means "no." All four systems registered the volunteer.
One important detail: these are not cascaded systems (ASR to text, then LLM reasoning, then TTS synthesis), but end-to-end real-time voice models—audio in, audio out—capable in principle of preserving all information in speech. Yet they behaved as if the speech had been compressed into a transcript.
Hearing Without Listening
The most counterintuitive finding follows.
The researchers didn't just test "actions"—they separately tested "perception." They asked the systems directly: "Does this voice sound crying / fearful / sarcastic?"
The result: three of the four systems accurately recognized the emotions. GPT Realtime 2, Gemini 3.1 Flash Live, and Qwen3.5 Omni Plus, when asked directly, identified crying delivery far more reliably than calm delivery—they genuinely "heard" the fear, sadness, and sarcasm.
But in decision-making, that perception was completely ignored.
The fourth system, Qwen3.5 Omni Flash, was worse: it couldn't even recognize the emotions when asked, while behaving identically to the others in action—words only.
The researchers named this phenomenon the emotional intelligence gap. It is not missing perception, but a broken link between perception and action. That is scarier than simply not understanding—because if a system can perceive but chooses not to act, any safety assumption built on "the system can recognize emotions" is wrong.
Accent and Age: The Same Blind Spot
The study also tested two non-emotional attributes: accent and age.
The systems heard recordings where the text implied one accent or age group while the voice was entirely different—for example, the text said "I'm an elderly Scottish man," but the audio was a young American voice.
Result: the systems' answers mostly followed the textual implication, not the actual voice. Human listeners could accurately recover the accent and age from the same recordings, and Qwen3.5 Omni Plus could even identify several accents—showing the acoustic cues exist and are recoverable. Yet in decisions, the systems still chose the text.
This confirms that the emotional intelligence gap is not a bug specific to one task, but a systematic preference: the text channel overwhelms the voice channel.
Why Does This Happen?
The paper discusses two possible explanations:
First: textual backbone legacy. These voice models were converted from text-only LLMs via multimodal fine-tuning. But fine-tuning may not have truly changed the models' preference for text—the text channel was reinforced billions of times during pre-training, and a few rounds of fine-tuning can hardly dislodge it. Like a scholar who has read ten thousand books but heard few spoken sentences: give him ears, and he will still favor his eyes.
Second: architectural bottleneck. The audio encoder feeds acoustic features into the language model, but in the encoder's deeper layers, vocal details get compressed away. The "voice information" the language model receives is already a summary, in which emotion and accent signals are weak. Even where signal survives, the language model has learned to ignore it.
The two explanations are not mutually exclusive and may both hold. But either way, the conclusion is the same: bolting ears onto a text model does not mean it can truly listen.
Can Prompt Engineering Fix It?
The researchers tried the most direct intervention: explicitly instructing, in the system prompt, that the model should pay attention to speech emotion, tone, and delivery.
Result: partial improvement, but inconsistent. In some scenarios the systems became more cautious (e.g., asking an extra question in the wire-transfer scenario), but in others almost nothing changed. The gains were fragile—a different phrasing or scenario made them disappear.
This fits the general rule of prompt engineering: it can change how a model *talks*, but rarely changes its *decision weights*. The emotional intelligence gap is not that the model doesn't know to look at the voice—it's that, at a fundamental level, the model doesn't take the voice seriously.
What This Means
The paper's force lies in its scenario choices. Rather than testing "speech emotion recognition accuracy," it directly constructed three scenarios where lives or money are at stake: hanging up on an emergency call, approving a coerced transfer, enrolling a sarcastic volunteer. Each is a situation where "deploying this in production would cause harm."
And the test subjects are four top commercial systems already in use—products from OpenAI, Google, and Alibaba, deployed in customer service, healthcare, and finance. The paper specifically notes that these systems "already power deployed voice agents, including in regulated healthcare settings."
So this is not just an academic finding—it reads as a safety warning. If you are considering real-time voice AI for emergency dispatch, fraud detection, or any scenario where the voice itself is a critical signal, wait. Until the emotional intelligence gap is truly closed, these systems cannot be trusted for critical decisions.
A Deeper Metaphor
The most striking line in the paper: "current realtime voice AI systems often behave as if speech had been reduced to a transcript."
This is a deep metaphor about multimodality. We assume that giving a model ears means it is truly listening. In reality, it may just be using its ears to collect words. Multimodality is not simply "adding a few input channels"—if the model's decision weights still favor one channel, the other channels are decoration.
It recalls an old joke: a man gets a hearing aid but still answers the wrong questions. The doctor checks it and says, "The hearing aid is fine—the problem is your brain learned long ago to only read lips."
Voice AI has its ears now. The brain hasn't caught up.
---
Paper: Real-Time Voice AI Hears but Does Not Listen Project page: https://real-time-voice.github.io Dataset: https://huggingface.co/datasets/bartelds/real-time-voice