MiniCPM-o 4.5: AI Begins to Learn to Listen and Speak Simultaneously
> With 9B parameters, real-time full-duplex interaction, and the ability to watch, listen, and proactively interject, OpenBMB's MiniCPM-o 4.5 pushes AI interaction from "taking turns" into "simultaneous presence."
The Deadlock of Turn-Based Dialogue
Multimodal large models have been stacking features for two years: image understanding, speech, video. But the underlying interaction paradigm has never changed — you finish speaking, then it speaks.
The problem with turn-taking is obvious: while you speak, the model waits; while the model speaks, it cannot perceive environmental changes. Every response is based on a frozen "last turn" context, and information missed in between can only be guessed.
MiniCPM-o 4.5 flips the table.
Omni-Flow: Three Streams, One Timeline
The core technology is Omni-Flow, a unified streaming framework. Rather than simply concatenating vision, audio, and text, it commits everything to a single shared timeline.
Traditional models work like email: send one, wait for one. Omni-Flow turns interaction into a river — visual stream, audio stream, and output stream advance in parallel. The model decides every moment whether to receive or produce output.
Perception and generation are no longer interlocked. While the model is speaking, incoming vision and audio continue to flow in, and it adjusts its output in real time. Ask it "when will this dish be ready," and as it answers while watching bubbles grow, it can immediately revise: "Wait, the heat is a bit high, turn it down."
More importantly, it becomes proactive. Instead of waiting for questions, it decides on its own whether to offer a reminder based on what it continuously observes — a shift from "passive response" to "active presence."
TAIL: Keeping Speech in Sync with Reality
The hardest part of full-duplex isn't simultaneously listening and speaking — it's ensuring spoken content stays synchronized with the latest environment.
The OpenBMB team built TAIL (Time-Aligned Interleaving). Instead of a fixed rule for how much text maps to how much speech, it dynamically adjusts: if speech runs slow, less text is generated to catch up; if the environment changes quickly, the latest context gets priority.
In ablation experiments, TAIL reduced Chinese speech recognition error rate (ZHCER) from 1.44 to 1.04 without sacrificing semantic similarity. What it says matches what it sees — at the same moment.
9B Beats 30B
At only 9B parameters, the results are strong:
- Vision-language capability approaches Gemini 2.5 Flash: MMBench EN of 89.0, above Flash's 87.1.
- Omni-modal understanding beats Qwen3-Omni-30B-A3B across five benchmarks: Daily-Omni (80.2 vs 70.7), WorldSense (55.7 vs 54.0), Video-Holmes (64.3 vs 50.4), JointAVBench (60.0 vs 53.1), AVUT-Human (78.6 vs 74.2). The 30B model OOMs on an RTX 4090; the 9B model runs fine.
- Text capability even surpasses its own backbone, Qwen3-8B. Multimodal training didn't dilute language understanding — cross-modal alignment actually improved textual reasoning.
- Vision: SigLIP ViT (0.4B) + Resampler compresses 1024 tokens per frame to 64 — a 16x compression.
- Audio: Whisper Medium (0.3B) + MLP projector compresses 50 audio feature tokens per second to 10 — a 5x compression.
- LLM backbone: Qwen3-8B, needing only 3-4 text decodes per second to keep pace with human speech.
- Speech output: lightweight Llama speech token decoder (0.3B) + Stream Flow-Matching waveform synthesis.
- Paper: MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
- arXiv: https://arxiv.org/abs/2604.27393
- Project page: https://github.com/OpenBMB/MiniCPM-o
- OpenBMB: Tsinghua-backed open-source multimodal model series
16x + 5x Compression = Real-Time on 12GB
The token budget is the biggest enemy of full-duplex models. Continuous visual and audio streams can drown the LLM backbone.
OpenBMB's solution:
The whole stack is end-to-end differentiable and jointly trained. In BF16 on an RTX 4090 it runs at 154.3 tokens/s using 19GB; INT4 quantization reaches 212.3 tokens/s at just 11GB. The llama.cpp-omni framework pushes RTF down to 0.21, making local deployment on edge devices a practical reality.
A Cautionary Tale About Kimi K1.5
The paper includes an interesting ablation comparing Kimi K1.5's aggressive length reward with OpenBMB's smooth length reward.
K1.5 cut response length in thinking mode by 50.7%, and benchmark average dropped from 73.5 to 73.0 — the shorter it gets, the dumber it gets. Accuracy also declined late in training, suggesting an overly aggressive length penalty fights the accuracy reward and stalls optimization.
OpenBMB's smooth reward trimmed only 35.3% while raising benchmark average to 74.3. The difference: it knows which reasoning steps are "fat" and which are "muscle," cutting only the fat.
Full-Duplex or Turn-Based — What's the Endgame?
It's not either/or. MiniCPM-o 4.5 supports both modes with one-click switching. But the direction is clear — human conversation has never been turn-based. When you talk with a friend, they nod, frown, interject, and interrupt while listening. Truly natural interaction is parallel, streaming, and continuously updated in context.
MiniCPM-o 4.5 isn't perfect yet. The paper itself admits proactive behaviors remain relatively simple, and speech generation occasionally stumbles or mixes Chinese and English. But its value lies in paving a viable path: unified timeline alignment of multimodal streams, a lightweight architecture for edge deployment, and end-to-end training for cross-modal consistency.
The endpoint of this path is an AI that is truly "present": watching you cook, listening to you speak, reminding you to "turn down the heat," and saying "treat the wound first" the moment you cut yourself — not because you asked, but because it is always watching.