MiniCPM-o 4.5: AI Learns to Listen and Speak Simultaneously
> 9B parameters, real-time full-duplex, seeing and hearing while speaking—and even proactively interjecting. OpenBMB (ModelBest) has pushed AI interaction from "taking turns" into "being simultaneously present."
The Deadlock of Turn-Based Communication
Multimodal large models have been stacking features for two years: images, speech, video. But the underlying interaction paradigm hasn't changed—you speak, then it responds.
The problem with turn-taking is obvious. While you speak, the model waits; while it speaks, it can't perceive environmental changes. Every response is based on a frozen "last turn" context; anything missed in between can only be guessed.
MiniCPM-o 4.5 overturns this table.
Omni-Flow: Three Streams, One Timeline
The core technology is Omni-Flow, a unified streaming framework. Rather than simply concatenating vision, audio, and text, it commits everything to a single shared timeline.
Traditional models work like email: send one, wait for one. Omni-Flow turns interaction into a river—vision, audio, and output streams advancing in parallel. Every second, the model decides: receive now, or emit now?
Perception and generation are no longer interlocked. While the model speaks, incoming vision and audio keep flowing in, and it adjusts its generation in real time. Ask "when is this dish done?" and it may answer, then immediately revise: "Wait, the heat's too high—turn it down."
More crucially, the model becomes proactive. Instead of waiting for questions, it decides on its own whether to speak based on what it continuously observes—a shift from "passive response" to "active presence."
TAIL: Keeping Speech in Sync with the Present
The hardest part of full-duplex isn't hearing and speaking simultaneously—it's ensuring what is said matches the latest environment.
OpenBMB's TAIL (Time-Aligned Interleaving) doesn't use fixed rules like "X text tokens per Y speech tokens." It dynamically adjusts: if speech lagged earlier, generate less text so audio catches up; when the environment changes fast, prioritize injecting the newest context.
In ablations, TAIL reduced Chinese speech recognition error rate (ZHCER) from 1.44 to 1.04 with no loss in semantic similarity—meaning what it says and what it sees belong to the same moment.
9B Outperforms 30B
At only 9B parameters, the results are solid:
- Vision-language capability approaches Gemini 2.5 Flash: MMBench EN of 89.0 vs Flash's 87.1.
- Omni-modal understanding beats Qwen3-Omni-30B-A3B across five benchmarks: Daily-Omni (80.2 vs 70.7), WorldSense (55.7 vs 54.0), Video-Holmes (64.3 vs 50.4), JointAVBench (60.0 vs 53.1), AVUT-Human (78.6 vs 74.2). The 30B model OOMs on an RTX 4090; the 9B runs comfortably.
- Text capability even surpasses its own backbone Qwen3-8B—multimodal training didn't dilute language understanding but pushed text reasoning higher through cross-modal alignment.
- Vision: SigLIP ViT (0.4B) + Resampler compresses 1024 tokens per frame to 64—a 16x compression.
- Audio: Whisper Medium (0.3B) + MLP projector compresses 50 audio feature tokens per second to 10—a 5x compression.
- LLM Backbone: Qwen3-8B, needing only 3-4 text decodes per second to match human speech rate.
- Speech output: lightweight Llama speech token decoder (0.3B) + Stream Flow-Matching waveform synthesis.
- Paper: MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
- arXiv: https://arxiv.org/abs/2604.27393
- Project page: https://github.com/OpenBMB/MiniCPM-o
- OpenBMB (ModelBest): open-source multimodal model series with Tsinghua roots
16x + 5x Compression = Real-Time on 12GB
Full-duplex models fear token budget explosion: continuous visual and audio streams can drown the LLM backbone. OpenBMB's solution:
The pipeline is end-to-end differentiable and jointly trained. In BF16 on an RTX 4090 it reaches 154.3 tokens/s using 19GB; with INT4 quantization, 212.3 tokens/s at only 11GB. The llama.cpp-omni framework pushes RTF down to 0.21—local deployment on edge devices is no longer just a paper number.
Kimi K1.5 as a Cautionary Tale
An interesting ablation compares Kimi K1.5's aggressive length reward with OpenBMB's smooth length reward.
K1.5 cut reasoning-mode response length by 50.7%, and its benchmark average dropped from 73.5 to 73.0—the shorter it gets, the dumber it becomes. Accuracy even declined late in training, indicating the overly aggressive length penalty fought the accuracy reward and choked optimization.
OpenBMB's smooth reward trimmed only 35.3% while the benchmark average rose to 74.3. The difference: it knows which reasoning steps are "fat" and which are "muscle"—it only cuts fat.
Full-Duplex or Turn-Taking?
It's not either/or—MiniCPM-o 4.5 supports both modes with a one-key switch. But the direction is clear: human conversation was never turn-based. When you talk with a friend, they nod, frown, interject, and interrupt while listening. Truly natural interaction is parallel, streaming, and continuously context-updating.
MiniCPM-o 4.5 isn't perfect yet. The paper admits proactive behaviors remain simple, and speech generation occasionally slips or mixes Chinese and English. But its value lies in paving a walkable path: unified timeline alignment of multimodal streams, lightweight architecture for edge deployment, and end-to-end training for cross-modal consistency.
The destination of this path is an AI that is truly "present": watching you cook, hearing you speak, reminding you to turn down the heat, telling you to treat a wound first—not because you asked, but because it's always watching.