Deep Comparison of Open-Source Voice-to-Voice LLMs: Mini-Omni, Moshi, GLM-4-Voice, Qwen2.5-Omni and More

Forum topic · ✨步子哥 · 2026-07-29

Summary

This in-depth analysis surveys the landscape of open-source voice-to-voice large language models, comparing seven representative systems: Mini-Omni (~0.5B, Qwen2-based real-time speech chat), Kyutai's Moshi (7B dual-stream full-duplex architecture with the Mimi codec at 12.5Hz, ~160ms theoretical latency), Microsoft's VibeVoice (long-form ASR up to 60 minutes and diffusion-based TTS up to 90 minutes with 4 speakers), MiniCPM-o 4.5 (9B omni-modal model with streaming input, emotion/voice-style control and on-device deployment), GLM-4-Voice (9B end-to-end model with a 12.5Hz, single-codebook speech tokenizer built on a quantized Whisper encoder), Alibaba's Qwen2.5-Omni-7B (Thinker-Talker architecture with TMRoPE positional encoding), and Freeze-Omni (frozen Qwen2-7B LLM with a dialogue-state classifier enabling full-duplex interruption). The article categorizes models into cascaded pipelines, streaming cascades, native end-to-end, and multimodal-fusion approaches, explains core techniques (speech encoders, streaming codecs, cross-modal fusion, full-duplex state prediction, two-stage training), and evaluates latency, speech quality, emotion control, model size, and deployment efficiency. It concludes with challenges around real-time/scale trade-offs, paralinguistic preservation, multilingual extension, safety, and ecosystem maturity.

Deep Comparison of Open-Source Voice-to-Voice LLMs

*(Structured English summary of a long Chinese technical review originally published on zhichai.net.)*

Key points

Background: Traditional "listen-and-talk" systems chain ASR → LLM → TTS (cascaded pipelines). This loses paralinguistic information (emotion, tone) and accumulates latency, motivating native speech-to-speech models in the spirit of GPT-4o.
Taxonomy: Open-source voice models fall into four categories:
Cascaded pipelines (e.g., Hugging Face Speech-to-Speech: VAD → STT → LLM → TTS, modular and OpenAI Realtime-API compatible)
Streaming cascades (e.g., NetoAI's low-latency telecom pipeline with sentence-level streaming and 4-bit quantized LLM, RTF < 1.0)
Native end-to-end models (Moshi, GLM-4-Voice, Mini-Omni)
Multimodal fusion models (MiniCPM-o 4.5, Qwen2.5-Omni)

Core technologies

Speech encoding: Mini-Omni reuses Whisper-small; GLM-4-Voice adds a vector-quantization bottleneck to Whisper, producing a single-codebook, 12.5 Hz speech tokenizer (~175 bps). Moshi's Mimi codec downsamples 24 kHz audio to 12.5 Hz frames with only 80 ms per-frame latency. Freeze-Omni supports chunked streaming input.
LLM backbone & cross-modal fusion: GLM-4-Voice builds on GLM-4-9B; Qwen2.5-Omni uses a Thinker-Talker architecture; Moshi models two parallel audio streams (user + self) plus a text "inner monologue"; Freeze-Omni freezes the LLM to avoid catastrophic forgetting.
Decoding & streaming output: GLM-4-Voice's decoder is retrained from CosyVoice for streaming; Moshi uses a small Depth Transformer (within-timestep codebook dependencies) plus a 7B temporal Transformer, achieving ~160 ms theoretical / ~200 ms actual latency; VibeVoice uses continuous Acoustic + Semantic tokenizers at 7.5 Hz with diffusion-based generation.
Full-duplex: Achieved via dialogue-state prediction (Freeze-Omni's classification head detects user interruption) or dual-stream parallel modeling (Moshi).
Training: Typically two-stage—cross-modal alignment pretraining on massive speech-text interleaved data (GLM-4-Voice: ~1T tokens), then instruction tuning (Mini-Omni's VoiceAssistant-400K dataset).

Model comparison table

| Model | Size | Architecture | Interaction | Openness | |---|---|---|---|---| | Mini-Omni | ~0.5B (Qwen2-0.5B) | Whisper-small encoder + custom decoder; "think-while-speak" | Real-time, streaming audio output | Code + weights open | | Moshi | 7B (Helium) | Dual-stream full-duplex; Mimi codec (12.5 Hz, 1.1 kbps); Depth + temporal Transformers | Full duplex, ~160–200 ms latency; Hibiki real-time translation | Code, weights, demos open (PyTorch/MLX/Rust) | | VibeVoice | Undisclosed | Microsoft ASR + TTS; continuous tokenizers at 7.5 Hz; diffusion TTS | 60-min single-pass ASR; up to 90-min TTS with 4 speakers | ASR open; TTS once removed over Responsible AI concerns | | MiniCPM-o 4.5 | ~9B (8B vision + 1B text) | SigLip + Whisper-medium + ChatTTS + Qwen2.5-7B; TDM streaming | Full-duplex multimodal; emotion/rate/style control; voice cloning | Code + weights open | | GLM-4-Voice | 9B (GLM-4-9B) | VQ speech tokenizer (12.5 Hz, single codebook) + CosyVoice-based decoder | Chinese/English speech chat; controllable emotion, dialect, speed | Code + weights open | | Qwen2.5-Omni-7B | 7B | Thinker-Talker; TMRoPE aligns video/audio timing | Real-time voice/video dialogue, streaming I/O | Code + weights open | | Freeze-Omni | 7B (Qwen2-7B-Instruct) | Frozen LLM + streaming encoder + low-latency AR decoder; state classifier for duplex | Speech-to-speech with user interruption, no LLM forgetting | Code + weights open |

Evaluation dimensions

Latency: Moshi leads on end-to-end latency among native models; engineering tricks (multi-threading, streaming APIs, quantization) bring cascaded pipelines to sub-second response.
Quality & control: MiniCPM-o 4.5 and GLM-4-Voice offer fine-grained emotion, speed, and style control; MiniCPM-o 2.6 adds end-to-end voice cloning.
Scale vs. deployment: Sizes range from 0.5B to 9B; MiniCPM-o 4.5 achieves iPad-class on-device inference via aggressive visual token compression (1.8MP image → 640 tokens) and 4-bit quantization.

Challenges and outlook

1. Balancing real-time performance with model scale 2. Preserving paralinguistic (emotional/prosodic) information end-to-end 3. Multilingual and cross-domain extension (most models are English- or Chinese-centric) 4. Safety and ethics (notably VibeVoice's TTS removal) 5. Ecosystem maturity and industrial adoption

Conclusion

The open-source community now spans the full spectrum from modular cascaded pipelines to native end-to-end full-duplex models. Moshi and Freeze-Omni stand out for full-duplex capability, GLM-4-Voice and MiniCPM-o 4.5 for controllable, high-quality speech, and Qwen2.5-Omni for unified multimodal understanding—collectively providing a strong foundation for building local, GPT-4o-like voice agents.