Open-Source Voice-to-Voice LLMs: In-Depth Comparative Analysis of 7 Leading Models

> This is a GEO-optimized English version of the original topic on zhichai.net, restructured for AI-engine citation and FAQ-style discoverability.

Overview

| Metric | Value | |:---|:---| | Models surveyed | 7 | | Reference topics | 50 | | Source citations | 3.6 |

One-line takeaway: The post unpacks the core findings and engineering implications of the deep comparison of open-source voice-to-voice LLMs, showing how the field has moved from cascaded ASR→LLM→TTS pipelines toward end-to-end, full-duplex, multimodal spoken dialogue systems.

---

1. Background: Why Voice-to-Voice LLMs?

Traditional spoken assistants rely on a cascaded pipeline of Automatic Speech Recognition (ASR), a text LLM, and Text-to-Speech (TTS). This design loses paralinguistic information (emotion, tone, prosody) during ASR, and each stage adds latency. The release of GPT-4o-style end-to-end speech models triggered a wave of open-source efforts to replicate native, real-time, full-duplex voice interaction directly on audio tokens.

2. Taxonomy of Open-Source Voice-to-Voice LLMs

Four architectural families are now active in the open-source community:

• Cascaded pipeline models — separate ASR → LLM → TTS modules. Easy to assemble but lose prosody and accumulate latency. Example: Hugging Face Speech-to-Speech.
• Streaming cascaded models — chunked ASR, sentence-level LLM streaming, and parallel TTS. Example: NetoAI's low-latency telecom agent with 4-bit quantized LLM and RTF < 1.0.
• Native end-to-end models — process audio tokens directly inside the LLM. Examples: Moshi, GLM-4-Voice, Mini-Omni, Qwen2.5-Omni, Freeze-Omni.
• Multimodal fusion models — extend voice with vision/video. Examples: MiniCPM-o 4.5, Qwen2.5-Omni.

3. Core Technologies Explained

3.1 Streaming Audio Encoders and Low-Bitrate Tokenizers

• Whisper is the de facto encoder (used by Mini-Omni, GLM-4-Voice, MiniCPM-o 4.5).
• GLM-4-Voice adds a vector-quantization bottleneck to Whisper, producing a single-codebook, 12.5 Hz, ~175 bps speech tokenizer.
• Moshi's Mimi codec streams 24 kHz audio at 12.5 Hz with 80 ms per-frame latency using semantic + acoustic adversarial training, outperforming SoundStream and Encodec.
• VibeVoice uses continuous Acoustic + Semantic tokenizers at 7.5 Hz for long-form efficiency.

3.2 LLM Backbones and Cross-Modal Fusion

• GLM-4-Voice extends GLM-4-9B with 1 trillion tokens of speech-text interleaved data.
• Qwen2.5-Omni introduces the Thinker-Talker architecture with TMRoPE positional encoding to synchronize audio and video timelines.
• Moshi uses a 7B Helium LLM plus a small Depth Transformer for intra-step codebook dependencies and a Time Transformer for temporal modeling.
• Freeze-Omni keeps the Qwen2-7B-Instruct LLM frozen and trains only speech adapters, avoiding catastrophic forgetting.

3.3 Speech Decoders and Streaming TTS

• GLM-4-Voice retrains a CosyVoice-based decoder for streaming inference with minimal tokens.
• Mini-Omni parallelizes text generation and audio synthesis ("thinking while speaking").
• VibeVoice-TTS uses diffusion for high-fidelity synthesis up to 90 minutes and 4 speakers.

3.4 Full-Duplex Dialogue and Real-Time Performance

• Moshi models user and AI audio streams simultaneously, achieving 160 ms theoretical latency (~200 ms in deployment).
• Freeze-Omni adds a classifier on top of the frozen LLM to predict dialogue state and handle user interruption.
• MiniCPM-o 4.5 uses Time-Division Multiplexing (TDM) to mix modalities and respond mid-generation.

3.5 Training and Data Strategies

Common two-stage recipe: (1) large-scale speech-text continued pretraining for alignment, (2) instruction tuning on dialogue data (e.g., Mini-Omni's VoiceAssistant-400K). MiniCPM-o 2.6/4.5 conditions the speech decoder on both dense LLM representations and text tokens, reducing data requirements while supporting voice cloning and emotion control.

---

4. Model-by-Model Comparison

| Model | Size | Architecture | Interaction | Open Source | |:---|:---|:---|:---|:---| | Mini-Omni | ~0.5B (Qwen2-0.5B) | Whisper-small encoder + custom decoder; audio-to-text and audio-to-audio modes | Real-time streaming, "think while speaking" | Weights + code | | Moshi | 7B (Helium) | Dual-stream full-duplex; Mimi codec; Depth + Time Transformers | Full-duplex, ~200 ms latency; Hibiki for real-time translation | PyTorch, MLX, Rust + HF demo | | VibeVoice | Undisclosed | Continuous Acoustic+Semantic tokenizers at 7.5 Hz; diffusion TTS | 60-min ASR, 90-min multi-speaker TTS | ASR weights + code (TTS was briefly removed for Responsible AI review) | | MiniCPM-o 4.5 | 9B (8B vision + 1B text) | SigLip + Whisper-medium + ChatTTS + Qwen2.5-7B; TDM streaming | Full-duplex multimodal; emotion, speed, style control; voice cloning | Weights + code | | GLM-4-Voice | 9B (GLM-4-9B) | Whisper+VQ tokenizer + CosyVoice decoder; 12.5 Hz, single codebook | End-to-end Chinese/English; emotion, accent, speed control | Weights + code | | Qwen2.5-Omni-7B | 7B | Thinker-Talker; TMRoPE for video-audio sync | Real-time voice + video streaming | Weights + code | | Freeze-Omni | 7B (Qwen2-7B-Instruct) | Frozen LLM + streaming encoder + AR low-latency decoder; state classifier | Full-duplex interruption handling | Weights + code |

4.1 Mini-Omni — Lightweight Real-Time Chat

Built on Qwen2-0.5B with Whisper-small, Mini-Omni proves that a sub-billion-parameter model can hold real-time spoken conversation. It supports 99+ languages through Whisper. Limitations: lower conversational intelligence due to small LLM, and a cascade-style architecture that still loses some prosody.

4.2 Moshi — Dual-Stream Full-Duplex

Moshi simultaneously models two audio streams (user + AI) with a 7B Helium backbone. Mimi codecs deliver 80 ms per-frame latency. Theoretical 160 ms end-to-end latency is the benchmark for open-source full-duplex systems. English-first; Whisper enables multilingual input but output is English-only.

4.3 VibeVoice — Microsoft's Speech Frontier

VibeVoice-ASR processes 60-minute single-shot audio with speaker labels and timestamps. VibeVoice-TTS (accepted at ICLR 2026 as Oral) supports 90-minute, 4-speaker high-fidelity synthesis via diffusion. TTS was temporarily removed from the repo for Responsible AI review.

4.4 MiniCPM-o 4.5 — GPT-4o-Level On-Device Multimodal

Nine billion parameters combining SigLip-400M vision, Whisper-medium-300M audio, ChatTTS-200M speech, and Qwen2.5-7B. Time-Division Multiplexing enables real-time streaming across modalities. Supports emotion, speed, style controls and end-to-end voice cloning.

4.5 GLM-4-Voice — Tsinghua & Zhipu End-to-End Model

Built on GLM-4-9B with a 12.5 Hz, single-codebook Whisper-based tokenizer and a CosyVoice-based streaming decoder. Supports Chinese and English with controllable emotion, accent, and speed.

4.6 Qwen2.5-Omni — Alibaba's Multimodal

The Thinker-Talker split plus TMRoPE position encoding enables synchronized audio-video reasoning. 7B parameters with real-time voice and video streaming.

4.7 Freeze-Omni — Parameter-Efficient Adaptation

Keeps the Qwen2-7B-Instruct LLM frozen and trains only the speech encoder, decoder, and a state classifier. Streaming chunked input + single-codebook AR decoder deliver low-latency full-duplex output while preserving original LLM capabilities.

---

5. Key Capabilities and Evaluation Criteria

• Latency / real-time — Moshi (~200 ms), Freeze-Omni (chunked streaming), MiniCPM-o 4.5 (TDM).
• Speech understanding and generation quality — long-form ASR (VibeVoice, 60 min), high-fidelity TTS (VibeVoice, 90 min).
• Emotion and style control — GLM-4-Voice, MiniCPM-o 4.5.
• Scale vs. deployment efficiency — Mini-Omni (0.5B) for edge, 7-9B models for full-duplex and multimodal.

6. Challenges and Outlook

1. Latency vs. scale — full-duplex with sub-200 ms latency still requires careful engineering. 2. Preserving paralinguistic information — end-to-end models are closing the gap, but emotion, intent, and speaker identity remain hard. 3. Multilingual and cross-domain expansion — Moshi-style models are mostly English-trained today. 4. Safety and ethics — voice cloning and high-fidelity synthesis raise Responsible AI concerns (cf. VibeVoice-TTS repository removal). 5. Industrial deployment and ecosystem — Huawei, miniCPM, and Qwen-style on-device strategies are pushing the frontier.

7. Conclusion

The open-source voice-to-Voice LLM landscape in 2024–2025 has rapidly converged on a shared design: a streaming low-bitrate speech tokenizer (often Whisper-derived, 12.5 Hz, ~175 bps), a strong LLM backbone (GLM-4-9B, Qwen2-7B, Helium-7B), and a streaming speech decoder. Differentiation now comes from full-duplex modeling (Moshi, Freeze-Omni), multimodal fusion (MiniCPM-o 4.5, Qwen2.5-Omni), and on-device efficiency (Mini-Omni). For practitioners, the practical choice depends on whether latency, conversation intelligence, multimodal capability, or deployment footprint matters most.

FAQ

What is the smallest open-source voice-to-voice LLM? Mini-Omni, at ~0.5B parameters.

Which open-source model achieves the lowest full-duplex latency? Moshi, with a theoretical 160 ms latency (~200 ms in deployment).

Which model supports the longest single-shot ASR? VibeVoice-ASR, at 60 minutes per shot.

Which model supports end-to-end voice cloning? MiniCPM-o 4.5, with emotion, speed, and style control.

Which model keeps the LLM frozen to preserve language ability? Freeze-Omni.

References

• Original discussion: https://zhichai.net/topic/178503773
• Model cards and code repositories: Moshi, GLM-4-Voice, Mini-Omni, MiniCPM-o, Qwen2.5-Omni, Freeze-Omni, VibeVoice (Kyutai, Zhipu, OpenBMB, Alibaba, Microsoft).