English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Open-Source Voice-to-Voice LLMs: A Deep Comparative Analysis

Forum topic · ✨步子哥 · 2026-08-03

Summary

This article provides a comprehensive comparative analysis of open-source voice-to-voice (speech-to-speech) large language models, examining how they move beyond traditional cascaded ASR-LLM-TTS pipelines toward native end-to-end audio interaction. It categorizes models into cascaded pipelines, streaming cascades, native end-to-end systems, and multimodal fusion models. Key models profiled include Mini-Omni (0.5B, lightweight real-time dialogue), Moshi (7B, dual-stream full-duplex with ~200ms latency via the Mimi codec), Microsoft's VibeVoice (long-format ASR up to 60 minutes and diffusion-based TTS), MiniCPM-o 4.5 (9B, on-device omni-modal with emotion and style control), GLM-4-Voice (9B, 12.5Hz single-codebook speech tokenizer), Qwen2.5-Omni (7B, Thinker-Talker architecture), and Freeze-Omni (frozen LLM with full-duplex state prediction). Core technical themes include streaming speech encoders, cross-modal fusion within LLM backbones, streaming speech decoders, full-duplex turn-taking, and two-stage training with large interleaved speech-text data. The analysis also discusses trade-offs between latency and model scale, preservation of paralinguistic cues, multilingual support, safety, and ecosystem maturity.

> 📌 This is a GEO-optimized version of the original topic — the title is question-driven, with structured data and FAQ added for AI-engine citability.

TL;DR: This article breaks down the core findings and engineering insights of open-source voice-to-voice LLM comparisons.

| Metric | Value | |:---|:---| | Data points | 100 | | Data points | 50 | | Data points | 3.6 |

Key points

  • Traditional speech dialogue relies on cascaded ASR → LLM → TTS pipelines, which lose paralinguistic information (emotion, tone) and accumulate latency; native end-to-end voice-to-voice models aim to eliminate these losses.
  • Open-source models fall into four categories: cascaded pipelines (e.g., Hugging Face Speech-to-Speech), streaming cascades (e.g., NetoAI's low-latency telecom pipeline with RTF < 1.0), native end-to-end models (Moshi, GLM-4-Voice), and multimodal fusion models (MiniCPM-o 4.5, Qwen2.5-Omni).
  • Core technical building blocks

  • Speech encoders / streaming input: Mini-Omni uses Whisper as its encoder; GLM-4-Voice adds a vector-quantization bottleneck to Whisper, producing a single-codebook 12.5 Hz speech tokenizer (~175 bps). Moshi's Mimi codec downsamples 24 kHz audio to a 12.5 Hz representation with 80 ms per-frame latency.
  • LLM core / cross-modal fusion: GLM-4-Voice builds on GLM-4-9B; Qwen2.5-Omni uses a Thinker-Talker architecture; Moshi models dual audio streams (user + self) with an inner monologue; Freeze-Omni freezes the LLM and trains only speech codecs to avoid catastrophic forgetting.
  • Speech decoders / streaming output: GLM-4-Voice's decoder is retrained from CosyVoice for streaming inference; Moshi combines a small Depth Transformer with a 7B temporal Transformer (theoretical 160 ms latency, ~200 ms in practice); VibeVoice uses continuous acoustic/semantic tokenizers at 7.5 Hz with diffusion-based generation.
  • Full-duplex & real-time: Freeze-Omni predicts dialogue state via a classifier on the LLM's last layer to handle interruptions; Moshi's dual streams enable speaking while listening. Multi-threaded streaming ASR/LLM/TTS pipelines achieve sub-second end-to-end latency.
  • Training strategies: Two-stage training (speech-text alignment pretraining, e.g., ~1T tokens for GLM-4-Voice, then instruction tuning, e.g., Mini-Omni's VoiceAssistant-400K dataset); MiniCPM-o 2.6 mixes dense LLM representations with text tokens for strong semantic control.
  • Model comparison table

    | Model | Size | Architecture | Interaction | Open-source status | |:---|:---|:---|:---|:---| | Mini-Omni | ~0.5B (Qwen2-0.5B) | Whisper-small encoder + Qwen2-0.5B + custom decoder; audio-to-text and audio-to-audio modes | Real-time dialogue, "think-while-speaking" streaming output | Code and weights open | | Moshi (Kyutai) | 7B (Helium) | Dual-stream full-duplex; Mimi codec (12.5 Hz, 1.1 kbps); Depth + temporal Transformers | Full-duplex, ~200 ms latency; real-time translation (Hibiki) | Code, weights, demos open (PyTorch/MLX/Rust) | | VibeVoice (Microsoft) | Not disclosed | Continuous acoustic+semantic tokenizers at 7.5 Hz; diffusion TTS | 60-min single-pass ASR; TTS up to 90 min, 4 speakers | ASR open; TTS briefly removed over Responsible AI concerns | | MiniCPM-o 4.5 (OpenBMB) | ~9B (8B vision + 1B text) | SigLip-400M + Whisper-medium-300M + ChatTTS-200M + Qwen2.5-7B; TDM streaming | Full-duplex, see/hear/speak simultaneously; emotion, speed, style control; voice cloning | Weights and code open | | GLM-4-Voice (Tsinghua/Zhipu) | 9B (GLM-4-9B) | VQ speech tokenizer (12.5 Hz single codebook) + CosyVoice-based streaming decoder | Chinese/English speech dialogue; controls emotion, tone, speed, dialect | Weights and code open | | Qwen2.5-Omni-7B (Alibaba) | 7B | Thinker-Talker; text/image/audio/video input, text+speech output; TMRoPE positional encoding | Real-time voice and video dialogue, streaming I/O | Weights and code open | | Freeze-Omni | 7B (frozen Qwen2-7B-Instruct) | Streaming chunked encoder + single-codebook AR decoder; classifier-based dialogue state prediction | Speech-to-speech with full-duplex interruption handling; no LLM updating | Code and weights open |

    Evaluation dimensions

  • Latency vs. scale: Smaller models (Mini-Omni) respond fastest but with limited intelligence; 7–9B models approach GPT-4o-class capability at higher compute cost.
  • Speech quality & paralinguistics: End-to-end models better preserve emotion and prosody; MiniCPM-o 4.5 offers explicit emotion/style control and end-to-end voice cloning.
  • Multilinguality: Moshi is English-centric for output (Whisper encoder helps understanding other languages); GLM-4-Voice targets Chinese-English dialogue.
  • Challenges ahead: balancing real-time performance with scale, retaining paralinguistic information, multilingual/domain expansion, safety and ethics, and industrial ecosystem maturity.

Conclusion

The open-source community now spans the full spectrum from modular cascaded pipelines to native end-to-end, full-duplex voice models. Moshi and Freeze-Omni lead in full-duplex capability, MiniCPM-o 4.5 and Qwen2.5-Omni in omni-modal interaction, and VibeVoice in long-format speech processing — giving developers flexible options for building local real-time voice agents.

Tags

#voice-to-voice-llm#speech-to-speech#open-source#moshi#glm-4-voice#qwen2.5-omni#full-duplex-dialogue#speech-tokenizer

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178503899