In-Depth Comparison of Open-Source Voice-to-Voice LLMs

Forum topic · ✨步子哥 · 2026-07-29

Summary

This forum post presents a comprehensive technical comparison of open-source voice-to-voice large language models, surveying architectures from modular cascade pipelines (ASR → LLM → TTS) to fully end-to-end speech models. It analyzes seven representative systems: Mini-Omni (a ~0.5B lightweight real-time model built on Qwen2-0.5B and Whisper), Kyutai's Moshi (7B, dual-stream full-duplex with the Mimi codec, ~160-200ms latency), Microsoft's VibeVoice (continuous acoustic/semantic tokenizers at 7.5Hz, 60-minute long-form ASR), MiniCPM-o 4.5 (9B omni-modal model with emotion and voice-clone control, deployable on devices), GLM-4-Voice (9B with a 12.5Hz single-codebook speech tokenizer and CosyVoice-based streaming decoder), Qwen2.5-Omni (7B Thinker-Talker architecture with TMRoPE for audio-video sync), and Freeze-Omni (frozen LLM with state classifiers enabling full-duplex turn-taking). Key evaluation dimensions include latency, streaming input/output, emotional and stylistic control, model scale, and training data strategies such as VoiceAssistant-400K instruction tuning. The post concludes by discussing trade-offs between real-time performance and scale, paralinguistic preservation, multilingual extension, safety, and ecosystem maturity.

Key points

This post is an in-depth technical comparison of open-source voice-to-voice (speech-to-speech) LLMs, covering background, core techniques, model-by-model analysis, evaluation, and future outlook.

1. Background and taxonomy

Traditional voice interaction chains ASR → LLM → TTS, which loses paralinguistic information (emotion, tone) and accumulates latency.
Open-source models now span four approaches:
Cascade pipelines (e.g., Hugging Face Speech-to-Speech with VAD → STT → LLM → TTS, compatible with OpenAI's Realtime API).
Streaming cascades (e.g., NetoAI's low-latency telecom agent with sentence-level streaming and 4-bit quantized LLM, RTF < 1.0).
Native end-to-end models (Moshi, GLM-4-Voice, Mini-Omni).
Multi-modal fusion models (MiniCPM-o 4.5, Qwen2.5-Omni).

2. Core techniques

Speech encoding / streaming input: Whisper-based encoders (Mini-Omni); GLM-4-Voice adds a vector-quantization bottleneck inside Whisper for a single-codebook, 12.5 Hz tokenizer (~175 bps); Moshi's Mimi codec downsamples 24 kHz audio to 12.5 Hz with 80 ms frame latency; Freeze-Omni uses chunked streaming input.
LLM core / cross-modal fusion: GLM-4-Voice builds on GLM-4-9B; Qwen2.5-Omni uses a Thinker–Talker architecture; Moshi models two parallel audio streams plus a text "inner monologue"; Freeze-Omni freezes LLM weights to avoid catastrophic forgetting.
Speech decoding / streaming output: GLM-4-Voice's decoder is retrained from CosyVoice for streaming; Moshi combines a small Depth Transformer with a 7B temporal Transformer (theoretical 160 ms latency, ~200 ms in practice); Mini-Omni synthesizes audio while generating text; VibeVoice uses continuous acoustic/semantic tokenizers at 7.5 Hz plus diffusion-based high-fidelity synthesis.
Full-duplex and real-time: Freeze-Omni adds a classifier on the LLM's last layer to predict dialogue state and handle user interruptions; engineering tricks (multi-threading, streaming APIs) bring sub-second end-to-end latency.
Training strategies: Large-scale interleaved speech-text pretraining (GLM-4-Voice, ~1T tokens), instruction tuning (Mini-Omni's VoiceAssistant-400K dataset), and hybrid conditioning (MiniCPM-o feeds both LLM hidden states and text tokens to the speech decoder).

3. Model comparison

| Model | Size | Architecture | Interaction | Open-source status | |---|---|---|---|---| | Mini-Omni | ~0.5B (Qwen2-0.5B) | Whisper-small encoder + custom decoder; "think while talking" | Real-time, streaming output | Code + weights | | Moshi | 7B (Helium) | Dual-stream full-duplex, Mimi codec (12.5 Hz, 1.1 kbps), Depth + temporal Transformers | Full-duplex, ~160–200 ms latency; Hibiki real-time translation | Code, weights, demos (PyTorch/MLX/Rust) | | VibeVoice | Unpublished | Acoustic + Semantic continuous tokenizers @7.5 Hz; diffusion TTS | 60-min single-pass ASR; up to 90-min TTS, 4 speakers | ASR weights/code open; TTS briefly removed over Responsible AI concerns | | MiniCPM-o 4.5 | ~9B (8B vision + 1B text) | SigLip + Whisper-medium + ChatTTS + Qwen2.5-7B; TDM streaming | Full-duplex; emotion/pace/style control; voice cloning; runs on iPad via 640-token visual compression and 4-bit quantization | Code + weights | | GLM-4-Voice | 9B (GLM-4-9B) | VQ speech tokenizer (12.5 Hz, single codebook) + CosyVoice streaming decoder | Chinese/English; controllable emotion, tone, pace, dialect | Code + weights | | Qwen2.5-Omni-7B | 7B | Thinker–Talker; TMRoPE syncs video/audio timing | Real-time text/image/audio/video in, text + speech out | Code + weights | | Freeze-Omni | 7B (Qwen2-7B-Instruct) | Frozen LLM; chunked streaming encoder; single-codebook AR decoder; state classifier for duplex | Speech-to-speech with interruption handling | Code + weights |

4. Evaluation dimensions

Latency: Moshi leads for end-to-end full-duplex (~200 ms); streaming cascades achieve sub-second latency with engineering.
Quality: Diffusion-based synthesis (VibeVoice) and hybrid LLM-conditioning (MiniCPM-o) improve naturalness; low-bitrate tokenizers (GLM-4-Voice at ~175 bps) reduce compute.
Emotion/style control: Strongest in GLM-4-Voice and MiniCPM-o 4.5.
Scale vs. deployment: MiniCPM-o 4.5 and Moshi reach 8–9B (near GPT-4o scale) but MiniCPM-o targets on-device use.

5. Challenges and outlook

Balancing real-time constraints with model scale.
Preserving paralinguistic information in end-to-end generation.
Multilingual and cross-domain extension (most models are English- or Chinese-centric).
Safety and ethics (e.g., voice cloning risks; VibeVoice TTS's temporary removal).
Industrial adoption and ecosystem maturity.

Conclusion

Open-source voice-to-voice LLMs now cover the full spectrum from modular cascade pipelines to native end-to-end full-duplex models. Moshi and Freeze-Omni stand out for full-duplex capability, MiniCPM-o 4.5 for on-device omni-modal interaction, and GLM-4-Voice / Qwen2.5-Omni for controllable, large-scale Chinese-English speech dialogue — together offering flexible building blocks for local real-time voice agents.