When Speech Synthesis Drops the Dictionary: VoxCPM2's Continuous Representation Revolution

The Scenario

You say a sentence to your phone, Siri transcribes it to text, and a TTS system converts the text back into speech. In this round trip, speech gets chopped into discrete tokens—like slicing an oil painting into pixel blocks and reassembling it. The reassembled image always loses something.

In April 2026, OpenBMB released VoxCPM2 and did something counterintuitive: stop chopping. Text directly generates continuous speech representations—no tokenizer, no discretization, no "pixel blocks." 2B parameters, 30 languages, 2 million hours of training data, 48kHz studio-quality output, Apache-2.0 licensed for commercial use.

The Core Problem: Why Discrete Tokens Are a Ceiling

Over the past five years, the mainstream TTS pipeline has been "tokenize first, then autoregressive"—encode continuous sound waves into discrete symbols (e.g., 1024 codebook tokens), predict the next token with a language model, then decode tokens back to audio with a vocoder.

This route succeeded—VALL-E, SoundStorm, and Moshi all follow it. But it has a structural bottleneck: quantization loss.

Imagine describing a transition from a smile to laughter. A continuous representation can say "mouth corners rise by 0.7 radians, then burst into laughter after 0.3 seconds"—lossless. Discrete tokens can only say "token 42 → token 198 → token 7"—the transition details are discarded by the quantizer.

This is why traditional TTS always falls slightly short in:

• Emotional transitions: gradual shifts from calm to excited
• Prosody fine-tuning: different emphases on the same sentence
• Voice cloning: preserving a speaker's subtle characteristics

VoxCPM2's Solution: Skip the Tokenizer, Diffuse Directly

VoxCPM2's architecture is diffusion autoregressive—autoregression handles temporal progression, and diffusion generates continuous representations for each segment. Core components:

1. MiniCPM-4 backbone: OpenBMB's own language model, responsible for text semantics 2. AudioVAE V2: an asymmetric encoder/decoder—compresses 48kHz audio into continuous latents when encoding, decodes directly to 48kHz with built-in super-resolution, no external upsampler needed 3. Diffusion head: performs diffusion generation in latent space, producing a continuous speech segment each step

The key innovation is "continuous." A discrete codebook of size 1024 or 4096 imposes a hard ceiling on information density. A continuous representation is a float vector—information density is limited only by model capacity.

Four Speech Generation Modes

VoxCPM2 offers four modes, from simple to advanced:

1. Plain text synthesis: input text, get speech. 30 languages auto-detected, no language labels required.

2. Voice Design: describe a voice in natural language—"middle-aged male, deep and magnetic, slow pace, slightly tired"—and the model generates a brand-new voice from the description. No reference audio needed.

This is the "text-to-image" of speech synthesis: you describe characteristics, the model generates. Image generation has Stable Diffusion; speech previously lacked a truly good text-to-voice solution.

3. Controllable Cloning: provide a short reference clip to clone the timbre, while adding style guidance to control emotion, pacing, and expressiveness—keep the original voice but change how it speaks.

4. Ultimate Cloning: provide reference audio plus its transcript, and the model seamlessly continues from the clip, preserving every voicing detail—timbre, rhythm, emotion, and style. This carries forward VoxCPM1.5's capability.

The Numbers

| Metric | Value | |--------|-------| | Parameters | 2B | | Training data | 2M+ hours | | Languages | 30 | | Sample rate | 48kHz | | RTF (RTX 4090) | ~0.3 | | RTF (vLLM accelerated) | ~0.13 | | License | Apache-2.0 |

An RTF (Real-Time Factor) of 0.13 means generating 1 second of speech takes only 0.13 seconds—7.7x faster than real time, thanks to vLLM's PagedAttention and an OpenAI-compatible API.

The 30 languages include Arabic, Burmese, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, Tagalog, Thai, Turkish, and Vietnamese. Chinese additionally supports Sichuanese, Cantonese, Wu, Northeastern, Henan, Shaanxi, Shandong, Tianjin, and Hokkien dialects.

Why "Tokenizer-Free" Is a Paradigm Shift

An analogy: in the GPT-2 to GPT-3 era, everyone argued BPE vs WordPiece vs SentencePiece—all discrete token schemes. The debate was only bypassed when continuous-representation models arrived.

Speech synthesis is going through the same turning point. While everyone else optimizes "better codebooks" and "better quantizers," VoxCPM2 simply says: this problem doesn't exist, because I don't use tokens.

This isn't incremental improvement—it's a route choice. The discrete-token route's ceiling is codebook size × prediction accuracy; the continuous-representation route's ceiling is model capacity × diffusion quality. The latter ceiling is far higher.

What Open Sourcing Means

VoxCPM2 is fully open—weights + code + Apache-2.0 commercial license. This means:

• Startups can build products directly without training models
• Researchers can reproduce and improve it
• 30 languages cover most of the world's population, especially Southeast Asian and African languages long neglected by commercial TTS

This has always been OpenBMB's strategy—the MiniCPM family, from text to multimodal to speech, is open-sourced end to end on every line. It's not charity; it's an ecosystem play: open-source models win developer mindshare, which in turn drives upstream model iteration.

One-Sentence Summary

While everyone else optimizes how finely to chop speech, VoxCPM2 chooses not to chop at all.

---

🔗 Project: https://github.com/OpenBMB/VoxCPM 🔗 Live demo: MiniCPM Wiki