Sound Is Not LEGO Bricks: What VoxCPM2 Is Really Doing With Tokenizer-Free TTS

This post from zhichai.net is a long-form, essay-style analysis of VoxCPM2, an open-source speech synthesis (TTS) system. It opens with an anecdote attributed to Feynman about a student who memorized quantum mechanics without understanding it, framing the TTS industry as one that recites jargon—Tokenizer, VQ-VAE, Codebook, FSQ—without asking why machine speech sounds like a machine.

Key points

The problem: discretization loses the "human-ness"

• Modern TTS systems discretize continuous audio into codebook tokens, like cutting an oil painting into mosaic tiles—each tile is recognizable, but the whole is degraded.
• Subtle pitch glides, breath sounds, oral resonance, and emotional inflection are lost, producing speech that "sounds human but isn't."
• Codebooks also suffer from vocabulary explosion: finer expression (e.g., "angry ah" vs. "happy ah") requires exponentially more entries.
• The industry is compared to the drunkard looking for keys under the streetlight—stuck searching within the discretization paradigm.

The insight: quantization as regularization, not a goal

• VoxCPM2 does not abandon quantization; it repurposes it. Discretization originally existed for stability—giving generation a fixed anchor so it doesn't drift.
• FSQ (Finite Scalar Quantization) acts as a soft, temporary "skeleton" constraint, while the final audio is generated in continuous space, preserving detail. Hence "Tokenizer-Free."

The four-stage pipeline

1. LocEnc (Local Encoder) — splits audio into patches and extracts local features, like a painter's sketch. 2. TSLM (Text-Semantic Language Model) — a 2-billion-parameter brain based on MiniCPM-4, understanding semantics and context, with LongRoPE for long-context memory. 3. FSQ + RALM (Residual Acoustic Language Model) — FSQ provides the stable skeleton; RALM, an 8-layer network running in parallel with TSLM, captures fine-grained residuals: timbre, pacing, emotion. Formula: final audio = FSQ(skeleton) + RALM(residual). 4. LocDiT (Local Diffusion Transformer) — generates the actual waveform using Flow Matching, sculpting noise into speech step by step; CFG (Classifier-Free Guidance) balances diversity vs. fidelity.

What you can do with it

• Voice Design: create entirely new voices from a text description ("a young female, gentle sweet voice") with no reference audio.
• Controllable voice cloning: clone a voice from a sample, then direct it via text (slower pace, more excited emotion).
• Ultimate Cloning: with reference audio plus an exact transcript, the model performs audio continuation, preserving even recording-equipment character.

Honest criticisms

• Instability: Voice Design and controllable cloning sometimes require multiple attempts; results fluctuate.
• Language imbalance: officially 30 languages and 9 Chinese dialects, but English, Chinese, Japanese, and Korean score high on similarity (SIM > 80), while Arabic (WER 13%) and Czech (WER 24%) lag—a data bias problem, not an architecture one.
• Misuse risk: released under Apache-2.0, voice cloning can enable fraud and deepfakes; detection mechanisms, usage norms, and public awareness are needed.

Closing thought

VoxCPM2's components (MiniCPM-4, RoPE, Diffusion, Flow Matching) are all existing tools; its real contribution is a shift in perspective—asking not "how to improve discretization" but "can discretization serve as a stabilizer instead of a goal." Asking the right question, the author argues, is how the answer appears.