dots.tts: A 2B Fully Continuous Autoregressive TTS That Ditches Discrete Tokens

Forum topic · ✨步子哥 · 2026-08-25

Summary

dots.tts is an open-source, 2B-parameter speech synthesis model from studio-dots-ai that generates speech end-to-end with fully continuous latent variables—no discrete audio tokens anywhere in the pipeline. A frozen AudioVAE encodes 48 kHz audio into continuous latents; a Qwen2.5-1.5B-initialized LLM (accepting raw BPE text, no phonemes) predicts latents patch by patch, and a flow-matching DiT head denoises each patch, conditioned on a frozen CAM++ speaker x-vector. Key techniques include a multi-objective AudioVAE training objective, full-history conditioning to suppress drift in long utterances, and reward-free self-corrective post-training. Released under Apache-2.0 with seven checkpoints (base, SOAR, latency-optimized MF variants, a double-streaming conversational model, and dots.tts.edit for instruction-controlled speech editing), the model claims open-source SOTA on Seed-TTS-Eval (average SIM 79.2), MiniMax 24-language (SIM 83.9), CV3-Eval, and EmergentTTS-Eval. Streaming latency reaches ~85/54 ms first-packet, with validated SGLang Omni serving at ~5 req/s on one H100. Limitations include weak long-tail languages, reliance on ~10 s reference audio for cloning, and H-class GPU requirements.

Key points

dots.tts (GitHub: studio-dots-ai/dots.tts) is a 2B-parameter, fully continuous, end-to-end autoregressive TTS model, released under Apache-2.0 with training, inference, fine-tuning, and distillation code plus weights (7 checkpoints).
Technical reports: arXiv:2606.07080 (dots.tts Technical Report) and arXiv:2608.02673 (dots.tts.edit).

Motivation

Discrete-token TTS pipelines (VALL-E, CosyVoice, F5-TTS style) suffer from:

Lossy quantization — finite codebooks discard acoustic detail
Codebook collapse — underused codes reduce expressiveness
Unstable decoupled prediction — drift on long utterances

dots.tts removes discrete tokens entirely: audio is encoded as 48 kHz continuous latents, predicted patch-by-patch by an LLM, and decoded via flow matching.

Architecture

Frozen AudioVAE encodes waveforms into continuous latents; BigVGAN-style causal decoder restores audio.
LLM backbone initialized from Qwen2.5-1.5B-Base; consumes raw BPE text (no phonemes).
Semantic encoder re-encodes generated VAE patches into compact embeddings, stripping high-variance acoustic detail.
AR flow-matching head (a DiT) denoises the next patch, conditioned on LLM hidden states plus the AR prefix; a frozen CAM++ speaker x-vector anchors voice identity.
Two modes: plain (text prefix, standard TTS) and double-streaming interleaved (speaks while the calling LLM streams tokens — for duplex conversation).
Claimed innovations: multi-objective AudioVAE training, full-history conditioning against drift, and reward-free self-corrective post-training (no reward model needed).

Checkpoints

| Checkpoint | Purpose | |---|---| | dots.tts-base | Pretrained baseline / fine-tune base | | dots.tts-soar | Highest speaker similarity; quality cloning | | dots.tts-mf / -2steps / -1step | Latency-optimized (NFE 4 / 2 / 1) | | dots.tts-mf-2steps-stts | Duplex LLM conversation (double-streaming) | | dots.tts.edit | Instruction-controlled speech editing (del/ins/sub/emo; no timestamps needed) |

Benchmarks (reported by the authors)

Seed-TTS-Eval: SOAR achieves best average SIM 79.2; Chinese WER 0.94; average WER 2.95 — ahead of Seed-TTS, VoxCPM 2, Qwen3-TTS, CosyVoice 3, F5-TTS.
MiniMax 24-language: average SIM 83.9, best in 19 of 24 languages; long-tail languages (e.g., Arabic, WER 36+) remain weak.
CV3-Eval: best on hard-en and cross-lingual SIM subsets.
EmergentTTS-Eval: beats all closed-source systems on syntactic complexity (65.7%, SOAR); top open-source emotion score (72.7%, Pretrain).

Efficiency & deployment

Single H800 (bf16, --optimize): streaming RTF ~0.13–0.21; first-packet latency 225/404 ms (voice cloning) or 69/79 ms (text-only). Paper reports 85/54 ms first-packet (streaming / double-streaming).
SGLang Omni on one H100: ~5 req/s at concurrency 32, zero failed requests, WER ≈ 1.3%; OpenAI-compatible /v1/audio/speech endpoint.
Cold start of --optimize (torch.compile) is ~3 minutes.
Community ports: audio.cpp (C++), MLX / mlx-swift (Apple Silicon), ComfyUI node.

Limitations

Weak long-tail / low-resource languages (high WER).
Zero-shot cloning needs ~10 s aligned reference audio; poor references degrade quality.
Voice-cloning deepfake risks acknowledged in README.
Hard Chinese subset WER still 6.6% (SOAR).
Requires H-class GPUs (PyTorch 2.8 + CUDA 12.8 recommended); training data is not released, so full reproduction is nontrivial.