Overview
dots.tts (repo: https://github.com/studio-dots-ai/dots.tts, ~1267 stars, 130 forks, Apache-2.0 code + weights, 7 released checkpoints) is a 2B-parameter fully continuous, end-to-end autoregressive TTS model — the entire pipeline from audio encoding to acoustic generation uses continuous latents with no discrete tokens. It claims open-source SOTA on Seed-TTS-Eval, MiniMax 24-language, CV3-Eval, and EmergentTTS-Eval.
Sources:
- dots.tts Technical Report: https://arxiv.org/abs/2606.07080 (v1 2026-06-05, v2 2026-08-10)
- dots.tts.edit: https://arxiv.org/abs/2608.02673
- Demo page: https://studio-dots-ai.github.io/dots.tts-demo/
- Lossy quantization — finite codebooks discard detail at tokenization time
- Codebook collapse — undertrained codebooks waste capacity
- Decoupled instability — LLM token prediction separated from acoustic detail causes drift on long utterances
- Frozen AudioVAE encodes 48 kHz mono waveforms into continuous latents; BigVGAN-style causal decoder reconstructs audio.
- Semantic encoder re-encodes generated VAE patches into compact embeddings, stripping high-variance acoustic detail so the LLM focuses on semantics.
- LLM backbone initialized from
Qwen2.5-1.5B-Base; consumes BPE text directly (no phonemes). - AR flow-matching head: a DiT conditioned on LLM hidden states + AR prefix denoises the next VAE patch, with a frozen CAM++ speaker x-vector as side input.
- Two sequence layouts: *plain mode* (standard TTS) and *double-streaming interleaved mode* (speech generated online as the caller streams BPE tokens — for LLM duplex/dialog use).
- Seed-TTS-Eval (zero-shot ~3s ref): SOAR avg WER 2.95 / SIM 79.2 (best SIM in table); zh WER 0.94; Pretrain avg 2.92/78.8. Compared: CosyVoice 3 (1.5B, 3.06/75.3), F5-TTS (0.3B, 4.10/71.4), Qwen3-TTS (1.7B, 3.07/74.5), Seed-TTS (closed, 3.65/77.8), VoxCPM 2 (2B, 3.65/76.7).
- MiniMax 24 languages: SOAR avg SIM 83.9 (highest); best single-language SIM in 19 of 24 languages, tied in 2; high WER on long-tail languages (e.g., Arabic WER 36+).
- CV3-Eval: best hard-en (MF₄ 4.37); leads both cross-lingual SIM subsets (75.0 / 72.8).
- EmergentTTS-Eval (Gemini-2.5-Pro judged, vs gpt-4o-mini-tts): SOAR syntactic complexity 65.7% beats all closed systems; Pretrain emotion score 72.7% tops open-source.
- Paper-reported first-packet latency: 85/54 ms (streaming / double-streaming).
- SGLang Omni, 1×H100: concurrency 16 → 4.76 req/s, RTF 0.81; concurrency 32 → 4.99 req/s, zero failed requests, WER ≈1.3%. OpenAI-compatible
/v1/audio/speech. --optimizecold-start compile ~3 minutes on H800.- Community ports: audio.cpp (C++), dots-tts-mlx / mlx-swift-dots-tts (Apple Silicon), Dots-TTS-ComfyUI
- SGLang Omni serving; Gradio Web Demo; React-based Edit Playground
- Acknowledged foundations: Qwen2.5, DiTAR, ARDiT, HoliTok, BigVGAN, CAM++
- Weak long-tail/low-resource languages (Arabic, Vietnamese: high WER)
- Zero-shot cloning needs ~10s reference audio with aligned text; poor references degrade clones
- Voice-cloning deepfake/misuse risk (documented in README Risks and Limitations)
- Chinese hard-set WER still 6.6% (SOAR)
- Deployment requires PyTorch 2.8 + CUDA 12.8 and H-class GPUs; 3-min compile cold start
- Training data is not public — reproduction cost lies in data and compute, not code
- Full-history conditioning prevents drift on long-form generation
- Reward-free self-correction lowers post-training cost and improves reproducibility
- Double-streaming mode makes it a natural fit for duplex LLM voice agents
- Structured editing tags make speech edits auditable and composable, more stable than natural-language instructions
- Verified high-concurrency serving (~5 req/s per H100, zero failures)
Problem: pain points of discrete tokens
Token-based TTS (VALL-E, CosyVoice, F5-TTS) discretizes audio via codecs (EnCodec etc.), then uses an LLM to predict token sequences. Drawbacks:
dots.tts instead encodes 48 kHz audio into continuous latents, predicts them patch by patch, and denoises via flow matching.
Architecture
Three claimed innovations
1. Multi-objective trained AudioVAE — a semantically structured, prediction-friendly continuous speech space 2. Full-history conditioning in the flow-matching head for long-range consistency and drift suppression 3. Reward-free self-corrective post-training — robustness/quality gains without a reward modelCheckpoint family
| Checkpoint | Use | Key settings |
|---|---|---|
| dots.tts-base | Pretrained baseline / fine-tune base | NFE 10–32 (default 10), CFG 1.2 |
| dots.tts-soar | Highest speaker similarity, quality cloning | NFE 10–32, CFG 1.2 |
| dots.tts-mf | Latency/concurrency-sensitive | NFE 4 recommended, CFG fused |
| dots.tts-mf-2steps / -1step | Low / ultra-low latency | fixed NFE 2 (sCM) / NFE 1 |
| dots.tts-mf-2steps-stts | LLM duplex dialog (double-streaming) | fixed 2-step sCM |
| dots.tts.edit | Instruction-controlled speech editing | NFE 10–32, CFG 1.2 |
Fixed-step models reject incompatible sampler overrides to keep train/inference aligned.
Benchmarks
Efficiency and deployment
Streaming steady-state on a single H800 (bf16, --optimize / torch.compile):
| Config | First packet p50/p90 (ms) | RTF mean | Peak VRAM (GB) | |---|---|---|---| | SOAR / voice_cloning | 225/404 | 0.21 | 7.86 | | SOAR / text_only | 69/79 | 0.18 | 7.85 | | MF / voice_cloning | 204/381 | 0.16 | 5.74 | | MF / text_only | 68/78 | 0.13 | 5.73 |
Two standout features
Double-streaming inference — DotsTtsRuntimeDoubleStreaming + push_text_token() lets the TTS speak while an LLM streams tokens, targeting duplex conversation.
Instruction-controlled speech editing (dots.tts.edit) — XML-style tags <del>, <ins>, <sub>, <emo> control text, emotion, prosody (pitch/speed), and pauses without timestamp alignment. Ships with doteBench (bilingual eval of instruction following, locality preservation, audio quality); post-edit zero-shot WER/SIM differences vs. the base model are negligible across Seed-TTS-Eval shards.