Xiaohongshu Open-Sources dots.tts: 2B-Parameter Continuous Autoregressive TTS Hits 2.95% WER/CER in Zero-Shot Voice Cloning

Forum topic · 小凯 · 2026-08-17

Summary

Xiaohongshu's dots team and Shanghai Jiao Tong University's X-LANCE Lab have open-sourced dots.tts, a 2-billion-parameter, fully continuous, end-to-end autoregressive text-to-speech foundation model. Unlike mainstream approaches that quantize speech into discrete tokens or rely on non-autoregressive diffusion, dots.tts models audio directly as continuous latent chunks in a 48 kHz AudioVAE space (named HoliTok), avoiding irreversible quantization losses in timbre, breathiness, and speaking rate. It achieves an average WER/CER of 2.95% and mean speaker similarity of 79.2 across Seed-TTS-Eval's Chinese, English, and hard subsets, plus top similarity on 19 of 24 languages in MiniMax-Speech benchmarks. The release includes six checkpoints (pretrained, self-correcting alignment, MeanFlow 4/2/1-step, 1T1A dual-stream), full training, inference, fine-tuning, and distillation code under Apache 2.0. SGLang-Omni provides native inference support with up to 16 parallel streams per GPU, and the 1T1A mode achieves 54.4 ms first-packet latency—making it suitable for real-time voice agents and duplex dialogue systems.

Overview

Xiaohongshu's dots team, together with Shanghai Jiao Tong University's X-LANCE Lab, has open-sourced dots.tts — a 2-billion-parameter, fully continuous, end-to-end autoregressive speech synthesis foundation model. On Seed-TTS-Eval, it achieves an average WER/CER of 2.95% and average speaker similarity (SIM) of 79.2 across the Chinese, English, and Chinese-hard subsets — topping both content accuracy and timbre similarity. On MiniMax-Speech's 24-language benchmark, the self-correcting alignment version reaches an average SIM of 83.9, ranking first in 19 of 24 languages.

The open-source release is comprehensive: six checkpoints (pretrained, self-correcting alignment, MeanFlow 4-step / 2-step / 1-step, and 1T1A dual-stream), plus full training, inference, fine-tuning, and distillation code, all under an Apache 2.0 license. The SGLang-Omni team already supports dots.tts natively, reaching up to 16 parallel streams on a single GPU on the Seed-TTS-Eval English set.

The Problem It Solves

TTS architecture routes have not converged. One camp uses non-autoregressive diffusion (good offline, poor real-time); another compresses speech into discrete tokens and applies the LLM paradigm — but discrete quantization imposes an irreducible information bottleneck on naturally continuous attributes like breathiness, nasality, and speaking-rate variation.

dots.tts instead models speech autoregressively, chunk by chunk, directly in the 48 kHz AudioVAE continuous latent space — no discrete tokens, no quantization loss. Reconstruction benchmarks quantify the gap: four mainstream discrete representations score PESQ-NB between 2.40–2.92 and SIM between 0.68–0.85, while dots.tts's VAE achieves on LibriSpeech test-other:

PESQ-NB: 4.09
SIM: 0.969
STOI: 0.973
WER: 4.14%

This representation is named HoliTok: a second-stage semantic training phase adds WavLM frame-level alignment and multi-task supervision (ASR, emotion, speaker identification), giving the continuous latent space both clear semantic structure and high-fidelity acoustic detail.

The hard part is error accumulation in continuous autoregression: with no discrete codebook to quantize away small deviations, errors propagate through the generation history, causing screeching, timbre drift, or text misalignment on long inputs. dots.tts reuses the AudioVAE's semantic encoder as a causal Semantic Encoder, feeding stable semantic history back into each generation step. This lets the same model train ASR and TTS jointly without extra tricks — a first industrial-grade path to unified speech understanding and generation.

What It Changes

Before this release, the consensus was that autoregressive TTS drifts on long text and loses voice detail. dots.tts addresses both, and pushes inference metrics to a level real-time voice agents can adopt directly:

1T1A dual-stream mode: first-packet audio latency as low as 54.4 ms
Full text fed as a prefix, audio generated chunk by chunk — natively streaming
16-way parallel inference per GPU with SGLang-Omni

For real-time voice agents, duplex dialogue, and low-latency narration, this is the first foundation model offering SOTA timbre + SOTA content accuracy + 16-way single-GPU + 54 ms first packet simultaneously. The timing matters: agents are already handling hour-plus workloads (OpenAI reported over 70% of Codex users doing so in May), and real-time voice interaction is the next gateway to multimodal agent execution.

What to Watch Next

Quality degrades when MeanFlow is compressed to single-step generation, so the team introduced a simplified consistency model (sCM) and Reward-Aware DMD to stabilize 1–2 step generation — also open-sourced, letting downstream developers train their own low-step TTS distillations. In the same week, CosyVoice, F5-TTS, VoxCPM, GPT-SoVITS, and IndexTTS all gained runnable builds on Apple Silicon unified-memory architectures — the open-source TTS foundation model space just got noticeably more crowded.

Sources: Xiaohongshu's dots.llm technical account, Synced (机器之心).