Xiaohongshu Open-Sources dots.tts: 2B-Parameter Continuous Autoregressive TTS Reaching 2.95% WER/CER

Forum topic · 小凯 · 2026-08-17

Summary

Xiaohongshu's dots team, in collaboration with Shanghai Jiao Tong University's X-LANCE lab, has open-sourced dots.tts, a 2-billion-parameter, fully continuous, end-to-end autoregressive text-to-speech foundation model under Apache 2.0. Instead of converting speech into discrete tokens, dots.tts models speech autoregressively block-by-block in the continuous latent space of a 48 kHz AudioVAE (dubbed HoliTok), avoiding quantization information loss in timbre, breathiness, and prosody. It achieves an average WER/CER of 2.95% and SIM of 79.2 across Seed-TTS-Eval's Chinese, English, and hard-case subsets, and an average speaker similarity of 83.9 across 24 languages on MiniMax-Speech benchmarks, ranking first in 19 languages. To address error accumulation in continuous autoregression, it reuses the VAE's semantic encoder as a causal Semantic Encoder. The release includes six checkpoints (pretraining, self-corrective alignment, MeanFlow 4/2/1-step, 1T1A dual-stream), full training, inference, fine-tuning, and distillation code. The 1T1A dual-stream mode reaches a 54.4 ms first-packet latency, supports streaming, and the SGLang-Omni engine provides native support with up to 16 concurrent streams on a single GPU, making it viable for real-time voice agents.

Overview

Xiaohongshu's dots team and Shanghai Jiao Tong University's X-LANCE lab have jointly open-sourced dots.tts, a 2-billion-parameter, fully continuous, end-to-end autoregressive speech synthesis foundation model. Key results:

Seed-TTS-Eval: average WER/CER of 2.95% and average SIM of 79.2 across the Chinese, English, and Chinese hard-case subsets — top average scores on both speaker similarity and content accuracy (SOTA on both boards).
MiniMax-Speech (24 languages): the self-corrective aligned version achieves an average speaker similarity of 83.9, ranking first in SIM for 19 of 24 languages.
License: Apache 2.0. Six checkpoints released (pretraining, self-corrective alignment, MeanFlow 4-step / 2-step / 1-step, and 1T1A dual-stream), with training, inference, fine-tuning, and distillation code all public. The SGLang-Omni team natively supports dots.tts, reaching up to 16 concurrent streams on a single GPU on the Seed-TTS-Eval EN set.

The Problem It Solves

The field of speech synthesis has not converged on a single underlying approach:

Non-autoregressive diffusion: suits offline generation but has poor real-time performance.
Discrete-token autoregression: follows the LLM paradigm, but because speech is inherently continuous, discrete quantization introduces an irreducible information bottleneck in breathiness, nasality, and speaking-rate variation.

dots.tts instead performs autoregressive block-by-block modeling directly in the continuous latent space of a 48 kHz AudioVAE — skipping discrete tokens and quantization loss entirely.

Reconstruction evaluations quantify the gap between the two routes: four mainstream discrete representations score PESQ-NB between 2.40–2.92 and SIM between 0.68–0.85, while the dots.tts VAE reaches PESQ-NB 4.09, SIM 0.969, STOI 0.973, WER 4.14% on LibriSpeech test-other. This representation is named HoliTok — a second-stage semantic training phase adds WavLM frame-level representation alignment plus multi-task supervision (ASR, emotion recognition, speaker identification), giving the continuous latent space clear semantic structure while preserving high-fidelity acoustic detail.

The hardest part is error accumulation in continuous autoregression: without a discrete codebook's quantization safety net, small per-step deviations propagate through the generation history, causing screeching, timbre drift, or text misalignment on long sentences. dots.tts suppresses this noise with a two-stage architecture: the AudioVAE's semantic encoder is reused directly as a causal Semantic Encoder, so each newly generated segment contributes a stable semantic history fed back into subsequent steps. This allows the same model to be trained jointly for ASR and TTS with no extra optimization tricks — the first industrially viable unified model for both speech understanding and generation.

What It Changes

Before this release, the prevailing view was that "autoregressive TTS drifts on long text" and "voice detail cannot be preserved." dots.tts addresses both, and pushes inference-side engineering metrics into territory real-time voice agents can directly adopt:

1T1A dual-stream mode: first-packet audio latency as low as 54.4 ms
Full text entered as prefix; the model generates audio chunk-by-chunk — natively streaming
SGLang-Omni engine: 16 concurrent streams on a single GPU

This gives real-time voice agents, duplex conversation systems, and low-latency narration scenarios a foundation model that, for the first time, combines SOTA timbre, SOTA content accuracy, 16-way single-GPU concurrency, and 54 ms first-packet latency. The timing is notable: agents are already handling "hour-plus work" (OpenAI reported over 70% of Codex users doing this in May), and real-time voice interaction is the next mandatory channel for multimodal agent execution.

What to Watch Next

When MeanFlow is compressed to single-step generation, quality drops noticeably; the team introduces a simplified consistency model (sCM) and Reward-Aware DMD to stabilize 1–2 step generation — this pipeline is also open-sourced, so downstream developers can train their own low-step-count TTS distillations.
In the same week, mainstream solutions including CosyVoice, F5-TTS, VoxCPM, GPT-SoVITS, and IndexTTS all gained practically runnable versions on Apple Silicon unified-memory architectures — the "open-source usable TTS foundation model" track just got noticeably more crowded.

Sources: Xiaohongshu tech WeChat account (dots.llm), Synced (Jiqizhixin).