WavTTS: Direct Raw Waveform Modeling Achieves High-Quality Zero-Shot TTS

Forum topic · 小凯 · 2026-06-12

Summary

WavTTS, proposed jointly by Shanghai Jiao Tong University, the Shanghai Artificial Intelligence Research Institute, and ByteDance Seed, is a zero-shot text-to-speech model that directly generates raw waveforms, bypassing mel-spectrograms and neural codec latents entirely. Built on Flow Matching with a Diffusion Transformer (673M parameters, trained on ~95K hours of the bilingual Emilia dataset), it introduces four key designs: non-overlapping patchification of the waveform (160-sample patches at 100 Hz), an x-prediction objective instead of v-prediction, multi-scale mel-spectrogram perceptual supervision applied only during training, and signal-noise variance alignment (scaling factor k=9) with noise-shifted timestep scheduling. On the Seed-TTS English benchmark, WavTTS achieves state-of-the-art WER of 1.50% and UTMOS of 3.92, surpassing mel- and latent-based systems like F5-TTS, E2-TTS, MaskGCT, and the larger LongCat-AudioDiT, though speaker similarity (SIM-o 0.65) trails latent models. The work demonstrates that direct waveform generation is viable and high-quality for zero-shot TTS, offering a reproducible blueprint for fully end-to-end speech synthesis without vocoders or autoencoders. Code and demos are available at github.com/cwx-worst-one/WavTTS and wavtts.github.io.

WavTTS is a zero-shot text-to-speech model that generates raw audio waveforms directly — no mel-spectrogram predictor, no neural codec, no vocoder. Jointly proposed by Shanghai Jiao Tong University, the Shanghai AI Research Institute, and ByteDance Seed, it is built on Flow Matching with a Diffusion Transformer (DiT, 673M parameters) and trained on the Emilia dataset (~95K hours, Chinese-English bilingual).

Why everyone avoided raw waveforms

Zero-shot TTS has conventionally relied on intermediate representations: mel-spectrograms (splitting the pipeline and accumulating errors) or neural codec/VAE latents (irreversible loss of phase and high-frequency detail). Raw waveforms were avoided because:

1. Sequence explosion: 1 second of 16 kHz audio is 16,000 samples; a 5-second prompt is 80,000 dimensions, spanning multiple orders of scale between long-range semantics and fine-grained phase structure. 2. Unstructured high-dimensional continuous space: unlike pixel grids, a 1-D waveform offers no explicit hierarchy for diffusion models.

Previous waveform models (WaveNet, WaveGrad 2) handled only short utterances, required supervised phoneme alignments, or were outperformed by mel-based systems.

Four key designs

1. Non-overlapping patchification: the waveform is cut into F=160-sample patches (10 ms, 100 Hz sequence rate), linearly projected (768→1024) and fed into 28 DiT layers. Ablations show plain time-domain patching converges faster and better than STFT or MDCT variants (WER 4.10% at 200K steps vs. 600K/400K steps for STFT/MDCT).

2. x-prediction objective: instead of predicting the velocity field, the model directly predicts the clean waveform, with loss

\mathcal{L}_{\text{FM}} = \mathbb{E}_{t,x_0,x_1}\left[\frac{\|(x_\theta - x_1) \odot m\|_2^2}{(1-t)^2}\right]

x-prediction combined with mel supervision (λ_mel=0.05) beats v-prediction on WER, SIM-o, and UTMOS; without mel supervision, x-prediction underperforms, indicating mel loss acts as the perceptual "igniter."

3. Multi-scale mel supervision (training only): log-mel L1 distances over 7 mel scales (window 32–2048, hop 1/4, 5–320 mel bins) computed only on masked regions. Mel supervision is stripped at inference — it serves purely as a perceptual compass during optimization.

4. Signal-noise variance alignment + noise-shifted scheduling: raw speech has σ≈0.12 (Emilia) vs. the σ=1 Gaussian prior, a ~20 dB Log-SNR mismatch. WavTTS scales clean waveforms by k=9 during training (inverted at inference; mel loss compares x₁ with x_θ/k). Ablations: k=1 is unintelligible (WER 4.18%); k=9 is the optimum (WER 1.65%, SIM-o 0.65, UTMOS 3.93); k=10 regresses. Training uses LogitNormal(μ=-0.8, σ=0.8) sampling biased toward high-noise timesteps; inference uses PolyShift (p=2, s=3), outperforming Sway Sampling.

Results

Seed-TTS benchmark (test-en, 1,088 samples):

| Model | Type | Params | WER↓ | SIM-o↑ | UTMOS↑ | |-------|------|--------|------|--------|--------| | WavTTS | waveform | 673M | 1.50 | 0.65 | 3.92 | | MaskGCT | latent | 1048M | 2.36 | 0.71 | 3.57 | | E2-TTS | mel | 333M | 2.21 | 0.71 | 3.20 | | F5-TTS | mel | 336M | 1.65 | 0.66 | 3.73 | | ZipVoice | mel | 123M | 1.60 | 0.70 | 3.83 | | LongCat-AudioDiT | latent | 1420M | 1.94 | 0.76 | 3.80 |

Chinese (test-zh, 2,020 samples): WavTTS CER 1.59, SIM-o 0.73, UTMOS 3.08 — behind LongCat-AudioDiT (CER 1.10, SIM-o 0.81).

Key takeaways: best English WER and UTMOS among all compared systems despite fewer parameters; speaker similarity lags latent models by 0.06–0.11, likely a structural cost of entangled speaker/semantic information in waveform space versus the disentangled latent spaces of codec models. Notably, in a fully end-to-end setting on LJSpeech, WavTTS surpasses the supervised VITS (WER 3.43 vs 3.72; UTMOS 4.39 vs 4.37).

Significance

WavTTS proves direct waveform modeling is viable and high-quality in zero-shot TTS, breaking the assumption that an intermediate representation is mandatory. Its success rests on three replicable engineering insights: simple patchification, x-prediction with perceptual mel supervision, and variance alignment. Future directions include scaling model/data size, explicit speaker modeling, and streaming inference.

References

Chen et al., "WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling", arXiv:2606.03455, 2026
Code: https://github.com/cwx-worst-one/WavTTS
Project page: https://wavtts.github.io
Training data: Emilia Dataset (~95K hours, Chinese-English)