WavTTS: Waveform Factions' Revenge, or Another Spiral Back to Mel?
*An analysis post from zhichai.net on a zero-shot TTS paper that directly models raw waveforms, turning "end-to-end" from a slogan into real, feasible engineering.*
Background: Why everyone avoids the waveform
Zero-shot TTS has an unwritten rule: to do speech synthesis well, you must find a "stand-in" for the waveform:
- Mel spectrograms: predict a 2D spectrogram, then let a vocoder translate it into a waveform. Information loss is tolerable, but the pipeline is split into two stages, and errors accumulate.
- Neural codecs / VAE latents: compress the waveform into discrete or continuous tokens, run diffusion in token space. Efficiency improves, but irreversible information loss hides in the compression.
- The target is directly the waveform, no noise-component interference;
- It naturally supports mel supervision (predicted and ground-truth waveforms can be compared in mel space);
- The \((1-t)^{-2}\) weighting penalizes high-noise (low-t) regions more.
- k=1: WER 4.18%, SIM-o 0.32, UTMOS 2.40 — unintelligible
- k=5: WER 1.51%, but electronic noise remains
- k=9: WER 1.65%, SIM-o 0.65, UTMOS 3.93 — best balance
- k=10: overscaling, performance regresses
- 🥇 Best English WER (1.50%) and UTMOS (3.92), beating even larger latent models.
- 🥉 Speaker similarity lags: 0.65 vs 0.71–0.76 (en), 0.73 vs 0.81 (zh).
- Under end-to-end settings it even beats supervised systems (LJSpeech: WER 3.43 vs VITS 3.72; UTMOS 4.39 vs 4.37), suggesting zero-shot generalization is a hidden advantage of waveform modeling.
- No reconstruction bottleneck: no irreversible loss of high-frequency detail and phase from codec compression.
- Fully end-to-end optimization: every loss acts directly on the final waveform.
- More natural generalization: no hard coupling to a fixed mel/codec distribution assumption.
- Chen et al., "WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modeling", arXiv:2606.03455, 2026
- Code & models: https://github.com/cwx-worst-one/WavTTS
- Project page: https://wavtts.github.io
- Training data: Emilia Dataset (~95K hours, Chinese-English bilingual)
- Baselines: MaskGCT, E2-TTS, F5-TTS, ZipVoice, LongCat-AudioDiT, CosyVoice, Spark-TTS, Llasa-1B
The shared logic: raw waveform sequences are too long—modeling them directly means counting every sample point, like at 16kHz or beyond. Nobody wanted the job.
WavTTS took the job.
Why waveforms are hard
1. Sequence explosion: at 16kHz, 1 second of audio = 16,000 floats; a 5-second prompt is an 80,000-dimensional vector. The model must capture both long-range semantic dependencies and fine-grained phase/periodicity structure—scales spanning several orders of magnitude. 2. Unstructured high-dimensional continuous space: unlike pixels with 2D topology, waveforms are 1D with semantic anchors buried in phase and spectral envelopes, with no explicit hierarchy.
Prior end-to-end waveform models (WaveNet, WaveGrad 2) were either short-text only, dependent on supervised phoneme alignment, or beaten on quality by mel-based systems. Zero-shot waveform TTS had never been cracked—until now.
The four key designs
WavTTS comes from Shanghai Jiao Tong University, the Shanghai AI research institution, and ByteDance Seed. It uses Flow Matching + a Diffusion Transformer (DiT), 673M parameters, trained on Emilia (~95K hours, Chinese-English bilingual).
1. Patchification
The waveform is cut into non-overlapping patches of F=160 samples (10ms at 16kHz, 100Hz sequence rate), each treated as a token, projected linearly (768→1024 dims), then fed to a 28-layer DiT.
Ablation (Section 5.3.5): at 200K steps, raw waveform modeling reached 4.10% WER; STFT needed 600K steps and MDCT 400K steps to converge to similar levels. The simplest representation is also the most effective.
2. x-prediction objective
Instead of predicting the velocity field \(v_t = x_1 - x_0\), WavTTS directly predicts the clean waveform \(x_1\):
x-prediction + λ_mel=0.05 beats v-prediction across WER, SIM-o, and UTMOS. Notably, x-prediction *without* mel supervision is worse than v-prediction—mel supervision is the igniter for x-prediction.
3. Multi-scale mel supervision: a perceptual "ear"
Mel supervision exists only during training and is fully removed at inference. It acts as a perceptual compass, not a generation target: 7 mel scales (window sizes 32–2048, hop 1/4, 5–320 mel bins), log-mel L1 loss, λ_mel=0.05, applied only on masked regions.
Waveform space is physical; mel space is perceptual. Direct physical-space optimization can hit local optima (e.g., scrambled phase with roughly correct spectral shape). This is a classic perceptual loss strategy, here systematically introduced into waveform TTS for the first time.
4. Signal-noise variance alignment + noise-shifted scheduling
Standard diffusion uses a standard Gaussian prior (σ=1), but raw speech has tiny std: ~0.12 on Emilia, 0.07 on LibriTTS—about two orders of magnitude below the noise. With linear interpolation, the signal drowns in noise (Log-SNR shifted down ~20dB), like recognizing faces in a blizzard.
Fix: scale factor k=9. Clean waveforms are amplified 9× during training (σ≈1, matching the noise), inverse-scaled by 1/9 at inference; mel loss compares x₁ with x_θ/k.
Ablation (Table 4):
Training uses LogitNormal(μ=-0.8, σ=0.8) biased toward high-noise regions; inference uses PolyShift (p=2, s=3), outperforming Sway Sampling-style optimizations.
Results
Seed-TTS benchmark (test-en, 1,088 samples):
| Model | Type | Params | WER↓ | SIM-o↑ | UTMOS↑ | |-------|------|--------|------|--------|--------| | Ground Truth | - | - | 1.79 | 0.73 | 3.53 | | WavTTS | waveform | 673M | 1.50 | 0.65 | 3.92 | | MaskGCT | latent | 1048M | 2.36 | 0.71 | 3.57 | | E2-TTS | mel | 333M | 2.21 | 0.71 | 3.20 | | F5-TTS | mel | 336M | 1.65 | 0.66 | 3.73 | | ZipVoice | mel | 123M | 1.60 | 0.70 | 3.83 | | LongCat-AudioDiT | latent | 1420M | 1.94 | 0.76 | 3.80 |
Chinese benchmark (test-zh, 2,020 samples): WavTTS CER 1.59 / SIM-o 0.73 / UTMOS 3.08, behind LongCat-AudioDiT (1.10 / 0.81 / 3.16).
Key takeaways:
Discussion: Why can't SIM-o close the gap?
The author argues this is a structural cost of waveform space, not a temporary bottleneck: latent models pre-compress speaker identity into a low-dimensional, separable factor during autoencoder training, while in waveform space, speaker traits (pitch contour, formants, speaking habits) and semantics are fully entangled across raw samples—a much harder disentanglement problem for a fixed-capacity DiT.
But waveform space has unique advantages:
The SIM-o gap (0.06–0.11) is academically significant, but in practice WER 1.50% and UTMOS 3.92 may matter more—users first demand intelligibility and naturalness, then similarity. And SIM-o 0.65 already far exceeds earlier end-to-end waveform models (WaveGrad 2 was ~0.3–0.4).
Conclusion
WavTTS's biggest value is proving that direct waveform modeling is feasible and high-quality in zero-shot TTS. It needs no pretrained autoencoder, no neural vocoder, no mel predictor—text in, waveform out, shortest possible pipeline.
Its success rests on three stackable engineering insights:
1. Simple, effective patchification—turning a physics problem into an engineering one; 2. x-prediction + mel supervision—embedding perceptual constraints into the objective; 3. Signal-noise variance alignment—calibrating diffusion from noise chaos to learnable structure.
None of these rely on expensive compute tricks—they reflect deep understanding of the problem itself, which is what makes WavTTS reproducible as a blueprint for end-to-end teams. Future directions: larger models (scaling analysis shows data and model capacity must scale together), explicit speaker-feature modeling, and streaming inference.