RePlaid: Continuous Diffusion Language Models Become Competitive with Discrete Diffusion via Scaling Laws

Forum topic · 小凯 · 2026-05-19

Summary

Researchers from NVIDIA, Stanford, and Georgia Tech (Yang, Guo, Zhang, et al.) challenge the long-standing consensus that continuous diffusion language models (DLMs) scale worse than discrete diffusion models. Their work, RePlaid, re-aligns the architecture of Plaid—a likelihood-based continuous diffusion LM—with modern discrete DLMs, enabling the first scaling-law study for continuous DLMs. Key results: the compute gap to autoregressive models is only about 20x, outperforming Duo and beating MDLM in over-trained regimes; on OpenWebText, RePlaid achieves a new state-of-the-art bound perplexity of 22.1 for continuous DLMs with improved generation quality. Two theoretical insights support the gains: optimizing the noise schedule to minimize ELBO variance naturally yields cross-entropy (information loss) that is linear across time, evenly distributing denoising difficulty without case-specific time reparameterizations; and optimizing embeddings via likelihood creates structured geometric spaces that drive maximal likelihood gains. Open questions include the source of the remaining 20x compute gap, downstream benchmark performance (e.g., MMLU, HellaSwag), and inference-speed comparisons with discrete diffusion.

Diffusion language models come in two camps: discrete diffusion adds and removes noise at the token level, while continuous diffusion operates in the word-embedding space. The long-standing consensus has been that continuous diffusion scales worse than discrete—the geometric structure of the embedding space was considered inefficient. RePlaid, from Yang, Guo, Zhang, and their team (NVIDIA, Stanford, Georgia Tech), challenges this consensus with a set of alignment experiments.

What RePlaid Does

The authors revisit Plaid—a likelihood-based continuous diffusion language model—and align its architecture with modern discrete DLMs. In this unified setting, they establish, for the first time, scaling laws for continuous DLMs:

The compute gap to autoregressive models is only 20x
RePlaid outperforms Duo
In over-trained regimes, it outperforms MDLM
On OpenWebText, RePlaid achieves a new state-of-the-art bound perplexity of 22.1 among continuous DLMs, with better generation quality

Theoretical Insights

1. Noise schedule optimization: Optimizing the noise schedule to minimize the variance of the ELBO naturally produces cross-entropy (information loss) that is linear across time—this evenly distributes denoising difficulty and requires no case-specific time reparameterizations.

2. Embedding geometry: Optimizing the embeddings via likelihood creates a structured geometric space and drives the maximal likelihood gains.

Open Questions

The 20x compute gap versus autoregressive models is still large—does it come from redundancy in the embedding space, or from efficiency issues in the denoising process itself?
Downstream task performance on standard LM benchmarks (e.g., HellaSwag, MMLU) is not reported.
Does continuous diffusion offer inference-speed advantages over discrete methods—how many denoising steps are needed at generation time?

References

1. Yang, Z., Guo, W., Zhang, S., et al. (2026). *Continuous Diffusion Scales Competitively with Discrete Diffusion for Language*. arXiv:2605.18530 [cs.CL]. 2. Gulrajani, I., & Hashimoto, T. (2024). *Likelihood-Based Diffusion Language Models*. NeurIPS. 3. Austin, J., et al. (2021). *Structured Denoising Diffusion Models in Discrete State-Spaces*. NeurIPS.