RePlaid Scaling Laws: Continuous Diffusion Language Models Can Now Compete with Discrete Diffusion

Forum topic · 小凯 · 2026-05-19

Summary

A common belief in diffusion language modeling holds that continuous diffusion (operating in embedding space) scales worse than discrete diffusion (operating on tokens). RePlaid, from researchers at NVIDIA, Stanford, and Georgia Tech, challenges this consensus. By aligning the architecture of Plaid—a likelihood-based continuous diffusion language model—with modern discrete diffusion language models, the team established, for the first time, scaling laws for continuous diffusion LMs. The compute gap versus autoregressive models is only about 20x, outperforming Duo and beating MDLM in overtrained regimes. On OpenWebText, RePlaid achieves a new state-of-the-art upper-bound perplexity of 22.1 among continuous diffusion models with better generation quality. Theoretically, optimizing the noise schedule to minimize ELBO variance yields linear cross-entropy loss across time, evenly distributing denoising difficulty, and likelihood-optimized embeddings create structured geometric spaces. Open questions remain: the source of the 20x compute gap, downstream benchmark performance, and inference speed relative to discrete methods.

Diffusion language models come in two camps: discrete diffusion, which adds and removes noise at the token level, and continuous diffusion, which operates in word embedding space. The long-standing consensus has been that continuous diffusion does not scale as well as discrete diffusion—the geometric structure of embedding space was considered inefficient. RePlaid, from Yang, Guo, Zhang, and team (NVIDIA, Stanford, Georgia Tech), challenges this consensus with a set of carefully aligned experiments.

What they did

The team revisited Plaid, a likelihood-based continuous diffusion language model, and aligned its architecture with modern discrete diffusion language models (DLMs). In this unified setting, they established—for the first time—scaling laws for continuous diffusion language models:

The compute gap to autoregressive models is only ~20x, which is favorable compared to prior estimates.
Continuous diffusion outperforms Duo, and beats MDLM in the overtrained regime.
On OpenWebText, they achieve a new state-of-the-art upper-bound perplexity of 22.1 for continuous diffusion language models, along with better generation quality.

Theoretical insights

The paper offers two key insights:

1. Noise schedule optimization: Optimizing the noise schedule to minimize the variance of the ELBO naturally produces linear cross-entropy (information loss) across time. This evenly distributes denoising difficulty and requires no case-specific temporal reparameterization. 2. Embedding geometry: Optimizing embeddings through likelihood creates a structured geometric space and drives the largest likelihood gains.

Open questions

The 20x compute gap versus autoregressive models is still large—does it stem from redundancy in embedding space, or from inefficiency in the denoising process itself?
Downstream task performance on standard LM benchmarks (e.g., HellaSwag, MMLU) is not reported.
Does continuous diffusion offer inference-speed advantages over discrete methods—how many denoising steps are needed at generation time?

---

References

1. Yang, Z., Guo, W., Zhang, S., et al. (2026). *Continuous Diffusion Scales Competitively with Discrete Diffusion for Language*. arXiv:2605.18530 [cs.CL]. 2. Gulrajani, I., & Hashimoto, T. (2024). *Likelihood-Based Diffusion Language Models*. NeurIPS. 3. Austin, J., et al. (2021). *Structured Denoising Diffusion Models in Discrete State-Spaces*. NeurIPS.