SEDD: Teaching Diffusion Models to Write Text - Score Entropy Discrete Diffusion Explained

Forum topic · 小凯 · 2026-02-26

Summary

SEDD (Score Entropy Discrete Diffusion) is a Stanford research breakthrough, awarded Best Paper at ICML 2024, that extends diffusion models from images to discrete text generation. The core obstacle is that score matching requires gradients, which do not exist over discrete tokens. SEDD instead learns the 'concrete score'—ratios of probabilities between neighboring sentences (Hamming distance one)—using a novel Score Entropy loss that avoids the intractable normalization constant Z. Training uses a denoising variant where ratios of known noising transitions are learnable, and sampling reverses a continuous-time Markov chain that progressively masks tokens. Compared with GPT-2, SEDD matches or beats perplexity on LAMBADA, WikiText-2, PTB, and WikiText-103, cuts perplexity 25-75% versus prior discrete diffusion models, and achieves comparable MAUVE scores. Unlike autoregressive models, it supports global context, flexible compute-quality trade-offs, and zero-shot infilling/suffix prompting. The technology was commercialized by Inception Labs (founded by Stefano Ermon, Aaron Lou, and others; backed by NVIDIA's NVentures, Microsoft's M12, and investors including Andrew Ng and Andrej Karpathy) as the Mercury and Mercury 2 diffusion LLMs, reaching 1009 tokens/s.

SEDD: From Image Generation to Text Generation with Diffusion Models

Diffusion models taught AI to paint (Midjourney) and make videos (Sora), so why have they lagged behind GPT for text generation? The answer lies in the mathematical gap between continuous and discrete data. In 2023, a Stanford paper introduced SEDD (Score Entropy Discrete Diffusion), which crossed this gap and won the ICML 2024 Best Paper Award. The technology has since been commercialized as the core engine of the Mercury 2 large language model.

Key points

The problem: Images are continuous (pixel values), while text is discrete (token indices). The score function ∇ₓ log p(x) requires a gradient, which does not exist over discrete tokens — like navigating a maze where you can only jump between adjacent rooms, not take small steps in any direction.
Concrete Score: Instead of modeling absolute probabilities p(x), SEDD models probability ratios p(y)/p(x) between neighboring sentences (differing in one position, i.e., Hamming distance 1). The ratio cancels the intractable normalization constant Z (for language models, Z would involve ~50257^1024 terms).
Score Entropy loss: A new objective Σ [s_θ(x)_y - (p(y)/p(x)) · log s_θ(x)_y] that learns arbitrary ratios without normalization, includes a logarithmic barrier preventing negative ratios, and is convex for stable optimization. A denoising variant makes it practical: sample from a simple distribution, apply known noising transitions, and learn the known conditional ratios.
Discrete diffusion process: The forward process is a continuous-time Markov chain (CTMC), dp_t/dt = Q · p_t, where tokens progressively turn into masks. The reverse process uses reverse transition probabilities proportional to p_t(y)/p_t(x) · Q(y,x) — exactly the learned concrete score. Sampling starts from pure noise and iteratively denoises with a neural network.

Why it beats autoregressive generation

1. Global context: GPT generates left-to-right; SEDD fills all positions simultaneously, seeing the full context and producing more coherent text. 2. Controllable compute-quality trade-off: 50 sampling steps for highest quality, 20 steps for 2.5x speed, 10 steps for 5x speed — flexible for production latency budgets. 3. Flexible prompting: Standard (left) prompting, infilling (fill the middle), and suffix prompting all work zero-shot.

Experimental results (vs GPT-2)

Perplexity: SEDD matches or beats GPT-2 on WikiText-2 (≤41.84 vs 42.43 small; ≤31.04 vs 31.80 medium), PTB (≤114.24 vs 138.43; ≤87.12 vs 123.14), and WikiText-103 (≤40.62 vs 41.60; ≤29.98 vs 31.39); GPT-2 remains ahead on LAMBADA.
Generation quality (MAUVE): SEDD 0.957 vs GPT-2 + nucleus sampling 0.955.
SEDD reduces perplexity 25–75% vs prior discrete diffusion models and achieves 6–8x better generative perplexity than unannealed GPT-2.

From paper to product: Mercury

The founding team includes Stefano Ermon (Stanford professor, CEO), Aaron Lou (SEDD first author), Chenlin Meng (Pika co-founder), Aditya Grover (UCLA), and Volodymyr Kuleshov (Cornell). Inception Labs raised $50M in November 2024 from NVentures (NVIDIA), M12 (Microsoft), and Menlo Ventures, with angel investors including Andrew Ng and Andrej Karpathy.

Timeline:

Feb 2025: Mercury launched — the first commercial-grade diffusion LLM
Feb 2025: Mercury Coder released
Feb 2026: Mercury 2 released — deep reasoning support, 1009 tokens/s

Limitations and outlook

Limitations: higher per-inference cost from multiple forward passes; Mercury 2's scale still trails top autoregressive models; immature tooling and ecosystem.

Future directions: scaling to GPT-4-class parameter counts, unified multimodal diffusion generation, real-time applications (voice assistants, live translation/captions), and open-source ecosystem growth.

Conclusion

SEDD's contributions in three sentences: (1) the Concrete Score replaces probabilities with ratios, bypassing the normalization constant; (2) Score Entropy extends score matching to discrete spaces; (3) it makes discrete diffusion genuinely viable for text. Autoregression is not the only answer to text generation — as AI shifts from "typewriter" to "editor," we may be witnessing the birth of the next generation of language models.

References

SEDD paper: https://arxiv.org/abs/2310.16834
Official blog: https://louaaron.github.io/blog/2024/discrete-diffusion/
Mercury 2 blog: https://www.inceptionlabs.ai/blog/introducing-mercury-2
GitHub: https://github.com/louaaron/Score-Entropy-Discrete-Diffusion
ICML 2024: *Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution* (Oral + Best Paper Award)