Key points
SEDD (Score Entropy Discrete Diffusion) is a Stanford research paper (arXiv 2023) that won the ICML 2024 Best Paper Award and was later commercialized as the core engine of Inception Labs' Mercury models. It solves the fundamental problem of applying diffusion models to discrete text data.
Why diffusion struggles with text
- Images are continuous (pixel values 0-255) and suit calculus-based score matching: ∇ₓ log p(x).
- Text is discrete (token indices); gradients don't exist on integers, so standard diffusion breaks down.
- Analogy: in continuous space you can take a small step in any direction; in discrete space you can only "jump" to neighboring states.
s_θ(x)_y ≈ p(y) / p(x)- The normalization constant Z cancels out (
p(y)/p(x) = e^{f(y)}/e^{f(x)}), avoiding an astronomically intractable computation for language models. - The Concrete Score tells you which "neighbor" sentence (differing in one position, Hamming distance 1) is more probable. 2. Score Entropy loss — extends cross-entropy to learn arbitrary ratios
- Global context: all positions are refined simultaneously rather than strictly left-to-right.
- Tunable compute-quality tradeoff: ~50 sampling steps for best quality, ~10 steps for ~5x speedup with acceptable quality.
- Flexible prompting: left-to-right completion, middle infilling, and suffix (right-to-left) prompting all work zero-shot.
- MAUVE: GPT-2 + nucleus sampling 0.955 vs SEDD standard prompting 0.957 (infill prompting 0.942).
- 25-75% perplexity reduction vs prior discrete diffusion models; 6-8x better generation perplexity than un-annealed GPT-2.
- Founders: Stefano Ermon (Stanford, CEO), Aaron Lou (SEDD first author), Chenlin Meng (Pika co-founder, advisor), Aditya Grover (UCLA), Volodymyr Kuleshov (Cornell).
- Nov 2024: $50M funding from NVentures (NVIDIA), M12 (Microsoft), Menlo Ventures; individual investors include Andrew Ng and Andrej Karpathy.
- Feb 2025: Mercury released (first commercial-scale diffusion LLM) plus Mercury Coder.
- Feb 2026: Mercury 2 released with deep reasoning support, up to 1009 tokens/s.
- Higher per-inference cost (multiple forward passes), smaller scale than top autoregressive models, and a less mature ecosystem/toolchain.
- Future directions: scaling to GPT-4-class sizes, unified multimodal diffusion generation, real-time applications, and open-source tooling.
- SEDD paper: https://arxiv.org/abs/2310.16834
- Official blog: https://louaaron.github.io/blog/2024/discrete-diffusion/
- Mercury 2 blog: https://www.inceptionlabs.ai/blog/introducing-mercury-2
- GitHub: https://github.com/louaaron/Score-Entropy-Discrete-Diffusion
- ICML 2024: "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution" (Oral + Best Paper Award)
SEDD's core innovations
1. Concrete Score — model probability *ratios* rather than absolute probabilities:p(y)/p(x) without normalization; log terms prevent negative ratios; the objective is convex.
3. Denoising Score Entropy — a practical training target: sample from a simple distribution, apply known forward transition probabilities, and learn the (known) conditional ratios.
4. Discrete diffusion via CTMC — tokens independently jump to other tokens per a rate matrix Q; the forward process gradually masks text, and the reverse process uses the learned ratios: reverse transition ∝ p_t(y)/p_t(x) · Q(y,x).Advantages over autoregressive (GPT-style) models
Experimental results (vs GPT-2)
| Dataset | GPT-2 Small | SEDD Small | GPT-2 Medium | SEDD Medium | |---------|-------------|------------|--------------|-------------| | LAMBADA | 45.04 | ≤50.92 | 35.66 | ≤42.77 | | WikiText-2 | 42.43 | ≤41.84 | 31.80 | ≤31.04 | | PTB | 138.43 | ≤114.24 | 123.14 | ≤87.12 | | WikiText-103 | 41.60 | ≤40.62 | 31.39 | ≤29.98 |
From paper to product (Inception Labs)
Limitations and outlook
Conclusion
SEDD's three contributions: (1) the Concrete Score, replacing probability modeling with ratio modeling; (2) Score Entropy, extending score matching to discrete spaces; (3) practical discrete diffusion for text. It demonstrates that autoregression is not the only path to text generation.