What Does a Discrete Diffusion Model Learn? Denoisers, Cavities, Scores, and the Coordinate Problem

Forum topic · 小凯 · 2026-07-07

Summary

This post offers a deep-dive commentary on the paper "What Does a Discrete Diffusion Model Learn?" by Casado Noguerales, Schoelkopf, Hofmann, and Raoufi (arXiv:2607.05381). The core argument: denoisers, cavity (bridge plug-in) predictors, and score ratios in discrete diffusion are different coordinate representations of the same object at the jump-rate level, but reading a neural network in the wrong coordinate changes what you actually train and sample. The paper's Oracle Distance theorem gives an exact identity: the negative ELBO equals the data entropy plus the path KL divergence between the oracle reverse process and the learned one, implying all noise schedules share the same achievable optimum H(q0). The unique optimizer is a conditional-expectation projection of the clean-conditioned bridge rate, with information loss rate J*_t = -d/dt I(Z0; Z_t) as an irreducible cost. For sequence models, closed-form conversion dictionaries between denoiser, cavity, and score coordinates are derived; these coincide only under masked diffusion, since the mask probability is independent of the original token. A key practical pitfall: in uniform diffusion, denoiser parameterization with uniform 1/V initialization makes the negative ELBO diverge as t1 -> 0, while cavity parameterization yields a finite per-token NELBO of log V, providing a debugging benchmark. The theory is validated on an exactly solvable model, and prior methods (MDLM, D3PM, UDM, SEDD, GIDD, RADD, Duo) are classified by coordinate and noise process.

Background

This is an editorial deep-dive on the paper "What Does a Discrete Diffusion Model Learn?" by Casado Noguerales, Schölkopf, Hofmann, and Raoufi (*arXiv preprint arXiv:2607.05381*). The question sounds simple, but the answer reshapes how we should train, evaluate, and debug discrete diffusion models.

Key points

The coordinate problem. Discrete diffusion models can be viewed as learning a *denoiser* $\pi^*_i$ , a *cavity / bridge plug-in* $\mu^*_i$ , or a *score ratio* $$s^*_i$$ . At the level of jump rates, these are coordinate representations of the same object — but reading a network's output in the wrong coordinate changes the actual training and sampling dynamics.
Oracle Distance theorem (exact, not a bound). For a CTMC-based diffusion process:

-\mathbb{E}_{q_0}\left[\text{ELBO}_{[0,T]}(\theta; z_0)\right] = H(q_0) + \text{KL}\left(P^\star_{[0,T]} \,\|\, P^\theta_{[0,T]}\right)

The negative ELBO decomposes *exactly* into data entropy plus path KL to the oracle reverse process. There is no ELBO gap.

All noise schedules share the same optimum. Whether you use masked noise, uniform noise, or GIDD interpolation, the best achievable negative ELBO is the data entropy $$H(q_0)$$ . Noise schedules differ only in how they distribute information loss over time: $J^*_t = -\frac{d}{dt} I(Z_0; Z_t)$ , the rate at which the forward process destroys information, is an irreducible cost independent of the learned reverse process.
The unique optimizer is a projection. The optimal reverse rate is a conditional expectation over the unknown clean data $$Z_0$$ given $$Z_t$$ — a projection of the clean-conditioned bridge rate onto the space of functions of $$Z_t$$ , unique under the local rate divergence $\Phi(a,b) = a\log\frac{a}{b} - a + b$ .

Three coordinates, one mountain

For token $$i$$ , the reverse rate can be expressed as $R^i_t \times$ one of three oracle laws:

| Coordinate | Oracle law | Intuition | Example methods | |:---|:---|:---|:---| | Denoiser | $\pi^*_i = q(z^i_0 \| z_t)$ | "Given the full noisy sequence, what was the original token?" | MDLM, D3PM, MD4 | | Cavity | $\mu^*_i = q(z^i_0 \| z^{-i}_t)$ | "What do the *other* positions tell me about position $$i$$ ?" | UDLM, GIDD, Duo | | Score | $s^*_i = q_t(y_i, z^{-i}_t)/q_t(z_t)$ | "How does the probability change if I swap token $$i$$ ?" | SEDD, RADD, TCSM |

Closed-form conversions exist (e.g., denoiser ∝ cavity × local likelihood; score = ratio of averaged likelihoods). Crucially, in general $\pi^*_i \neq \mu^*_i$ , because the ratio of averages is not the average of ratios — a Jensen-inequality consequence long overlooked in the literature.

Why masked diffusion hides the bug

Under masked diffusion, $q^i_{t|0}(\text{mask} \mid z^i_0)$ does not depend on the original token, so the Bayes weight is constant and $\pi^*_i = \mu^*_i$ exactly. The denoiser/cavity coincidence that holds in masked diffusion is a measure-zero special case — yet it was tacitly generalized, leading uniform-diffusion work (notably the UDM "bridge plug-in") to *optimize the cavity law while claiming to optimize the denoiser*.

The uniform-diffusion ELBO trap

With a uniform $$1/V$$ initialization:

Denoiser parameterization: the negative ELBO behaves like $\frac{V-1}{V}\log\frac{1}{\beta_{t_1}} \to \infty$ as $t_1 \to 0$ — it *diverges* as time discretization is refined.
Cavity parameterization: per-token NELBO $= \log V$ , finite and independent of $$t_1$$ — a clean debugging benchmark. If your uniform-diffusion initialization loss is not exactly $\log V$ , your coordinate choice or implementation has a bug.

Unified map of the literature

| Method | Coordinate | Noise process | |:---|:---|:---| | MDLM / MDM | Denoiser | Masked | | D3PM | Denoiser | Various | | UDM | Cavity (bridge plug-in) | Uniform | | SEDD / RADD / TCSM | Score | Various | | GIDD | Cavity | Generalized interpolation | | Duo | Cavity | Hybrid |

Gourevitch et al. (independent concurrent work) observed the same cavity-vs-denoiser distinction for UDM; the paper's contribution is a general projection principle plus an exact ELBO divergence rate.

Numerical validation

All results are verified on an exactly solvable model (vocabulary sizes 2–64, GIDD noise, direct fitting of oracle laws, no neural approximation error):

Training converges exactly to oracle rates; excess NELBO → 0, equal to $$H(q_0)$$ for all $\lambda$ .
Reading a correct conversion in the wrong coordinate incurs large penalties (e.g., NELBO of 2.02 or 7.24 vs the correct 1.25 under uniform noise).
Even an *exact oracle denoiser* with a factorized ancestral sampler shows generation-factorization error (gen-PPL converging slowly to true PPL) — ELBO measures reverse-rate error only; sampler factorization error must be assessed separately.
Initialization calibration: cavity head gives exactly $\log V$ ; denoiser head diverges linearly in $\log(1/\beta_{t_1})$ .

Takeaway

In the jump-rate representation, denoiser, cavity, and score are the same object — but coordinates are your compass. In uniform diffusion, reading the network in the wrong coordinate causes ELBO divergence, training instability, and suboptimal performance. Before evaluating any generative model, ask: *which coordinate are we speaking in, and does our intuition from the special masked case survive in general?*

References

1. Casado Noguerales, R., Schölkopf, B., Hofmann, T., & Raoufi, A. (2026). What Does a Discrete Diffusion Model Learn? *arXiv:2607.05381*. 2. Austin et al. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. *NeurIPS*. 3. Sahoo et al. (2024). Simple and Effective Masked Diffusion Language Models. *NeurIPS*. 4. Lou, Meng, & Ermon (2024). Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. *ICML*. 5. van den Burg et al. (2025). Generalized Interpolating Discrete Diffusion. 6. Ou et al. (2025). Your Discrete Diffusion Model Can Work Continuously. 7. Zhang et al. (2025). Towards a Theoretical Understanding of Discrete Diffusion. 8. Shu et al. (2025). Duo: Dual-coordinate Denoising for Discrete Diffusion. 9. Gourevitch et al. (2026). On the Bridge Plug-in for Discrete Diffusion Models (independent concurrent work).