What Does a Discrete Diffusion Model Learn? Denoiser, Cavity, and Score as Coordinate Systems

This is an English translation/summary of a Chinese forum post analyzing the paper *"What Does a Discrete Diffusion Model Learn?"* by Casado Noguerales, Schölkopf, Hofmann, and Raoufi (arXiv:2607.05381).

Key points

• The coordinate question. Discrete diffusion models can be described as learning a denoiser, a cavity (bridge plug-in), or a score ratio. At the level of jump rates, these are different coordinate representations of the same object—but reading a neural network in the wrong coordinate changes the training and sampling processes themselves.
• Oracle Distance theorem (exact, not a bound). For a CTMC diffusion process, the negative ELBO decomposes exactly:
\[-\mathbb{E}_{q_0}\left[\text{ELBO}_{[0,T]}(\theta; z_0)\right] = H(q_0) + \text{KL}\left(P^\star_{[0,T]} \,\|\, P^\theta_{[0,T]}\right)\]

There is no variational gap. Consequently all noise processes share the same optimal achievable negative ELBO: the data entropy \(H(q_0)\)—whether masked, uniform, or GIDD interpolating noise.

• Unique optimizer as a projection. The optimal reverse rate is the conditional expectation of the clean-conditioned bridge rate given the noisy state:
\[\bar{Q}^{\theta^*}_t(z_t, y) = \mathbb{E}\left[\bar{Q}_t(Z_t, y \mid Z_0) \mid Z_t = z_t\right]\]

This is a projection (unique under the divergence \(\Phi(a,b) = a\log\frac{a}{b} - a + b\)). The irreducible cost per time step is \(J^*_t = -\frac{d}{dt} I(Z_0; Z_t)\), the rate at which the forward process destroys information about the data.

• Three exact coordinates. For token-decomposable noise processes, the reverse rate can be expressed via:

| Coordinate | Oracle law | Intuition | Methods | |:---|:---|:---|:---| | Denoiser | \(\pi^*_i = q(z^i_0 \| z_t)\) | "Given the noisy sequence, what was the original token?" | MDLM, D3PM, MD4 | | Cavity | \(\mu^*_i = q(z^i_0 \| z^{-i}_t)\) | "What do the other positions tell me?" | UDLM, GIDD, Duo | | Score | \(s^*_i = q_t(y_i, z^{-i}_t)/q_t(z_t)\) | "How does probability change if I swap token \(i\)?" | SEDD, RADD, TCSM |

Closed-form conversion dictionaries exist between them. Crucially, \(\pi^*_i \neq \mu^*_i\) in general because the ratio of averages does not equal the average of ratios (Jensen).

• Why masked diffusion hid the problem. In masked diffusion, the mask probability is independent of the original token, so the Bayes weight \(q^i_{t|0}(z^i_t \mid \cdot)\) is constant and denoiser = cavity exactly. This measure-zero coincidence was mistaken for a general fact and carried into uniform diffusion, where it fails.
• Uniform diffusion ELBO blow-up. With a standard uniform (\(1/V\)) initialization, denoiser parameterization gives a negative ELBO that diverges as \(\frac{V-1}{V}\log\frac{1}{\beta_{t_1}} \to \infty\) when the time-window lower bound \(t_1 \to 0\). Cavity parameterization instead gives a finite per-token NELBO of exactly \(\log V\)—a clean debugging benchmark: if your initialization loss is not \(\log V\), your coordinate choice or implementation has a bug.
• Taxonomy of the literature. MDM/MDLM and D3PM use denoiser coordinates; UDM, GIDD, and Duo use the cavity (bridge plug-in); SEDD, RADD, and TCSM use score coordinates. The UDM "bridge plug-in" optimizes the cavity law, not the denoiser—an error without consequence for masked diffusion but substantial for uniform. An independent concurrent work (Gourevitch et al.) observed the same special case; this paper unifies it under a general projection principle with exact ELBO divergence rates.
• Numerical verification. All results are verified on exactly solvable GIDD models (fitting the oracle law directly, no network approximation error): correct coordinate conversion yields NELBO \(= H(q_0)/L\) in every case, while misreading a denoiser head as cavity or score under uniform noise inflates NELBO to 2.02 or 7.24 (vs. 1.25). Even with an exact oracle denoiser, a factorized ancestral sampler introduces separate generation-PPL error that shrinks with more sampling steps (slower for masked diffusion, which cannot self-correct).

Takeaway

In the level of jump rates, denoiser, cavity, and score are the same object—but coordinates are your compass. The paper's practical warning: before evaluating any generative model, ask which coordinate you are in, what you are actually optimizing, and whether your intuitions hold only in a special case.

Reference

Casado Noguerales, R., Schölkopf, B., Hofmann, T., & Raoufi, A. (2026). *What Does a Discrete Diffusion Model Learn?* arXiv preprint arXiv:2607.05381.