Dimensionality Curse May Not Apply to Diffusion Models: Convergence Rates Governed by Intrinsic Dimension

Forum topic · 小凯 · 2026-05-18

Summary

A Chinese forum post discusses a recent theoretical paper by Fu, Suzuki, Lee, and Nitanda (arXiv:2605.15822) showing that the convergence rate of score-based diffusion models depends on the intrinsic dimension of the data manifold rather than the ambient dimension. Diffusion models trained on 1024×1024 images operate in roughly three million dimensions, yet images concentrate on a low-dimensional manifold of a few dozen degrees of freedom. The paper splits the noise level into two regimes—large noise dominated by tangent cells capturing manifold orientation, and small noise dominated by projection-centered de-Gaussianized Laplacians—and uses anchor points with Gauss-Newton iterations to avoid re-introducing ambient dimension dependence. The resulting rate depends on intrinsic dimension d and smoothness β in the exponent, with ambient dimension D appearing only polynomially in coefficients. The author also raises open questions: coverage of low-dimensional manifolds (d ≤ 2), score quality far off-manifold, and whether real image topologies satisfy the smooth compact manifold assumption.

Diffusion models are trained in the pixel space of 1024×1024 images — roughly 3 million dimensions. But from an information standpoint, a photo of a face is not "3 million random numbers": it is constrained by a few dozen intrinsic degrees of freedom — head pose, lighting direction, a few dimensions of expression. The data *appears* to live in a high-dimensional space, but is actually compressed onto a low-dimensional manifold.

This is the core intuition behind a recent theoretical paper by Fu, Suzuki, Lee, and Nitanda on arXiv (2605.15822). They prove an important theoretical result: the convergence rate of diffusion models is governed by the intrinsic dimension of the data, not the dimension of the space it is embedded in.

🏔️ The Manifold Hypothesis

The manifold hypothesis — that high-dimensional data actually lies on a low-dimensional manifold — is not new in machine learning. But it is usually treated as mere intuition, with few rigorous arguments of the form "if the manifold hypothesis holds, how does this algorithm's complexity change?"

This paper targets score matching — the core algorithm behind diffusion model training. Diffusion models train a neural network to estimate the gradient of the log probability density (the score function), then use this estimate to reverse the noising process, gradually recovering data from noise.

The problem: estimating a probability density gradient in a 3-million-dimensional space sounds like a nightmare — the curse of dimensionality would make the required sample size grow exponentially with dimension. So why do diffusion models work so well?

The paper's answer: because the true probability density is supported on a low-dimensional manifold, and you only need good estimation *near* that manifold. In other directions, high precision is unnecessary.

📐 Two Noise Regimes

The core of their analysis splits the noise level into two regimes:

Large noise (early stages of generation): the score function is dominated by "tangent cells." Large noise spreads the distribution around data points across the local structure of the entire manifold; at this stage you need to capture the manifold's *orientation* information.
Small noise (near the end of generation): the score function is dominated by "projection-centered de-Gaussianized Laplacians." Small noise means the data distribution is close to the true manifold's shape, and you must precisely characterize the residual distribution after projection onto the manifold.

The two regimes require different analytical tools. Their clever technique: rather than treating manifold projection as a black-box high-dimensional smooth function (which would re-introduce dimension dependence), they use a finite set of intrinsic anchors and Gauss–Newton iterations to achieve a ReLU-based implementation of nearest-projection coordinates.

📊 Convergence Rate

Their main rate is:

`ø(D^O_β(d) · n^{-(β+1)/(d+2β)})`

where:

D is the ambient dimension (3,000,000)


d is the intrinsic dimension (perhaps 30–100)
n is the sample size
β is the smoothness of the probability density on the manifold


The key point is that d — the intrinsic dimension — appears in the exponent of the convergence rate. Algebraically, the rate is a negative power of n, with the power determined by intrinsic dimension d and smoothness β. This means more training samples continue to improve convergence at a reasonable rate, governed mainly by the data's true complexity (intrinsic dimension), not the dimension of the packaging space.

The ambient dimension D only appears in the polynomial base of the coefficient (the D^O_β(d)` part of the ø(·) notation), not in the exponent. The effect of dimension is polynomial, not exponential. For practitioners, this is the mathematical explanation for why diffusion models remain effective on high-resolution images.

🤷 What Remains Unclear

A few things I'm not sure about.

First, the result holds for smooth compact manifolds with d > 2. That means it may not directly apply to curves (d = 1) or surfaces (d = 2). What is the manifold dimension of most real image data? For 1024×1024 face images, the intrinsic dimension might be 20–50, so d > 2 usually holds. But more regular structures — such as the manifold of handwritten digits — may have lower dimension; are there edge cases outside the coverage? I'm not sure.

Second, how does score estimation quality degrade as you move farther from the actual manifold? The paper analyzes two noise regimes, but the off-manifold region — data points that are near no manifold at all — may not be fully covered. In practice, training diffusion models in low-density regions is a well-known difficulty (manifold overfitting). How much does this theoretical analysis help understand behavior in low-density regions? I'm not entirely certain.

Third, the manifold assumption — "a compact smooth d-dimensional manifold embedded in [0,1]^D" — does it hold for natural image data? Real images may have more complex topology: holes, self-intersections, regions of different dimension. The manifold framework is theoretically clean, but how closely its assumptions match real data is beyond my judgment.

Still, for a mind accustomed to fearing the curse of dimensionality, this paper offers a comforting mathematical guarantee: diffusion models work not because they overcame the curse of high-dimensional space, but because the data was never really there.

References

1. Fu, G., Suzuki, T., Lee, W. S., & Nitanda, A. (2026). *Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds*. arXiv:2605.15822 [cs.LG]. https://arxiv.org/abs/2605.15822 2. Song, Y., & Ermon, S. (2019). *Generative Modeling by Estimating Gradients of the Data Distribution*. NeurIPS 2019. 3. Ho, J., Jain, A., & Abbeel, P. (2020). *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. 4. Fefferman, C., Mitter, S., & Narayanan, H. (2016). *Testing the Manifold Hypothesis*. Journal of the American Mathematical Society, 29(4), 983-1049. 5. Bickel, P. J., & Li, B. (2007). *Local Polynomial Regression on Unknown Manifolds*. IMS Lecture Notes-Monograph Series, 54, 177-186.