Diffusion Models Train in High-Dimensional Space, But the Curse of Dimensionality May Not Apply

Diffusion models are trained in 1024×1024 pixel space — roughly 3 million dimensions. But from an information perspective, a photo of a face is not "3 million random numbers"; it is constrained by a few dozen intrinsic degrees of freedom: head pose, lighting direction, and several expression dimensions. Data appears to live in a high-dimensional space, but is actually compressed onto a low-dimensional manifold.

This is the core intuition of a recent theoretical paper by Fu, Suzuki, Lee, and Nitanda on arXiv (2605.15822). They prove an important theoretical result: the convergence rate of diffusion models is determined by the data's intrinsic dimension, not the dimension of the space it is packed into.

🏔️ The Manifold Hypothesis

The manifold hypothesis — that high-dimensional data actually lies on a low-dimensional manifold — is not new in machine learning. But it is usually treated as an intuitive assumption, with few rigorous arguments of the form "if the manifold hypothesis holds, how does this algorithm's complexity change?"

This paper targets score matching, the core training algorithm of diffusion models. Diffusion models train a neural network to estimate the gradient of the log probability density (the score function), then use this estimate to reverse the noising process, gradually recovering data from noise.

The problem: estimating a probability density gradient in a 3-million-dimensional space sounds like a nightmare — the curse of dimensionality would make the required sample size grow exponentially with dimension. So why do diffusion models work so well?

The paper's answer: because the true probability density is only supported on a low-dimensional manifold, and you only need good estimates near that manifold. In other directions, high precision is unnecessary.

📐 Two Noise Regimes

The core of their theoretical analysis splits noise levels into two regimes.

• When noise is large — early in the generation process — the score function is dominated by "tangential units." Large noise spreads the distribution around data points across the local structure of the entire manifold; at this stage, you need to capture directional information about the manifold.
• When noise is small — near the end of generation — the score function is dominated by "projection-centered de-Gaussianized Laplacians." Small noise means the data distribution closely follows the true manifold's shape, and you must precisely characterize the residual distribution after projection onto the manifold.

These two regimes require different analytical tools. Their clever technique: instead of approximating the manifold projection as a black-box high-dimensional smooth function (which would reintroduce dimension dependence), they use a finite set of intrinsic anchors and Gauss-Newton iterations to implement nearest-projection coordinates with a ReLU network.

📊 Convergence Rate

Their main rate is:

ø(D^O_β(d) · n^{-(β+1)/(d+2β)})

where:

• D is the ambient dimension (3,000,000)
• d is the intrinsic dimension (possibly 30–100)
• n is the sample size
• β is the smoothness of the probability density on the manifold

The key point is that d — the intrinsic dimension — appears in the exponent of the convergence rate. Algebraically, the rate is a negative power of n, with the exponent determined by intrinsic dimension d and smoothness β. This means more training samples continue to improve convergence at a reasonable rate, governed mainly by the data's true complexity (intrinsic dimension), not the dimension of the packaging space.

The ambient dimension D only appears in the polynomial coefficient basis (the D^O_β(d) part of the ø(D^...) notation), not in the exponent. The effect of dimension is polynomial, not exponential. For practitioners, this is the mathematical explanation of why diffusion models still work on high-resolution images.

🤷 Unclear Points

Several things remain uncertain:

1. The result holds for d > 2 compact smooth manifolds — so it may not directly apply to curves (d=1) or surfaces (d=2). Most real image data likely has intrinsic dimension 20–50 for 1024×1024 face images, so d > 2 usually holds. But more regular geometric structures — like manifolds of handwritten digits — may have lower dimensionality; are there boundary cases not covered? I'm not sure.

2. How does score-estimation quality degrade far from the actual manifold? The paper analyzes two noise regimes, but the off-manifold region — where data points are near no manifold at all — may not be fully covered. Training diffusion models in low-density regions is a well-known practical difficulty (manifold overfitting); how much does this theoretical analysis help understand low-density behavior? I'm not entirely sure.

3. Does real image data roughly satisfy the assumption of a "compact d-dimensional smooth manifold embedded in [0,1]^D"? Real images may have more complex topology — holes, intersections, regions of varying dimension. The manifold framework is theoretically clean, but how closely its assumptions match real data is beyond my judgment.

For a mind accustomed to the "curse of dimensionality," this paper offers a comforting mathematical guarantee: diffusion models work not because they overcome the curse of high-dimensional space, but because the data simply isn't there.

References

1. Fu, G., Suzuki, T., Lee, W. S., & Nitanda, A. (2026). *Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds*. arXiv:2605.15822 [cs.LG]. https://arxiv.org/abs/2605.15822 2. Song, Y., & Ermon, S. (2019). *Generative Modeling by Estimating Gradients of the Data Distribution*. NeurIPS 2019. 3. Ho, J., Jain, A., & Abbeel, P. (2020). *Denoising Diffusion Probabilistic Models*. NeurIPS 2020. 4. Fefferman, C., Mitter, S., & Narayanan, H. (2016). *Testing the Manifold Hypothesis*. Journal of the American Mathematical Society, 29(4), 983–1049. 5. Bickel, P. J., & Li, B. (2007). *Local Polynomial Regression on Unknown Manifolds*. IMS Lecture Notes–Monograph Series, 54, 177–186.