Back to Basics: Kaiming He's Manifold-Aware Signal Estimation Unifies Denoising and Image Generation

Forum topic · ✨步子哥 · 2025-11-22

Summary

This post from zhichai.net provides an in-depth commentary on the November 2025 arXiv preprint 'Back to Basics: Unifying Denoising and Generation via Manifold-Aware Signal Estimation' (arXiv:2511.13720) by Kaiming He and the Meta AI team. The paper argues that diffusion models like DDPM take an unnecessary detour by predicting added noise rather than directly estimating the clean image. Grounded in the manifold hypothesis—natural images occupy a low-dimensional surface in high-dimensional pixel space—direct signal prediction is more efficient than high-dimensional noise prediction. The authors introduce JiT (Just image Transformers), a plain Transformer that predicts the clean image x0 directly from a noisy input via x0_hat = f_theta(x_t, t), trained with mean-squared-error loss and sampled with deterministic DDIM-style steps. Notably, JiT works best with large patch sizes (16x16, 32x32), the opposite of noise-prediction models, which degrade with large patches. Reported results include FID ~2.5 on ImageNet 256x256 generation and stable performance at 512x512 where noise-prediction baselines collapse. The post discusses implications for high-resolution and video generation, limitations (text conditioning untested, possible extra sampling steps), and the philosophical bridge to classical signal processing and compressed sensing.

Introduction

This post reviews the November 2025 Meta AI preprint "Back to Basics: Unifying Denoising and Generation via Manifold-Aware Signal Estimation" (arXiv:2511.13720) by Kaiming He and colleagues. Its central message: instead of predicting added noise (the DDPM convention since 2020), generative models should return to the classical signal-processing goal of directly estimating the clean signal.

Key points

Noise prediction is a detour. Since DDPM, models learn to predict the noise ε added to images, then indirectly reconstruct x₀. The paper argues this violates the signal-processing principle of direct estimation and ignores the manifold hypothesis.
The manifold hypothesis. Natural images lie on a thin, low-dimensional surface within high-dimensional pixel space, whereas noise fills the entire volume. Predicting noise forces the model to learn an enormous space of high-dimensional noise variants; predicting the clean image keeps the model on the low-dimensional manifold.
The JiT architecture (Just image Transformers). A plain Transformer encoder implements the direct mapping:

\hat{x}_0 = f_\theta(x_t, t)

trained with an MSE loss $L = ||x_0 - \hat{x}_0||^2$ and sampled via deterministic DDIM-style updates. No U-Net, pretraining, codebooks, or auxiliary losses are needed.

Large patches work—inverting convention. Noise-prediction models need small patches (4×4, 8×8) and degrade with larger ones; JiT thrives with large patches (16×16, 32×32) because clean-signal prediction preserves global semantic structure rather than pixel-level noise detail.
Empirical results (as summarized in the post):

| Resolution | Patch | FID | IS | Params | |---|---|---|---|---| | 256×256 | 16×16 | 2.5 | 210+ | ~300M | | 256×256 | 32×32 | 2.8 | 205+ | ~150M | | 512×512 | 32×32 | 3.0 | 200+ | ~500M | | 512×512 | 16×16 | 2.7 | 215+ | ~400M |

Noise-prediction baselines reportedly collapse at high resolution with large patches (FID > 10), while JiT remains stable.

Scaling-law reversal. Because large patches reduce sequence length, attention cost grows with patch count rather than pixel count, making high-resolution generation far more memory-friendly than diffusion baselines.

JiT vs. traditional DDPM

| Aspect | DDPM (noise prediction) | JiT (clean-image prediction) | |---|---|---| | Target | Predict ε | Predict x₀ | | Manifold efficiency | High-dimensional; inefficient | Low-dimensional; efficient | | Patch size | Small patches required | Large patches beneficial | | Architecture | Complex (U-Net + attention) | Plain Transformer | | Sampling | Stochastic noise subtraction | Deterministic denoising steps | | ImageNet 256 FID | ~2–5 | ~2.5 | | High resolution | Memory-intensive | Scales well |

Limitations and open questions

Only class-conditional generation is tested; text conditioning (CLIP-style cross-attention) is unexplored.
Deterministic sampling may need more steps to refine fine details; distillation techniques (e.g., consistency-style methods) could compress steps.
Generalization beyond images (audio, text, video) remains unverified; the manifold assumption may not hold for all data types.
The paper concedes noise prediction's variance-injection advantage for stochastic/creative sampling; hybrid noise+signal prediction could combine strengths.

Implications

The post situates JiT in Kaiming He's lineage of simplification (ResNet, MAE) and suggests impact on video generation (shared Transformer over spatiotemporal patches), simplified DiT/UViT-style architectures, faster training convergence, and democratized high-resolution generation for small teams. It frames the work as a philosophical bridge to classical compressed sensing and Bayesian estimation: generation as reconstruction from undersampled data.

References

1. Back to Basics: Unifying Denoising and Generation via Manifold-Aware Signal Estimation — https://arxiv.org/abs/2511.13720 2. Meta AI blog on diffusion fundamentals 3. NeurIPS 2025 discussions on manifold assumptions in generative models