Back to Basics: Kaiming He's JiT Replaces Noise Prediction with Direct Clean-Image Estimation

Forum topic · ✨步子哥 · 2025-11-22

Summary

This article reviews the November 2025 arXiv preprint "Back to Basics: Unifying Denoising and Generation via Manifold-Aware Signal Estimation" (arXiv:2511.13720) by Kaiming He and Meta AI researchers. The paper argues that diffusion models' widespread reliance on noise prediction (predicting the added epsilon) is an inefficient detour that violates the manifold hypothesis: natural images lie on a low-dimensional manifold, while noise fills the full high-dimensional space. The authors propose Just image Transformers (JiT), a plain Transformer encoder that directly estimates the clean image x̂₀ from a noisy input xₜ using simple MSE loss, with deterministic DDIM-style sampling. Notably, JiT works well with large patch sizes (16x16, 32x32), unlike noise-prediction baselines that collapse at high resolution. Reported results on class-conditional ImageNet include FID around 2.5 at 256x256 and stable performance at 512x512 where noise baselines degrade sharply, with roughly half the parameters. The article also compares JiT with traditional DDPMs, discusses limitations (no text conditioning, possible need for more sampling steps), and explores implications for video generation, scaling laws, and connections to classical signal processing.

Overview

This post discusses the arXiv preprint "Back to Basics: Unifying Denoising and Generation via Manifold-Aware Signal Estimation" (arXiv:2511.13720) by Kaiming He and the Meta AI team (November 2025). Written in an accessible, essay-like style, it argues that modern diffusion models have drifted from the classical goal of signal denoising: directly estimating the clean signal rather than predicting the noise that was added.

Key points

Noise prediction is a detour. Since DDPM (2020), models learn to predict the added noise ε and only indirectly reconstruct the clean image x₀. The author argues this violates the *manifold hypothesis*: natural images occupy a thin low-dimensional manifold in high-dimensional space, while noise fills the entire volume. Predicting noise therefore forces models to cover a vastly larger output space.
JiT: Just image Transformers. The proposed model uses the simple formulation:

\hat{x}_0 = f_\theta(x_t, t)

A plain Transformer encoder (no U-Net, no pretraining, no codebooks) is trained with an MSE loss on the clean image, and samples via deterministic DDIM-style denoising: $x_{t-1} = \hat{x}_0 + \alpha_t (x_t - \hat{x}_0)$ .

Large patches work. Unlike noise-prediction models, which need small patches (4x4/8x8) and collapse with large ones, JiT benefits from large patch sizes (16x16, 32x32) that capture global image structure instead of pixel-level noise.

Reported experimental results

| Resolution | Patch Size | FID | Inception Score (IS) | Params (approx) | Notes | |---|---|---|---|---|---| | 256x256 | 16x16 | 2.5 | 210+ | 300M | Stable, efficient generation | | 256x256 | 32x32 | 2.8 | 205+ | 150M | Half capacity, still effective | | 512x512 | 32x32 | 3.0 | 200+ | 500M | Noise baselines collapse here | | 512x512 | 16x16 | 2.7 | 215+ | 400M | Outperforms traditional baselines |

JiT vs. DDPM comparison: noise-prediction models map into high-dimensional noise space and are memory-intensive at high resolution; JiT operates on the low-dimensional manifold, scales more gracefully, and reportedly speeds convergence (~30% faster in DiT/UViT-style setups, per the author's reading).
Theoretical framing: the paper connects generation to classical signal processing — compressed sensing and Bayesian denoising — treating generation as recovery of a low-dimensional signal, effectively a MAP estimate $$p(x_0 | x_t)$$ .

Limitations discussed

JiT is evaluated only on class-conditional ImageNet generation; text conditioning (as in Stable Diffusion) is not implemented and would require added cross-modal modules.
Deterministic sampling may need more steps to refine fine details compared to stochastic DDPM sampling, which offers useful variance for creative variation.
Generalization to audio, text, and video remains untested; large patches may blur fast motion in video.
Data that is not strictly low-dimensional (e.g., random textures) may not suit the manifold assumption.

Takeaway

The post frames JiT as a philosophical "return to basics" in generative AI — echoing Kaiming He's history of simplification (ResNet, MAE) — suggesting that direct clean-signal estimation, combined with plain Transformers and large patches, can outperform noise-prediction diffusion pipelines in efficiency and high-resolution robustness. The author notes that hybrid approaches (predicting both noise and signal) and broader empirical validation are open directions.

References

1. arXiv:2511.13720 — Back to Basics: Unifying Denoising and Generation via Manifold-Aware Signal Estimation 2. Meta AI Blog: Revisiting Diffusion Fundamentals 3. Hugging Face Discussion: JiT vs. DDPM Benchmarks 4. NeurIPS 2025 5. Towards Data Science: Kaiming He's Paradigm Shifts