A Chinese forum post on zhichai.net discusses a NeurIPS 2025 Oral paper that answers a curious question: why don't diffusion models memorize their training data?
The puzzle
Large language models can reproduce training text verbatim. Diffusion models, despite also being heavily overparameterized and trained on massive datasets, learn concepts like "cat" and generate novel images—yet almost never reproduce a specific training image unless deliberately prompted. Both have the *capacity* to memorize, so why does one choose not to?
Two timescales in training dynamics
Citing the paper *"Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training"* (Tony Bonnaire, Raphaël Urfin, Giulio Biroli, Marc Mezard), the post explains two characteristic times:
- τ_gen (generalization onset): the time when the model starts producing high-quality samples. Crucially, this is roughly constant regardless of dataset size.
- τ_mem (memorization onset): the time when the model starts replicating specific training examples. This grows linearly with dataset size—e.g., ~150 epochs for 1,000 images vs. ~10,050 for 100,000.
- 1,000 images: window ≈ 100 epochs
- 10,000 images: window ≈ 1,000 epochs
- 100,000 images: window ≈ 10,000 epochs—effectively unreachable
- Generalization signals are repeated: shared features (pointed ears, whiskers) appear in every cat image and are reinforced every epoch.
- Memorization signals are sparse: each image's unique details are seen once per epoch (or a noisy version), requiring dedicated parameter capacity per example.
- Title: Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training
- Authors: Tony Bonnaire, Raphaël Urfin, Giulio Biroli, Marc Mezard
- Venue: NeurIPS 2025 (Oral)
- Link: https://openreview.net/forum?id=BSZqpqgqM0
The safety window
Since τ_gen stays fixed while τ_mem recedes, larger datasets create an ever-widening "safety window": normal training stops long before memorization begins. The post illustrates with numbers:
Why this pattern emerges
Training dynamics thus grant generalization a natural priority—an implicit regularization, no extra design required.
Reconciling two observations
The result unifies classical memorization-capacity results (overparameterized models *can* memorize even random labels) with the empirical fact that diffusion models rarely do. Capacity and time are different things: given enough epochs, diffusion models would memorize everything, but the safety window prevents it in practice.
Beyond a critical point
The paper also finds a critical dataset size n* related to model size: beyond it, τ_mem is pushed to infinity—the model never overfits, even with unlimited training, because generalization signals completely dominate memorization signals.
Validation
Findings are demonstrated with U-Net experiments on real and synthetic data, plus a solvable random-feature model analyzed in the high-dimensional limit.