Why Diffusion Models Don't Memorize: Two Timescales in Training Dynamics

Forum topic · 二一 · 2026-05-12

Summary

A NeurIPS 2025 Oral paper by Tony Bonnaire, Raphaël Urfin, Giulio Biroli, and Marc Mezard explains why diffusion models generalize rather than memorize training data, despite being overparameterized. The authors identify two characteristic timescales in training dynamics: τ_gen, a constant marking when the model starts generating high-quality samples, and τ_mem, which grows linearly with dataset size and marks when memorization of individual training examples begins. This creates a widening 'safe window' between generalization and memorization that expands as data grows—making memorization practically unreachable under normal training. The paper also shows a critical dataset size n* exists beyond which τ_mem is pushed to infinity, so overfitting never occurs. Experiments with U-Nets on real and synthetic data, plus a solvable random-feature model analysis, support the theory, framing implicit dynamical regularization as the reason capacity to memorize does not translate into actual memorization.

Introduction

Large language models can recite long passages of their training data, yet diffusion models—also massively overparameterized—manage to learn concepts without reproducing any specific training image. A NeurIPS 2025 Oral paper from Marc Mezard's group (statistical physics) offers an elegant answer: the secret lies in two distinct timescales in the training dynamics.

Key Points

Two timescales govern training:
τ_gen — the time when the model begins producing high-quality samples. Remarkably, τ_gen is constant: it barely changes whether you train on 1,000 or 100,000 images.
τ_mem — the time when the model starts copying individual training examples. τ_mem grows linearly with dataset size: more data pushes memorization further into the future.
The 'safe window': Combining these gives a window that widens with more data:
1,000 images: τ_gen ≈ 50 epochs, τ_mem ≈ 150 epochs → a 100-epoch window.
10,000 images: τ_mem ≈ 1,050 epochs → a 1,000-epoch window.
100,000 images: τ_mem ≈ 10,050 epochs → normal training essentially never reaches it.
Why generalization is fast: shared statistical features (e.g., 'cats have pointed ears') recur in every sample and are reinforced every epoch.
Why memorization is slow: each image is unique; its details require dedicated parameter capacity, and the per-image signal appears only once per epoch. Training dynamics thus give generalization a natural priority—an *implicit dynamical regularization*.
Resolving an apparent contradiction: overparameterized models *can* memorize (even random labels), but 'having the capacity' is different from 'having the time.' Standard training stops before τ_mem arrives.
Beyond a critical threshold: when dataset size exceeds a model-size-dependent threshold n*, τ_mem is pushed to infinity—the model never overfits, because the generalization signal completely dominates.
Validation: experiments with U-Nets on real and synthetic data, plus theoretical analysis via a solvable random-feature model in the high-dimensional limit.

Takeaway

No explicit regularization tricks are needed: generalization is simply the fast process, memorization the slow one. With enough data, the memorization 'train' never arrives—an emergent property of the training dynamics themselves.

Paper Info

Title: Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training
Authors: Tony Bonnaire, Raphaël Urfin, Giulio Biroli, Marc Mezard
Venue: NeurIPS 2025 (Oral)
Link: OpenReview