Why Diffusion Models Aren't Copy Machines: Creativity Comes From Being Imperfect

Forum topic · 小凯 · 2026-05-19

Summary

A Chinese forum post analyzes a 2026 arXiv paper by Itamar Levine and Yair Weiss (Hebrew University of Jerusalem), "Diffusion Models, Denoiser Architecture and Creativity" (arXiv:2605.16415), which offers the first theoretical explanation of why diffusion models generate images beyond their training sets. The paper proves that a Bayes-optimal denoiser would collapse generation to verbatim reproduction of training data—zero creativity. Real diffusion models are creative precisely because their denoiser architectures are not optimal. The authors derive explicit output distributions for three simplified architectures: linear denoisers (convex combinations of training samples), polynomial denoisers (nonlinear interpolation), and bottleneck denoisers (compressed feature spaces resembling UNets, enabling generalization). Experiments show that minor UNet modifications—channel widths, skip connections, bottleneck size—dramatically alter creativity under identical training conditions. The paper's core thesis: creativity requires alignment between architectural inductive bias and target distribution structure—good enough to learn real structure, imperfect enough to avoid memorization. The post also raises caveats about the simplified-architecture gap, ambiguous definitions of creativity, and possible extensions to LLMs.

Why Diffusion Models Aren't "Copy Machines" — Because They Aren't Good Enough Yet

| Item | Detail | |------|--------| | Title | Diffusion Models, Denoiser Architecture and Creativity | | Authors | Itamar Levine, Yair Weiss (Hebrew University of Jerusalem) | | arXiv | 2605.16415 (cs.CV, cs.LG) | | Date | May 2026 | | Core contribution | First theoretical explanation of why diffusion models generate creative images beyond the training set — creativity is not a byproduct of model scale, but the result of "interaction" between denoiser architecture and the target distribution | | Link | https://arxiv.org/abs/2605.16415 |

How do diffusion models work?

Ask any deep learning researcher and you'll get roughly the same answer: add noise to training data, train a denoiser, and at inference start from noise and denoise step by step — like dripping ink into water and training a machine to suck the ink back out.

But this explanation misses a key point. Strictly speaking, if your denoiser is the Bayes-optimal denoiser for the training set — the one that best reconstructs the data in a statistical sense — then the diffusion model's behavior is extremely boring: it reproduces training samples verbatim, one by one. Zero creativity.

Reality is the opposite. Stable Diffusion generates faces that don't exist in the training set; Imagen draws a giraffe riding a skateboard with natural lighting. That's not copying — that's creating.

So the question isn't "how do diffusion models generate images" but "how do diffusion models create images." Levine and Weiss's paper gives a precise mathematical answer.

🎨 1. Bayes-optimal denoiser = zero creativity

The generation process: start from pure noise, take small steps forward, and at each step a denoiser estimates the direction toward a clean image.

In the ideal case, with a Bayes-optimal denoiser that knows the true distribution and produces the conditional expectation over all data, the generated distribution completely collapses to the training data distribution itself. Feed it noise and it outputs the "most likely" training sample. It produces nothing new.

So a mathematically perfect diffusion model should have no creativity at all. Yet real diffusion models are creative — meaning real denoiser architectures are not Bayes-optimal, and their structural limitations are exactly the source of creativity.

🔬 2. Three architectures, three creativity modes

The authors derive explicit forms of the generated distribution for three simplified architectures:

Linear denoiser. With a simple matrix multiplication, the learned distribution consists of linear combinations of training samples — a face might be A's eyebrows + B's eyes + C's mouth. This creativity is limited and predictable: only convex combinations, never true outliers. Like a tuner who can adjust notes but never invent new ones.

Polynomial denoiser. Adding higher-order terms allows nonlinear interpolation between training samples. The model finds "structures" in the data and recombines them — after 100 cats and 100 dogs, it might generate something with cat eyes and a dog body.

Bottleneck denoiser. The most interesting case, closest to a real UNet. The denoiser must represent inputs with few intermediate features (like UNet's down-up sampling). The generated distribution becomes a conditional distribution over the bottleneck feature space. The bottleneck forces compressed representations: the model must extract "what is the essence of a cat" and generate new cats from that essence — naturally producing generalization.

🧪 3. Experiment: change the UNet, change the creativity

The paper makes tiny modifications to a popular UNet — adjusting channel counts, skip connections, bottleneck size — and observes the outputs.

Result: with everything else identical (same data, same training time, same loss), small changes produce completely different creativity profiles. Some variants nearly copy the training set (low creativity); others remix and recombine it (high creativity); some generate outright unreal images (creativity overshooting into hallucination).

Conclusion: creativity is not a property of the diffusion model itself, but of the denoiser architecture's inductive bias — not the result of "learning more data," but of "being forced to constrain what is learned in a particular way."

🏗️ 4. Core argument: alignment is the precondition for creativity

One sentence: diffusion models succeed when the denoiser's inductive bias is highly aligned with the true structure of the target distribution.

Bias mismatched with the distribution (e.g., linear architecture, complex distribution): no creativity, just a blurry average.
Bias perfectly matched (Bayes-optimal): no creativity, just perfect copying.

Creativity lives in the "roughly right but not quite right" zone — the approximation is good enough to learn real structure, imperfect enough to fail to memorize every sample. This structural underfitting creates room for novel combinations and variants.

🤔 5. Honest questions

Simplified vs. real architectures. Real UNets have self-attention, group normalization, multi-scale skip connections, dozens of layers — creativity patterns will be far more complex than the analytical solutions. The paper offers no continuous bridge from simplified to complex architectures.

Defining "creativity." Operationalizing creativity as "generating samples different from the training set" conflates meaningful novelty with noise, artifacts, and meaningless deviation. Creativity means generating things both new and plausible — new but implausible is just nonsense.

The LLM analogy. The framework — architectural bias + distribution alignment determines creativity — might extend to autocratic LLM architectures (causal masks, positional encodings, layer norms), but the paper doesn't address this.

🎯 6. My verdict

The paper's most precious contribution is its worldview: diffusion models are creative not because they are "big enough," but because they are "not good enough." Given unlimited expressive capacity, they would memorize perfectly and stop creating.

This echoes an old truth of human creation: constraints produce style. The sonnet's strict form didn't hinder Shakespeare — it empowered him. Constraints force a system into a compressed representation of true structure, and within that compressed space, new combinations emerge naturally.

If you're designing a generative model, don't just ask "how big?" — ask "how limited, and how like the target distribution?" Both questions matter equally.

📚 References

1. Levine, I., Weiss, Y. (2026). Diffusion Models, Denoiser Architecture and Creativity. arXiv:2605.16415. 2. Ho, J., Jain, A., Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. 3. Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. 4. Kadkhodaie, Z., et al. (2023). Generalization in Diffusion Models Arises from Geometry-Adaptive Harmonic Representations. ICLR 2024.