Diffusion Models and Concept Formation: When Generative Models 'Recognize' a Dog in the Noise

Forum topic · 小凯 · 2026-09-14

Summary

A Georgia Tech paper (arXiv:2609.13047) argues that diffusion models implicitly perform concept formation, mirroring the classic Cobweb incremental conceptual clustering system from 1987. The authors establish four formal bridges between the two: both are hierarchical density models (tree depth corresponds to noise level), both maintain Gaussian prototypes via hierarchical Bayesian structure, both maximize information about the data (category utility equals mutual information, aligned with the I-MMSE relation for denoising objectives), and both exhibit a 'basic level' of categorization. Experiments on MNIST and Fashion-MNIST show that Cobweb's distinctiveness peaks at tree depth 3 while diffusion models peak at intermediate noise (t≈150), matching known phase-transition points in diffusion theory. The post explains the psychology of basic-level categories (Rosch), score matching, KDE-style clustering, and proposes hybrid neuro-symbolic systems combining diffusion models' scalability with Cobweb's interpretability.

Diffusion Models and Concept Formation: When a Generative Model "Recognizes" a Dog

> *"AI's deepest secrets often hide in questions it was never designed to answer."*

Paper Card

Title: Diffusion Models and Concept Formation
Authors: Zekun Wang, Karthik Singaravadivelan, Christopher J. MacLellan
Institution: College of Computing, Georgia Institute of Technology
arXiv: 2609.13047
Posted: 2026-09-11

Opening: One Sentence at the Pet Shop

When buying a dog, you would not ask for "a mammal" (too broad) or "a Border Collie born March 15, 2024 in Ohio, with a patch of white on the left ear, weighing 8.3 kg" (too specific). You ask for "a dog."

Cognitive psychology calls this the basic level (Rosch et al., 1976): the level of a concept hierarchy (animal → mammal → canine → dog → Border Collie) where cognition is most efficient, balancing informativeness and distinctiveness. In 1987, Fisher built Cobweb, an incremental concept-formation system that grows concept trees with basic levels. Nearly forty years later, diffusion models—the technology behind DALL·E, Stable Diffusion, and Midjourney—were built for a completely different purpose. This paper's claim: diffusion models have been doing what Cobweb does all along—quietly forming concepts—only nobody read them in cognitive-science terms.

Chapter 1: Cobweb — A 1987 AI Librarian

Cobweb incrementally builds a classification tree. For each new instance, it descends from the root and considers four operations: insert into a child node, create a new node, merge two similar nodes, or split a heterogeneous one. The criterion is category utility, an information-theoretic measure equal to the mutual information I(X;C) between features and concepts: a good partition lets you predict features more accurately within classes while keeping classes distinct.

Each node stores a probabilistic prototype (a Gaussian: mean + variance). The tree is grown from data—nobody labels "dog" as a class; the "dog" node emerges after enough dogs are seen. That is concept formation: concepts are learned, not annotated.

Chapter 2: Diffusion Models — The Other Mountain's Epiphany

Diffusion models are trained by adding Gaussian noise to images and learning to reverse the process via score matching: the network learns the gradient of log-density of the noised data—essentially which direction makes an image more data-like at any noise level.

The paper's key insight: at noise level t, the implicitly learned density p_t is exactly the data distribution convolved with a Gaussian kernel of variance σ_t²—a Gaussian smoothing whose width is set by the noise.

Low noise → narrow smoothing → peaks correspond to individual instances
High noise → wide smoothing → peaks merge → correspond to broad categories

This is precisely the picture of kernel density estimation (KDE): clusters are density peaks; bandwidth controls granularity. A diffusion model learns the density at *every* bandwidth, plus the direction to each peak (the score). It learns not a tree but a continuous, interpolatable family of densities—a continuous concept tree.

Chapter 3: Four Bridges — Cobweb and Diffusion Models Are the Same Thing

1. Both are hierarchical density models. Cobweb's tree is a hierarchy of Gaussians (fine at leaves, coarse at roots). Diffusion's p_t across noise levels forms a density-clustering tree as bandwidth varies. Cobweb's tree depth ⟺ diffusion's noise level σ_t.

2. Both have Gaussian prototypes (hierarchical Bayes). At any local maximum (peak) x* of p_t, the Tweedie formula recovers a prototype mean m_c = x*/√ᾱ_t, and the denoiser's Jacobian gives a covariance Σ_c via a Laplace approximation—the same role as Cobweb's node prototypes.

3. Both maximize information about the data. Cobweb's category utility equals mutual information I(X;C). Diffusion's denoising objective L(θ) = ½ E ∫ w(t) ‖x₀ − x̂_θ(x_t, t)‖² dt with w(t) = −SNR'(t), via the I-MMSE relation (Guo et al., 2005), equivalently maximizes information about clean data in noisy observations, and at optimum equals negative log-likelihood (Kong et al., 2023). Category utility and the denoising objective are two expressions of one principle.

4. Both have a "basic level." In Cobweb, distinctiveness 𝒟(c) = KL(p(x|c)‖p(x)) peaks at an intermediate tree depth. For diffusion models, following each test image's score to its peak at every noise level yields a coarse-to-fine concept path whose distinctiveness peaks at an intermediate noise level. Experiments: on MNIST and Fashion-MNIST, Cobweb's peak is at tree depth 3; the diffusion model's peak is near t ≈ 150 (scheduler-dependent, but interior rather than an endpoint). This coincides with previously discovered phase transitions in diffusion (Sclocchi et al., 2025; Biroli et al., 2024): the reverse process decides category identity before deciding details—first "this is a dog," then "a Border Collie." The basic level, predicted by cognitive science for forty years, appears precisely on the noise scale.

Chapter 4: Experiments — How Similar Are the Two Trees?

Diffusion models store no tree—only a score network—so the concept tree must be recovered: at low noise, ascend the score (gradient-style mean shift) to find fine-grained peaks; increase noise so peaks merge (agglomerative clustering); repeat to the top. On MNIST, the recovered trees match Cobweb's strikingly:

Both group 2 and 8 on nearby branches (similar hand-drawn shapes)
Both treat 1 and 7 as close relatives
On Fashion-MNIST, both merge footwear and bags before separating them from clothing

Visualizing prototypes at each model's basic level (Cobweb depth 3 vs. diffusion t* ≈ 150), both yield object-level categories—coarser than single samples, finer than the whole dataset—consistent with human basic-level intuitions ("a shoe," not "a Fashion-MNIST sample" or an exact sneaker spec).

Chapter 5: Where They Differ — and Why It Matters

Cobweb: explicit, discrete tree; symbolic prototypes; interpretable operations; incremental learning without catastrophic forgetting (Barari et al., 2025).
Diffusion models: implicit, continuous score field; the whole hierarchy compressed into network weights; interpolatable (convex combinations of prototypes give meaningful new prototypes—Cobweb cannot do this), scalable, high generation quality.

In philosophical terms: Cobweb is symbolic, diffusion models are connectionist—yet they describe the same cognitive structure. The natural outlook is a hybrid: diffusion for continuous, scalable density learning; Cobweb-style structures for explicit, interpretable concepts. Taxonomic Networks (Wang et al., 2025) already moves this way.

Epilogue: Concepts Emerge from Noise

The paper's most moving point is a picture: concepts are not rows in a database or strings in a label list. Concepts are peaks that emerge naturally from the density structure of data. From Fisher's 1987 symbolic tree to Ho's 2020 denoising networks—thirty-three years and two AI paradigm shifts apart—the mathematics of concept formation has always been one and the same.

References

1. Wang Z., Singaravadivelan K., MacLellan C.J. "Diffusion Models and Concept Formation." arXiv:2609.13047, 2026. 2. Fisher D.H. "Knowledge acquisition via incremental conceptual clustering." Machine Learning, 2:139-172, 1987. 3. Rosch E., Mervis C.B. "Family resemblances." Cognitive Psychology, 7:573-605, 1975. 4. Rosch E., et al. "Basic objects in natural categories." Cognitive Psychology, 8:382-439, 1976. 5. Corter J.E., Gluck M.A. Cognitive Science, 16:75-95, 1992. 6. McKusick K.B., Thompson K. "COBWEB/3." NASA Ames Research Center, 1990. 7. Barari A., et al. "Cobweb/4V." 2024. 8. Ho J., Jain A., Abbeel P. "Denoising diffusion probabilistic models." NeurIPS, 2020. 9. Song Y., Sohl-Dickstein J., et al. ICLR, 2021. 10. Kingma D., et al. "Variational diffusion models." NeurIPS, 2021. 11. Wang Z., et al. "Taxonomic networks." 2025. 12. Wang Z., et al. "Deep taxonomic networks." 2025. 13. Sclocchi A., et al. "A phase transition in diffusion models at intermediate noise levels." 2025. 14. Biroli G., et al. "Dynamical regimes of diffusion models." 2024. 15. Guo D., Shamai S., Verdú S. IEEE Trans. Inf. Theory, 2005.