Diffusion Models and Concept Formation: When a Model 'Recognizes' a Dog in the Noise

Forum topic · 小凯 · 2026-09-14

Summary

A Georgia Tech paper (arXiv:2609.13047) draws a formal connection between diffusion models—the technology behind DALL·E and Stable Diffusion—and Cobweb, the 1987 incremental concept-formation system from cognitive science. The authors establish four correspondences: both are hierarchical density models where Cobweb's tree depth maps to the diffusion noise level; both use Gaussian prototypes (peak locations via the Tweedie formula, covariances via the denoiser Jacobian); both maximize information about the data (category utility equals mutual information, and the I-MMSE relation links denoising error to information maximization); and both exhibit a 'basic level'—an intermediate abstraction level where distinctiveness peaks. Experiments on MNIST and Fashion-MNIST recover concept trees from diffusion models via gradient mean-shift, aligning with Cobweb's hierarchy: Cobweb peaks at tree depth 3 while diffusion peaks near t≈150, matching known phase-transition points where the reverse process commits to category identity before fine details. The key difference is representational: Cobweb is explicit and symbolic with interpretable, incremental updates; diffusion models are implicit, continuous score fields that support interpolation. The paper suggests hybrid neural-symbolic architectures, such as taxonomic networks.

Diffusion Models and Concept Formation: When a Model "Recognizes" a Dog in the Noise

> *"AI's deepest secrets often hide in questions it was never built to answer."*

Paper Card

Title: Diffusion Models and Concept Formation Authors: Zekun Wang, Karthik Singaravadivelan, Christopher J. MacLellan Institution: College of Computing, Georgia Institute of Technology arXiv: 2609.13047 Posted: 2026-09-11

Opening: One Sentence at the Pet Store

When you buy a dog, you don't ask for "a mammal" (too broad) or a Border Collie born March 15, 2024 in Ohio, 8.3 kg, with a white patch on the left ear (too specific). You ask to see their dogs.

Cognitive science calls this the basic level (Rosch et al., 1976): the level in a concept hierarchy—animal → mammal → canine → dog → Border Collie—where cognition is most efficient, balancing informativeness against distinctiveness.

In 1987, Fisher built Cobweb, an incremental concept-formation system that grows concept trees with basic-level structure. Nearly four decades later, a very different community built diffusion models—the engines behind DALL·E, Stable Diffusion, and Midjourney.

This paper's claim: diffusion models have been doing what Cobweb did all along—forming concepts implicitly; no one had read them in cognitive-science language.

Chapter 1: Cobweb—A 40-Year-Old AI Librarian

Cobweb inserts each instance by walking down a tree and choosing among four operations: place into a child node, create a new node, merge two nodes, or split one. The criterion is category utility—an information-theoretic measure equal to the mutual information I(X;C) between features and concepts. A good partition makes features predictable within categories while keeping categories distinct.

The tree is grown from data, not designed. Each node stores a Gaussian prototype (mean + variance) summarizing its members. No one tells Cobweb "dog" is a category—the node emerges after enough examples.

Chapter 2: Diffusion Models and the Key Insight

Diffusion models train a network to reverse a noising process, learning the score—the gradient of log-density of noised data at every noise level (score matching).

The paper's starting point is an elegant mathematical fact: the density p_t implicitly learned at noise level t is the data distribution convolved with a Gaussian kernel of variance σ_t².

In plain language: a Gaussian smoothing whose width is set by the noise level.

Low noise → narrow smoothing → peaks correspond to individual instances
High noise → wide smoothing → peaks merge → peaks correspond to broad categories

This is a family of nested densities—exactly the picture of kernel density estimation (KDE), where clusters are density peaks and bandwidth controls granularity. Diffusion models learn the density at every bandwidth, plus the direction to every peak (the score): not a tree, but a continuous concept tree.

Chapter 3: Four Bridges Between Cobweb and Diffusion

Bridge 1: Both are hierarchical density models

Cobweb's tree depth ⟺ diffusion's noise level σ_t

Bridge 2: Both have Gaussian prototypes (hierarchical Bayes)

Peak location gives the prototype mean via Tweedie's formula: m_c = x*/√ᾱ_t
Local shape gives the covariance via the denoiser's Jacobian: Σ_c = (1−ᾱ_t)/√ᾱ_t · ∇x̂₀(x*)—a Laplace approximation of p_t at that peak.

Each diffusion "peak" carries a Gaussian prototype, exactly the role of a Cobweb node.

Bridge 3: Both maximize information about the data

> L(θ) = ½ E ∫ w(t) ‖x₀ − x̂_θ(x_t, t)‖² dt, with w(t) = −SNR'(t)

By the I-MMSE relation (Guo et al., 2005), minimizing mean-square denoising error is equivalent to maximizing the information noisy observations carry about clean data; at the optimum it equals negative log-likelihood (Kong et al., 2023). Category utility and the denoising objective are two expressions of one principle.

Bridge 4: Both have a "basic level"

The curve peaks at an intermediate noise level.

Experiments: on MNIST and Fashion-MNIST, Cobweb's distinctiveness peaks at tree depth 3; diffusion models peak near t≈150 (scheduler-dependent, but an interior point). Moreover, t≈150 coincides with the phase transition identified in prior theory (Sclocchi et al., 2025; Biroli et al., 2024): at this noise, the reverse process commits to a sample's category identity before its fine details—"dog" first, Border Collie later.

The "basic level" predicted by cognitive science appears precisely on the noise scale of diffusion models.

Chapter 4: Experiments—How Similar Are the Two Trees?

Diffusion models store no tree—only a score network—so it must be recovered:

1. At low noise, add noise to each test image and follow the score to convergence → fine-grained prototypes (gradient mean-shift) 2. Increase noise gradually; peaks merge via agglomerative grouping → parent nodes 3. Repeat up to the highest noise → full hierarchy

On MNIST, the trees match remarkably:

Both Cobweb and the diffusion model group 2 with 8 (similar handwriting shapes)
Both treat 1 and 7 as close relatives
On Fashion-MNIST, both merge footwear and bags before separating them from garments

Visualizing prototypes at each system's basic level—Cobweb's depth-3 nodes vs. diffusion's peaks at t*≈150—shows both are object-level categories: coarser than a single sample, finer than the whole dataset, matching digit and garment classes and human basic-level intuition ("a shoe," not "a Fashion-MNIST sample" and not "a brown size-38 Nike Air Force 1 for the left foot").

Two independent systems, from entirely different learning mechanisms, converged on the same cognitive structure.

Chapter 5: Where They Differ—and Why It Matters

Cobweb: an explicit, discrete tree. Symbolic prototypes, interpretable operations (insert/create/merge/split), incremental learning, no catastrophic forgetting (Barari et al., 2025 traced this to the closed-form information-theoretic updates).

Diffusion models: an implicit, continuous score field. The whole hierarchy is compressed into network weights. Advantages: interpolability (convex combinations of prototypes yield meaningful new prototypes—Cobweb cannot do this), scalability, high generation quality.

Philosophically: Cobweb is symbolic, diffusion models are connectionist—yet they describe the same cognitive structure. The natural outlook is grafting them together: diffusion for continuous, scalable density learning; Cobweb for explicit, interpretable concept structure. Taxonomic Networks (Wang et al., 2025) already move in this direction.

Epilogue: Concepts Emerge from Noise

The paper's most moving point is not that it "proves diffusion models do concept formation," but the picture it paints:

Concepts are not rows in a database or strings in a label list. Concepts are peaks that emerge naturally from the density structure of data.

A baby forms "dog" not because someone supplies the label, but because that cluster of furry four-legged things is tight in perceptual space, separated from other clusters by clear valleys.

Cobweb captured this with discrete probability trees. Diffusion models perfected the same process with a continuous noise-weighted score field—finding, at t≈150 on the noise scale, the "dog" that cognitive science predicted forty years ago.

From Fisher's 1987 symbolic tree to Ho's 2020 denoising networks: thirty-three years and two AI paradigm shifts apart. But the mathematics of concept formation was only ever one.

References

1. Wang Z., Singaravadivelan K., MacLellan C.J. "Diffusion Models and Concept Formation." arXiv:2609.13047, 2026. 2. Fisher D.H. "Knowledge acquisition via incremental conceptual clustering." Machine Learning, 2:139-172, 1987. 3. Rosch E., Mervis C.B. "Family resemblances: Studies in the internal structure of categories." Cognitive Psychology, 7:573-605, 1975. 4. Rosch E., et al. "Basic objects in natural categories." Cognitive Psychology, 8:382-439, 1976. 5. Corter J.E., Gluck M.A. "Explaining basic categories." Cognitive Science, 16:75-95, 1992. 6. McKusick K.B., Thompson K. "COBWEB/3: A portable implementation." NASA Ames Research Center, 1990. 7. Barari A., et al. "Cobweb/4V: Concept formation for continual learning in image classification." 2024. 8. Ho J., Jain A., Abbeel P. "Denoising diffusion probabilistic models." NeurIPS, 2020. 9. Song Y., Sohl-Dickstein J., et al. "Score-based generative modeling through stochastic differential equations." ICLR, 2021. 10. Kingma D., et al. "Variational diffusion models." NeurIPS, 2021. 11. Wang Z., et al. "Taxonomic networks: A structured approach to neural-symbolic integration." 2025. 12. Wang Z., et al. "Deep taxonomic networks: Learning hierarchical Gaussian mixture priors." 2025. 13. Sclocchi A., et al. "A phase transition in diffusion models at intermediate noise levels." 2025. 14. Biroli G., et al. "Dynamical regimes of diffusion models." 2024. 15. Guo D., Shamai S., Verdú S. "Mutual information and minimum mean-square error in Gaussian channels." IEEE Trans. Inf. Theory, 2005.