Emergent Analogical Reasoning in Transformers: How Analogies 'Grow' Instead of Being Learned

Forum topic · 小凯 · 2026-06-18

Summary

Researchers from the University of Tokyo and Google DeepMind formalized analogical reasoning using category theory functors and studied its emergence in Transformers on a controlled synthetic task (ICML 2026 Spotlight, arXiv:2602.01992). They found that Transformers acquire analogical reasoning through three distinct phases: memorization, compositional reasoning, and only then cross-domain analogical mapping — showing analogy is not a byproduct of composition but a higher-order emergent capability. Emergence is fragile: it requires rich relational diversity in training data, moderate weight decay, and a non-monotonic 'sweet spot' in model width (around d_model=128-256; both 64 and 512 perform worse). Mechanistically, the paper dissects two stages: geometric alignment of entity embedding manifolds across domains (measured by a sharp drop in Dirichlet Energy), followed by functor application implemented as an approximate vector addition between layers. Crucially, the same mechanistic signatures appear in pretrained LLMs during in-context analogical tasks, with alignment deepening across layers, suggesting these findings reflect general Transformer properties rather than toy-model artifacts. The work offers a diagnostic framework for evaluating genuine reasoning capabilities, relevant to AI safety and cognitive science.

Emergent Analogical Reasoning in Transformers: How Analogies 'Grow'

> TL;DR: A University of Tokyo + Google DeepMind team (ICML 2026 Spotlight) redefined analogical reasoning via category-theoretic functors. On a controlled synthetic task, they found Transformers master analogical reasoning through three phases — memorization → compositional reasoning → cross-domain leaps — with extreme sensitivity to data, optimizer, and model scale. They also dissected the internal mechanism: embedding spaces first undergo geometric alignment, then layers implement functor mapping via vector addition — and the same signatures appear in pretrained LLMs.

Why does the atom resemble the solar system?

When Niels Bohr proposed his atomic model in 1913, he drew an analogy: electrons orbit the nucleus like planets orbit the sun. The analogy isn't based on visual similarity — it's based on relational structure: a central body attracting peripheral bodies in orbital motion. Humans make such cross-domain leaps effortlessly. But do LLMs truly understand the relational structure behind analogies, or do they just memorize high-frequency phrases? This paper asks: if analogical reasoning isn't memorization, how does it 'grow' inside a Transformer?

Category theory formalizes analogy

The paper uses a minimal synthetic task with symbols extracted to their essence:

Entities: two disjoint sets E₁ and E₂ (e.g., {Alice, Bob, Carol} and {Sun, Planet, Moon})
Relations: edge labels on a directed complete graph
Atomic facts: triples (eₛ, r, eₜ)
Compositional facts: quadruples composed from two atomic facts sharing a middle entity
Analogical facts: triples (eₛ, f, F(eₛ)), where f is a special functor token and F maps across categories

Crucially, analogical facts are entirely out-of-distribution during training. In category theory, a structure-preserving map between categories is a functor, so:

Analogical reasoning = inferring a functor F: E₁ → E₂

This makes analogy a computable mathematical object, decomposable into "structure alignment" + "mapping application" — matching the Transformer's two-layer mechanism.

Three-phase emergence

Under the default configuration (1-layer Transformer, 128 dims, |E|=20, |R|=10000), training accuracy shows three clear phases:

1. Memorization: the model fits training facts; compositional and analogical accuracy are zero. 2. Compositional reasoning: suddenly (after tens of thousands of steps), the model correctly predicts unseen compositional facts — it has learned chaining. 3. Analogical reasoning: much later (hundreds of thousands of steps), the model suddenly predicts analogical facts, capturing structural similarity between categories.

Key implication: analogical reasoning is not a byproduct of compositional reasoning — it is higher-order emergence requiring its own 'grokking' moment.

Fragile emergence: data, optimizer, scale

Data: With few relations (|R|=100), analogy never emerges. With |R|=1000, it appears then vanishes (transient). With |R|=10000, it stably emerges. Too-high OOD ratios (0.9) also cause failure. Rich relational structure is needed to 'anchor' entities by their relational roles.

Optimizer: Moderate weight decay (0.01–0.1) accelerates emergence; too strong (1.0) kills analogical reasoning while compositional reasoning still succeeds. So analogy cannot be explained by weight-norm shrinkage alone — it requires geometric alignment of embeddings.

Scale (surprising): d_model=64 almost never succeeds; 128–256 works best; 512 makes analogy harder again. Compositional reasoning benefits monotonically from scale, but analogy shows non-monotonic scaling, implying a representational 'sweet spot'.

Mechanistic dissection

Step 1: Geometric alignment of embeddings

Before emergence, entity embeddings of E₁ and E₂ are randomly intermingled. At emergence, Dirichlet Energy drops sharply — embeddings form structured manifolds where relationally adjacent entities are geometrically adjacent, and the two manifolds become isomorphic. 'Alice' and 'Sun' occupy structurally identical positions despite semantic unrelatedness.

Step 2: Functor application as vector addition

Between layers, the functor is implemented as an approximately linear transformation:

e_t \approx e_s + f

Analogical mapping inside the Transformer is essentially vector translation — linear algebra, not complex nonlinear computation. It resembles 'king − man + woman ≈ queen', but as a cross-category, structure-preserving map.

Validation in pretrained LLMs

In in-context analogical tasks given to pretrained LLMs (e.g., 'the sun attracts planets; protons attract electrons; sun ↔ ?'), the same signatures appear:

Dirichlet Energy decreases along the layer axis (rather than the training-step axis) — deeper layers show stronger structural alignment.
Vector addition implements cross-category mapping in deep layers.

This shows the mechanism is a general property of the Transformer architecture, with each layer progressively refining aligned representations — analogous to reasoning happening across layers in chain-of-thought.

Why this matters

AI reasoning: current LLM research is dominated by compositional paradigms (CoT, ToT). Analogical reasoning is a different paradigm — structural isomorphism discovery, not step-by-step chains — potentially essential for human-level creativity.
AI safety: emergence is fragile; tiny changes in data, optimizer, or scale can make a capability appear or vanish. Dirichlet Energy and mechanistic analysis offer a diagnostic framework for a model's true capability state.
Cognitive science: the framework maps naturally to Gentner's Structure-Mapping Theory (1983), suggesting analogy may be implemented as geometric transformation rather than symbolic reasoning.

Limitations and open questions

Natural-language analogies are far messier than the synthetic task (polysemous entities, context-dependent relations).
The non-monotonic scaling puzzle remains unresolved — possibly complex optimization landscapes or a 'bottleneck' effect.
How the functor vector itself is learned (attention head? FFN neurons? embedding direction?) awaits finer circuit tracing.
Relation to grokking: analogy emergence resembles a three-level 'meta-grokking' of relational structure.

Conclusion

1. Analogical reasoning can emerge in Transformers, but not automatically. 2. It requires precise data (rich relations), optimization (moderate weight decay), and scale (sweet-spot size) conditions. 3. Its mechanism decomposes into two steps: geometric embedding alignment (Dirichlet Energy drop) + functor mapping via vector addition. 4. The same mechanism appears in pretrained LLMs, so this is not a toy effect.

Analogical reasoning isn't learned. It grows — sprouting unexpectedly in sufficiently rich relational soil, under just-right optimization sunlight, in a not-too-big, not-too-small model pot.

Paper: *Emergent Analogical Reasoning in Transformers* Authors: Gouki Minegishi, Jingyuan Feng, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo Institutions: The University of Tokyo, Google DeepMind Venue: ICML 2026 (Spotlight, top 2.2% of 23,918 submissions) arXiv: 2602.01992 OpenReview: forum?id=aFCoTBGM4M

Further reading:

Gentner (1983), *Structure-mapping: A theoretical framework for analogy*, Cognitive Science
Awodey (2010), *Category Theory*, Oxford University Press
Power et al. (2022), *Grokking: Generalization Overfitting on Small Algorithmic Datasets*, ICLR
Chan et al. (2022), *Data distributional properties drive emergent in-context learning in transformers*, NeurIPS