Emergent Analogical Reasoning in Transformers: It's Grown, Not Taught
> TL;DR: A University of Tokyo + Google DeepMind team redefines analogical reasoning using category-theoretic functors, and shows on a controlled synthetic task that Transformers acquire analogical reasoning through three stages—memorization → compositional reasoning → cross-domain leaps—with extreme sensitivity to data, optimizer, and model scale. Mechanistically, the embedding space first becomes geometrically aligned, then layers apply a functor mapping via vector addition—and the same signatures are found in pretrained LLMs.
---
Why do atoms resemble solar systems?
When Niels Bohr proposed his atomic model in 1913, he used an analogy: electrons orbit the nucleus like planets orbit the sun. The analogy rests not on surface similarity but on relational structure: a central body attracting peripheral bodies into orbits.
Humans make such cross-domain leaps effortlessly. But do LLMs actually grasp the relational structure behind analogies, or merely memorize frequent metaphors? This paper asks: if analogical reasoning isn't memorization, how does it grow inside a Transformer?
---
1. Category theory enters: redefining analogy mathematically
The synthetic task
Instead of messy natural language, the authors designed a minimal symbolic task:
- Entities: two disjoint sets E₁ and E₂ (e.g., {Alice, Bob, Carol} and {Sun, Planet, Moon})
- Relations: edge labels on a directed complete graph
- Atomic facts: triples (e_s, r, e_t)
- Compositional facts: quadruples chaining two atomic facts through a shared entity (e.g., Alice → mother of → father of → Carol = grandmother)
- Analogical facts: triples (e_s, f, F(e_s)), where f is a special "functor" token and F is a cross-domain mapping
- Too few relations (|R|=100): analogy never emerges—entities can't be distinguished by relational roles.
- Moderate (|R|=1000): analogy appears but is transient—the model learns it, then forgets.
- Sufficient (|R|=10000): analogy emerges stably.
- High OOD ratio (0.9): analogy fails.
- Moderate weight decay (0.01–0.1) accelerates emergence.
- Excessive weight decay (1.0) kills analogical reasoning entirely, even though compositional reasoning survives.
- d_model=64: almost never succeeds
- d_model=128–256: most likely to succeed
- d_model=512: becomes harder again
- Dirichlet Energy decreases across layers (rather than training steps)—deeper representations show stronger structural alignment.
- Vector addition implements cross-domain mapping in deeper layers.
- AI reasoning: current research is dominated by compositional paradigms (CoT, Tree-of-Thoughts). Analogy offers a different paradigm: cross-domain structural leaps rather than chained steps—likely necessary for human-level creativity.
- AI safety: emergence is fragile; small changes in data, optimizer, or scale can make capabilities vanish. Dirichlet Energy and mechanistic analysis offer a diagnostic framework for a model's *true* capability state—critical when LLM reasoning is trusted in high-stakes decisions.
- Cognitive science: geometric alignment + functor mapping maps naturally onto Gentner's Structure-Mapping Theory (1983), suggesting analogy may be implemented as geometric transformation rather than symbolic reasoning.
- Gap to natural language: real-world analogy involves polysemous entities and implicit, context-dependent relations.
- Non-monotonic scaling mystery: why do larger models struggle? Possible hypotheses: more complex optimization landscapes, or a required "bottleneck" for alignment signals.
- Origin of the functor vector: how the additive functor vector is learned (attention head? FFN neurons? embedding direction?) awaits circuit tracing.
- Relation to Grokking: delayed emergence resembles Grokking, but with a three-level jump—perhaps "meta-grokking" of relational structure.
- Gentner (1983), *Structure-mapping: A theoretical framework for analogy*, Cognitive Science
- Awodey (2010), *Category Theory*, Oxford University Press
- Power et al. (2022), *Grokking: Generalization Overfitting on Small Algorithmic Datasets*, ICLR
- Chan et al. (2022), *Data distributional properties drive emergent in-context learning in transformers*, NeurIPS
Crucially, analogical facts never appear during training (OOD). The model must infer that the two domains share the same relational structure and infer the functor F.
Formalization
Analogy is defined as correspondence inference across categories: analogy = inferring a functor F: E₁ → E₂. This makes analogy a computable object, decomposable into "structure alignment" + "mapping application"—which maps onto two distinct Transformer mechanisms.
---
2. Three-stage grokking
With the default setup (1-layer Transformer, d=128, |E|=20, |R|=10000), training accuracy shows three clear stages:
1. Memorization: the model fits training facts; compositional and analogical accuracy stay at zero. 2. Compositional reasoning: suddenly (after tens of thousands of steps), the model correctly predicts unseen compositional facts—learning chain composition. 3. Analogical reasoning: much later (hundreds of thousands of steps), the model suddenly predicts analogical facts, capturing the structural similarity between domains.
Key implication: analogical reasoning is not a byproduct of compositional reasoning—it is a higher-order emergence requiring its own "grokking" moment.
---
3. The fragility of emergence
Data: relational diversity is the lifeline
Core insight: analogy needs rich relational structure to anchor entities.
Optimizer: weight decay is not a panacea
So analogy cannot be explained by weight-norm shrinkage alone—it needs geometric alignment of embeddings.
Scale: bigger is not better
Compositional reasoning benefits monotonically from scale; analogical reasoning shows non-monotonic scaling—it requires a representational sweet spot.
---
4. Dissecting the mechanism
Step 1: Geometric alignment of embeddings
Before emergence, entities of E₁ and E₂ are randomly intermingled in embedding space. At emergence, Dirichlet Energy drops sharply: neighboring entities in relational space become geometrically close, forming two isomorphic manifolds. "Alice" and "Sun" are semantically unrelated, but occupy structurally identical positions—both play the "central body" role.
Step 2: Functor application as vector addition
Across layers, the functor is implemented as approximate vector addition:
Analogical reasoning is not a complex nonlinear operation but basic linear algebra—like "king − man + woman ≈ queen," but as a structure-preserving cross-domain mapping.
---
5. Validation in pretrained LLMs
Using an in-context analogy task on pretrained LLMs (e.g., solar system ↔ atomic structure mappings), the authors find the same mechanistic signatures:
The mechanism found in the toy setting is a general property of the Transformer architecture, not an artifact. The layer axis (rather than training-time axis) reflects that pretrained models are *executing* an analogical ability acquired during pretraining—similar to reasoning occurring across layers in chain-of-thought.
---
6. Why this matters
---
7. Limitations and open questions
---
8. Conclusion
1. Analogical reasoning can emerge in Transformers, but not automatically. 2. It requires precise conditions: rich relational data, moderate weight decay, and a model-size sweet spot. 3. Its mechanism decomposes into two steps: geometric alignment (Dirichlet Energy drop) + inter-layer functor mapping (vector addition). 4. The same signatures appear in pretrained LLMs, showing this is not a toy effect.
Analogical reasoning isn't learned—it grows, in sufficiently rich relational soil, under just-right optimization sunlight, in a not-too-big-not-too-small model pot.
---
Paper: *Emergent Analogical Reasoning in Transformers* Authors: Gouki Minegishi, Jingyuan Feng, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo Institutions: The University of Tokyo, Google DeepMind Venue: ICML 2026 (Spotlight, top 2.2% of 23,918 submissions) arXiv: 2602.01992 OpenReview: forum?id=aFCoTBGM4M
Further reading: