Paper link: https://arxiv.org/abs/2608.07261
A Puzzling Failure
Ask an LLM: "Which university did the wife of the author of *One Hundred Years of Solitude* attend?"
This is two-hop reasoning:
- Hop 1: Who is Márquez's wife? (Mercedes Barcha)
- Hop 2: Which university did Mercedes Barcha attend?
- The experimental setting is symbolic and controlled; real two-hop reasoning involves natural language, entity disambiguation, and relation ambiguity. Appendix validations on real datasets are limited in scale.
- Why parameter sharing in the looped architecture helps (regularization vs. something deeper) remains not fully explained.
- The "mapping vs. reasoning" distinction rests on probes; probe-detectable information ≠ information the model actually uses. Stronger causal-intervention experiments are needed.
The model answers each hop correctly in isolation, but fails when the hops are combined. The model knows each hop—why can't it compose them? This paper explains this "two-hop generalization paradox" by digging into transformer internals, arriving at a rather counterintuitive answer.
Experiments in a Controlled Setting
To eliminate pretraining-data confounds, the authors trained transformers from scratch on a fully controlled symbolic knowledge graph. Each training sample is a triple (head entity, relation, tail entity); the model must answer two-hop queries: given a head entity and two relations, find the final tail entity.
Key finding: when the second hop's relation pattern is in-distribution (ID), accuracy is nearly 100%; when it is slightly out-of-distribution (OOD), accuracy drops to ~30%.
Note: the first hop is always correct. The model finds the correct bridging entity. The problem is the second hop—even with the bridging entity correctly present in context, the model won't use it.
Mechanism: Upper Layers Do "Mapping," Not "Reasoning"
The most interesting part of the paper. Using mechanistic interpretability tools:
1. Locating the bridging entity's representation. Logit lens and patching experiments show the bridging entity is correctly represented in intermediate layers (~layers 4–8). 2. Probe experiments. A linear probe trained on intermediate representations can predict the correct tail entity. The information is there, hidden in the representations. 3. Why doesn't it surface? The upper (final) layers perform "mapping" rather than "reasoning". In-distribution, the upper layers learn a direct lookup table: bridging entity → tail entity. It's memorized from training-time (bridging, tail) pairs, not derived relationally. Once a novel bridging entity appears, the lookup fails.
Analogy: memorize 100 two-digit addition problems like 23+45=68. You ace 23+45 (mapping hit) but stumble on 23+46 (mapping miss)—even though you "know" 3+6=9 and 20+40=60. You memorized answers, not the algorithm. The transformer's upper layers are that memorizing student.
Solutions: Force Upper Layers to Actually Reason
Approach 1: Explicitly train upper layers for representation-based reasoning. Freeze lower layers and train upper layers to infer the tail entity from intermediate bridging-entity representations instead of looking it up. Works, but requires carefully designed data.
Approach 2: Looped Transformer. Share parameters across upper layers, letting the model "think repeatedly"—each loop re-processes the intermediate representations, forcing a general reasoning procedure rather than a fixed mapping.
Result: Looped Transformer lifts OOD two-hop accuracy from ~30% to 80%+ with no ID degradation. Logit-lens verification confirms the upper layers now perform representation-based reasoning rather than a bigger lookup table.
Why This Matters
1. It pinpoints a concrete mechanistic defect—"upper layers map instead of reason"—rather than vague "weak reasoning." 2. Looped architectures have deep implications: they make the transformer iterate over the same information, echoing prefrontal-cortex-style reprocessing—not for sequence handling, but to give reasoning "thinking time." 3. It reframes CoT (chain-of-thought): CoT may work not because intermediate steps are verbalized, but because generating them re-represents information in context, giving upper layers a second pass. Loops internalize this—no external CoT needed. 4. Direct relevance to RAG: "retrieved the right evidence but the model won't use it" may stem from exactly this mapping-not-reasoning defect.
Honest Assessment
Limitations:
---
Paper: Zhang, Wang, Wang, Wan, Luo. *Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models*. arXiv:2608.07261, 2026.