The Two-Hop Reasoning Paradox: Transformers Know Each Hop but Cannot Compose Them

> Paper link: https://arxiv.org/abs/2608.07261

A Confusing Failure Case

Ask an LLM: "What is the alma mater of the wife of the author of *One Hundred Years of Solitude*?"

This is a two-hop query:

• Hop 1: Who is the wife of Gabriel García Márquez? (Mercedes Barcha)
• Hop 2: What is the alma mater of Mercedes Barcha?

A model can answer each hop in isolation correctly. But when the two hops are chained, it produces nonsense.

The model clearly knows each hop individually. Why can it not compose them?

This is the "two-hop generalization paradox" that the paper investigates. Rather than stopping at "the model is bad at reasoning," the authors dig into the internal mechanisms of transformers and arrive at a counterintuitive answer.

Experiments: Starting from a Controlled Environment

To eliminate interference from pretraining data, the authors train transformers from scratch on a fully controllable symbolic knowledge graph. Each training example is a triple (head entity, relation, tail entity), and the model must learn to answer two-hop queries: given a head entity and two relations, predict the final tail entity.

Key finding: when the second-hop relation pattern is in-distribution (ID), the model is nearly 100% accurate; when it is even slightly out-of-distribution (OOD), accuracy collapses to around 30%.

Crucially, hop 1 remains correct. The model finds the correct bridge entity. The problem lies in hop 2: even though the bridge entity is present in the context, the model cannot use it.

Mechanism: Upper Layers Are Doing "Lookup," Not "Reasoning"

This is the most interesting part of the paper. The authors apply a battery of mechanistic interpretability tools.

Step 1: Locate the bridge-entity representation. Using logit lens and activation patching, the authors find that the bridge entity is correctly represented in middle layers (roughly layers 4–8). The model "sees" the bridge entity; it exists in the hidden space.

Step 2: Probing experiments. The authors train a linear probe on the middle-layer representations to see whether they encode the correct final tail entity. Result: the middle-layer representations already contain the correct tail-entity information. The answer is there, latent in the residual stream.

Step 3: Why does it not emerge? This is the crux. The authors show that the upper layers (the last few) perform "mapping" rather than "reasoning."

In distribution, the upper layers learn a direct lookup table from bridge entity to tail entity. This mapping is rigid. It is not derived from relational reasoning; it is hard-coded from the (bridge, tail) pairs seen during training. As soon as a novel bridge entity appears, the lookup table fails.

An analogy: suppose you memorize 100 two-digit addition problems, including 23+45=68 and 12+34=46. You can answer 23+45 on the test (lookup hits), but you freeze on 23+46 (lookup misses), even though you "know" 3+6=9 and 20+40=60. You memorized answers, not the algorithm.

The transformer's upper layers are exactly that student. Inside the training distribution, they memorize a (bridge entity → tail entity) lookup table. Given a new bridge entity, they lack any compositional procedure to derive the answer.

Solutions: Forcing the Upper Layers to Actually Reason

Since the problem is that upper layers do mapping instead of reasoning, the authors force them to learn reasoning. Two approaches:

Approach 1: Explicit representation-level training of the upper layers. Freeze the lower layers and train only the upper layers to derive the tail entity from the middle-layer bridge representation, rather than querying a lookup table. This works but requires carefully designed training data.

Approach 2: Looped Transformer. Share parameters across upper layers so that the model effectively "thinks multiple times." Each loop iteration reprocesses the middle-layer representation. This forces the upper layers to learn a general reasoning procedure rather than a fixed mapping.

Result: the Looped Transformer improves OOD two-hop accuracy from ~30% to 80%+ without sacrificing ID performance. Logit lens analyses confirm that, after loop training, the upper layers genuinely perform representation-based reasoning rather than memorizing a larger lookup table.

Why This Paper Matters

1. It pinpoints a specific mechanistic defect. Instead of vague diagnoses like "reasoning is weak," the paper identifies an actionable failure mode: upper layers do mapping rather than reasoning.

2. The Looped Architecture has deeper implications. It effectively turns a transformer into something analogous to a prefrontal cortex that reprocesses the same information to extract different features. This echoes the temporal unfolding of RNNs, but with a different motivation: not for handling sequences, but for giving reasoning "thinking time."

3. It sheds light on Chain-of-Thought (CoT). CoT may help not because it forces the model to verbalize intermediate steps, but because generating those steps forces the model to re-represent information in context, giving upper layers a second reasoning pass. The Looped Architecture internalizes this process into the model itself, removing the need for external CoT.

4. It has direct implications for knowledge-graph-augmented LLMs. RAG systems frequently encounter the failure mode of "the right evidence was retrieved, but the model cannot use it." The paper's diagnosis, that upper layers do mapping rather than reasoning, may explain this phenomenon at a mechanistic level.

Honest Assessment

The paper has limitations:

• The experimental environment is symbolic and controlled. Real-world two-hop reasoning involves natural language, entity disambiguation, and relation ambiguity at a far higher complexity. The authors verify their findings on real datasets in the appendix, but only at limited scale.
• The mechanism behind Looped Architecture gains needs more depth. Why does parameter sharing beat independent parameters? Is it regularization, or something deeper? The paper offers partial evidence but no conclusive answer.
• The "mapping vs. reasoning" distinction rests on probing. Information detectable by a probe is not necessarily information the model uses. Bridging that gap requires stricter causal intervention experiments.

Still, this is an elegant mechanistic interpretability paper. Rather than "opening the black box to peek inside," it identifies a concrete computational defect and then fixes it. That diagnosis-and-repair loop is what makes mechanistic interpretability truly valuable.

---

Reference: Zhang, Wang, Wang, Wan, Luo. *Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models*. arXiv:2608.07261, 2026.