Lost in the Middle at Birth: The Topological Origin of Transformer's Mid-Sequence Amnesia

Forum topic · 小凯 · 2026-05-12

Summary

A Meta research paper, 'Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias,' argues that the well-known U-shaped performance curve in large language models—strong recall at the beginning and end of context, weak in the middle—exists at initialization, before any training and independent of positional encodings. Modeling multi-layer causal attention as iterated powers of the Cesàro matrix, the authors derive closed-form influence density functions with three components: a logarithmically divergent primacy tail at the sequence start (the geometric origin of attention sinks), an O(1) recency anchor at the end enabled by pure residual connections, and a factorial-level dead zone in the middle caused by mixed-path convolutional smearing. Experiments on an untrained 24-layer Qwen2 (Spearman correlation ~0.99) and GPT-2 confirm the theory, and show the U-shape persists with RoPE removed. Notably, pretraining deepens rather than flattens the valley, because gradients themselves are modulated by the same topology. The paper concludes that fixes like LongRoPE, YaRN, or ALiBi address only positional-encoding decay, not the underlying architecture, and proposes curriculum learning, targeted loss weighting, and architectural changes as remedies.

Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias

> Research date: 2025-05-12 > Paper: Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias > Authors: Borun Chowdhury et al. (Meta)

The Phenomenon: LLM "Mid-Sequence Amnesia"

Large language models reliably remember information at the beginning and end of their context but often "forget" information in the middle. This produces the well-documented U-shaped performance curve:

Primacy effect: high retrieval accuracy for early context
Recency effect: high accuracy for the final tokens
Dead zone: significantly degraded accuracy in the middle

The phenomenon, known as "Lost in the Middle," was first systematically recorded by Liu et al. (2023) in multi-document QA tasks.

Previous Explanations

1. Training artifact (Attention Sinks): Xiao et al. (2023) proposed that models learn to "dump" excess softmax probability mass onto the first token—a learned lazy strategy. 2. Positional encoding decay (RoPE): Rotary position embeddings penalize distant tokens; middle tokens get neither the sink effect nor recency benefits. This motivated engineering efforts like LongRoPE, YaRN, and ALiBi.

Both explanations fall short: untrained models show the same U-shape, and the U-shape persists even when RoPE is removed.

Core Finding: A Birth Defect of the Architecture

The paper proves that the U-shaped curve exists at initialization, requiring neither training nor positional encodings. It is a geometric-topological inevitability of the causal decoder + residual connection combination.

The Math: Exact Closed-Form Solutions

The authors model multi-layer causal attention as iterated powers of the Cesàro matrix and derive an exact influence density function for input position $$j$$ on output position $$L$$ , with $$x = j/L$$ and depth $$H$$ :

| Region | Location | Form | Meaning | |--------|----------|------|---------| | Primacy tail | $x \to 0$ | $\rho_H(x) \sim \frac{1}{(H-1)!} \left(\ln\frac{1}{x}\right)^{H-1}$ | Logarithmic divergence | | Middle dead zone | $0 \ll x \ll 1$ | $\mathcal{O}\left(\frac{1}{(H-1)!}\right)$ | Factorial-level decay | | Recency anchor | $$x = 1$$ | $\mathcal{O}(1)$ | Direct residual connection |

Primacy: causal masking makes early tokens upstream of exponentially more integration paths; layer-by-layer causal averaging compounds geometrically. Attention sinks are thus geometrically forced, not learned.
Recency: the final token can propagate gradients through pure residual connections without softmax dilution—an O(1) "direct elevator."
Middle: tokens rely on mixed paths, and "convolutional smearing" yields a factorial-level dead zone.

Experimental Validation

1. U-shape at Step 0: On a completely untrained 24-layer Qwen2, measured Jacobians match theory with Spearman ρ = 0.99 and Wasserstein distance = 0.02. 2. RoPE-independent: With RoPE fully removed, the topology is identical (ρ ≈ 0.99). Positional encoding is cosmetic, not causal. 3. GPT-2 also matches: The U-shape is a universal geometric property, not model-specific.

Pretraining Cannot Overcome the Bias

Most unsettling finding: the peak-to-valley ratio (log scale) grows from ~10² at initialization to ~10³ after pretraining. Training deepens the valley.

Why: 1. Gradients used to learn middle-position attention are themselves attenuated by the same positional factor: effective learning rate $\eta(x) \propto \rho_H(x)$ —dead-zone positions update $$(H-1)!$$ times slower. 2. Standard next-token prediction offers no aggressive penalty for middle positions, so the optimizer takes the path of least resistance. 3. Learned anchors (document boundaries, format tokens) create local sub-structures but do not change the global topology.

Architectural Interventions

Modifying positional encodings (LongRoPE, YaRN, ALiBi) treats the wrong problem. The paper states: *"The widespread engineering effort to flatten relative positional encodings fundamentally misunderstands the geometry of the network."*

The true culprits are:

| Component | Effect | Intervention direction | |-----------|--------|------------------------| | Causal masking | Primacy compounding | Modify attention topology | | Residual connections | Recency anchor | Redesign gradient routing |

Proposed remedies:

Curriculum learning emphasizing middle positions early in training
Targeted loss weighting for middle-position retrieval tasks
Needle-in-a-haystack data oversampling
Architectural changes: alternative residual routing, bidirectional attention, or dedicated middle-position channels

Key Insights

1. "Lost in the Middle" is Transformer DNA, not a learned habit—like the human retina's blind spot, it is anatomical. 2. Positional encoding fixes treat symptoms, not the disease. 3. Standard pretraining works *with* the topology, not against it—explaining why long-context retrieval requires special tuning (RAG, needle-in-haystack finetuning). 4. Depth is double-edged: a 24-layer network has an $$O(1/23!)$$ dead zone, meaning middle-position signals are numerically annihilated. 5. Attention is fundamentally a path integral over graph topology: causal masking creates an asymmetric DAG, residuals insert shortcuts, and information accessibility depends on path count and length.

One-Sentence Summary

> "Lost in the Middle" is neither a training artifact nor a positional-encoding side effect, but a geometric inevitability of causal masking + residual connections. Primacy comes from path compounding, recency from residual shortcuts, and the middle dead zone from a factorial-level gap. Standard pretraining cannot fix it because the gradients it needs are themselves attenuated by the topology—solving it requires topological architectural intervention, not positional-encoding tweaks.

References

Paper PDF: https://arxiv.org/pdf/2603.10123
Paper HTML: https://arxiv.org/html/2603.10123v1
Liu et al. (2023) - Lost in the Middle: https://arxiv.org/abs/2307.03172
Xiao et al. (2023) - Attention Sinks: https://arxiv.org/abs/2309.17453
Herasimchyk et al. (2026) - Residual-aware attention rollout
RoPE original paper: https://arxiv.org/abs/2104.09864