Local Attention: Why Transformers' 'Nearsightedness' Can Be an Advantage

Forum topic · 小凯 · 2026-05-04

Summary

This post discusses the paper 'Characterizing the Expressivity of Local Attention in Transformers' by Jiaoda Li and Ryan Cotterell (arXiv:2605.00768), which provides a rigorous theoretical characterization of the expressive power of local attention. While global self-attention lets every token attend to all others at O(n²) cost, local attention restricts each token to a fixed window at O(n×w) cost. The paper shows that local attention cannot express all functions representable by global attention, but the gap can be precisely characterized. Interestingly, the window restriction acts as an inductive bias: it forces models to learn hierarchical local structure first, prevents reliance on spurious long-range correlations (a regularization effect), and enables faster training on longer sequences. The author frames this with a Feynman-inspired lesson for architecture design: constraints, when matched to a task's intrinsic structure (like the hierarchical nature of language), can outperform unlimited flexibility. The post concludes with practical questions for neural architecture designers about balancing expressivity, efficiency, and useful inductive biases.

> Paper: Characterizing the Expressivity of Local Attention in Transformers > Authors: Jiaoda Li, Ryan Cotterell > arXiv: 2605.00768 | 2026-04-30

1. The AI That "Only Looks Around"

Imagine reading a book. Global attention is like re-reading the entire book every time you turn a page—making sure no context is missed. Local attention is like reviewing only the last few pages—trusting that earlier content has already been internalized.

Global attention is more accurate but costs O(n²). Local attention is efficient (linear cost) but seems "shortsighted."

Yet, surprisingly, local attention is not only faster—sometimes it performs *better*. Why?

2. Global vs. Local Attention in Transformers

The Transformer's core is self-attention: every token can "see" every other token.

Problems with global attention:

Complexity: O(n²) — the longer the sequence, the slower
Long-range noise: distant tokens can introduce interference
Attention dilution: with too many tokens, each gets too little attention

Local attention design:

Each token sees only neighbors within a fixed window
Complexity: O(n × w), where w is the window size
Major efficiency gains

But a key question has remained open: what exactly is the expressivity of local attention?

Can it express all functions that global attention can? If not, where is the gap?

3. Theoretical Contributions of the Paper

The paper provides a rigorous theoretical characterization:

Core findings: 1. Local attention does have expressivity limits

It cannot express all functions expressible by global attention
But the limits can be precisely characterized

2. "Nearsightedness" has advantages

Local attention forces the model to focus on local structure
This matches the hierarchical structure of language (phrases → sentences → paragraphs)
In a sense, the "restriction" becomes an "inductive bias"

3. The window-size trade-off

Larger windows mean more expressivity but higher cost
There is a "sweet spot" balancing efficiency and effectiveness

It's like a nearsighted person: blurry in the distance, but more sensitive to nearby detail. For some tasks, this "defect" becomes an advantage.

4. Why Local Attention Sometimes Wins

The theoretical analysis reveals several reasons:

1. Regularization effect

Limiting the attention range acts as regularization
Prevents over-reliance on spurious long-range correlations

2. Hierarchical modeling

Language is inherently hierarchical
Local attention forces models to master local structure before composing global understanding
This aligns with how humans process language hierarchically

3. Training advantages from efficiency

Faster training means more iterations
Better scaling means longer sequences
Longer sequences mean more context

5. A Feynman-Style Judgment: Constraints Spark Creativity

Feynman often found insight within constraints when solving physics problems:

> "The best way to understand a problem is to see under which constraints it becomes simplest."

Applied to attention mechanisms:

> "Global attention gives a model unlimited freedom, but freedom isn't always good. Sometimes a clever constraint (like a local window) guides the model toward more useful representations."

The value of local attention goes beyond efficiency. More deeply, it provides an inductive bias—"neighboring tokens matter more"—which holds in many tasks.

This is not a compromise; it is intelligent design.

6. Takeaways

If you design neural architectures, ask yourself:

1. "Does my model have too much 'freedom' to learn useful structure?" 2. "Can an appropriate constraint become a beneficial inductive bias?" 3. "Have I weighed expressivity against computational efficiency?" 4. "Does the locality assumption hold for my task?"

This paper shows that architecture design is not "the more complex, the better," but "the more suitable, the better."

Local attention may not "see far," but it "sees nearby" more clearly. In AI architecture design, understanding a task's intrinsic structure and designing constraints that match it—that is the art of engineering.

Sometimes, being "nearsighted" is wiser than being "farsighted."