Local Attention in Transformers: How Myopia Can Be a Strength

Forum topic · 小凯 · 2026-05-04

Summary

This post discusses the paper "Characterizing the Expressivity of Local Attention in Transformers" by Jiaoda Li and Ryan Cotterell (arXiv: 2605.00768), which provides a rigorous theoretical characterization of local attention's expressive power. Unlike global attention, which scales quadratically at O(n²), local attention restricts each token to a fixed window of neighbors, achieving linear cost. The paper shows that local attention cannot express every function global attention can, but its limitations can be precisely characterized—and these constraints can act as beneficial inductive biases. Forcing models to focus on local structure matches the hierarchical organization of language (phrases, sentences, paragraphs), acts as a regularizer against spurious long-range correlations, and improves training efficiency, allowing longer sequences and more iterations. The post explains why local attention sometimes outperforms global attention despite its "near-sightedness," drawing an analogy to how constraints can spark insight in physics. It concludes with design lessons for neural architects: question whether your model has too much freedom, whether smart constraints can serve as inductive biases, and whether expressiveness and efficiency are properly balanced. Architecture design, it argues, is about fit—not complexity.

> Paper: Characterizing the Expressivity of Local Attention in Transformers > Authors: Jiaoda Li, Ryan Cotterell > arXiv: 2605.00768 | 2026-04-30

1. The AI That "Only Looks Nearby"

Imagine reading a book. Global attention is like rereading the entire book every time you turn to a new page—making sure no context is missed. Local attention is like reviewing only the last few pages—trusting that earlier content has already been internalized.

Global attention is more precise, but its compute cost scales quadratically. Local attention is efficient (linear cost), but seems to "not see far enough."

Yet, surprisingly, local attention is not only faster—it sometimes performs *better*. Why?

2. Attention in Transformers: Global vs. Local

Self-attention lets every token "see" all other tokens.

Problems with global attention:

Complexity: O(n²) — slower as sequences grow
Long-range noise: distant tokens can introduce interference
Attention dilution: too many tokens means each receives too little attention

The design of local attention:

Each token only attends to neighbors within a fixed window
Complexity: O(n×w), where w is the window size
Major efficiency gains

But a key question has remained open: what exactly is the expressivity of local attention?

Can it express all functions that global attention can? If not, where is the gap?

3. Theoretical Contributions of the Paper

The paper rigorously characterizes the expressive power of local attention:

Key findings: 1. Local attention does have expressivity limitations

It cannot express every function global attention can
But the exact form of these limitations can be precisely characterized

2. "Myopia" can be an advantage

Local attention forces the model to focus on local structure
This matches the hierarchical structure of language (phrases → sentences → paragraphs)
In a sense, the "restriction" becomes an "inductive bias"

3. The window-size trade-off

Larger windows mean more expressivity but higher cost
There is a "sweet spot" balancing efficiency and effectiveness

It's like a near-sighted person: unable to see far, but more sensitive to nearby details. For some tasks, this "defect" becomes an advantage.

4. Why Local Attention Sometimes Wins

The theoretical analysis reveals several reasons:

1. Regularization effect

Constraining the attention range acts as regularization
Prevents over-reliance on spurious long-distance correlations

2. Hierarchical modeling

Language is inherently hierarchical
Local attention forces the model to master local structure first, then compose global understanding
This aligns with how humans process language hierarchically

3. Training advantages from efficiency

Faster training enables more iterations
Better scaling enables longer sequences
Longer sequences mean more context

5. A Feynman-Style Judgment: Constraints Spark Creativity

Feynman often found insight within constraints when solving physics problems:

> "The best way to understand a problem is to see under which constraints it becomes simplest."

Applied to attention mechanisms:

> "Global attention gives the model unlimited freedom, but freedom is not always good. Sometimes a clever constraint—like a local window—guides the model toward more useful representations."

Local attention's value goes beyond efficiency. More deeply, it provides an inductive bias—"nearby tokens are more relevant"—a bias that holds for many tasks.

This is not a compromise; it is intelligent design.

6. Takeaway Lessons

If you design neural architectures, ask yourself:

1. "Does my model have so much 'freedom' that it fails to learn useful structure?" 2. "Can appropriate constraints become beneficial inductive biases?" 3. "Have I balanced expressivity and computational efficiency?" 4. "Does a locality assumption hold for my task?"

This paper teaches us: architecture design is not "the more complex, the better," but "the more fitting, the better."

Local attention may "not see far," but it excels at "seeing clearly up close." In AI architecture design, understanding a task's intrinsic structure and designing constraints that match it—that is the art of engineering.

Sometimes, being near-sighted is wiser than being far-sighted.