Local Attention's 'Myopia' Can Be an Advantage: On the Expressivity of Local Attention in Transformers

> Paper: Characterizing the Expressivity of Local Attention in Transformers > Authors: Jiaoda Li, Ryan Cotterell > arXiv: 2605.00768 | 2026-04-30

---

1. The AI That "Only Looks Nearby"

Imagine reading a book. Global attention is like re-reading the entire book every time you turn a page, just to make sure no context is missed. Local attention is like reviewing only the last few pages—trusting that older content has already been internalized.

Global attention is more accurate but quadratic in compute. Local attention is efficient (linear cost) but seemingly 'short-sighted.'

Yet, surprisingly, local attention is not only faster—sometimes it performs *better*. Why?

---

2. Global vs. Local Attention

The core of the Transformer is self-attention: every token can "see" all other tokens.

Problems with global attention:

• O(n²) complexity; slower as sequences grow
• Long-range noise: distant tokens may introduce interference
• Attention dilution: too many tokens, too little attention per token

Local attention design:

• Each token sees only neighbors within a fixed window
• Complexity: O(n×w), where w is the window size
• Major efficiency gains

But a key question remained open: how expressive is local attention? Can it express all functions that global attention can? If not, where does the gap lie?

---

3. The Paper's Theoretical Contributions

The paper rigorously characterizes the expressivity of local attention:

Core findings: 1. Local attention is indeed limited in expressivity

• It cannot express every function global attention can
• But the gap can be precisely characterized

2. "Myopia" has advantages

• The locality constraint forces the model to focus on local structure
• This matches the hierarchical structure of language (phrases → sentences → paragraphs)
• The "limitation" becomes an inductive bias

3. Window-size trade-off

• Larger windows mean more expressivity but higher compute
• There is a "sweet spot" balancing efficiency and effectiveness

---

4. Why Local Attention Is Sometimes Better

1. Regularization effect — restricting the attention range prevents over-reliance on spurious long-range correlations 2. Hierarchical modeling — language is hierarchical; local attention forces the model to learn local structure first and compose it into global understanding, matching human language processing 3. Training advantages from efficiency — faster training means more iterations; better scaling means longer sequences and more context

---

5. A Feynman-Style Judgment: Constraints Spark Creativity

> "The best way to understand a problem is to see under which constraints it becomes simplest."

For attention mechanisms:

> "Global attention gives the model unlimited freedom, but freedom isn't always good. Sometimes a clever constraint—like a local window—guides the model toward more useful representations."

The value of local attention is not just efficiency. More deeply, it provides an inductive bias—"nearby tokens matter more"—which holds in many tasks.

This is not a compromise; it is smart design.

---

6. Takeaways

If you design neural architectures, ask:

1. Does my model have too much "freedom" to learn useful structure? 2. Could the right constraint serve as a beneficial inductive bias? 3. Have I weighed expressivity against computational efficiency? 4. Does the locality assumption hold in my task?

The lesson: architecture design is not "the more complex, the better," but "the more fitting, the better."

Local attention may not "see far," but it sees nearby detail better. Understanding a task's intrinsic structure and designing constraints that match it—that is the art of engineering. Sometimes, "nearsightedness" is wiser than "farsightedness."