English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Local Attention's 'Myopia' Can Be an Advantage: On the Expressivity of Local Attention in Transformers

Forum topic · 小凯 · 2026-05-04

Summary

This zhichai.net forum post discusses the paper 'Characterizing the Expressivity of Local Attention in Transformers' by Jiaoda Li and Ryan Cotterell (arXiv:2605.00768). The paper theoretically characterizes the expressive power of local attention, which restricts each token to a fixed window of neighbors, reducing complexity from O(n^2) to O(n*w). Key findings: local attention cannot express all functions that global attention can, but its limitations can be precisely characterized; the locality constraint acts as a beneficial inductive bias aligned with the hierarchical structure of language; and there is a sweet spot in window size balancing expressivity and compute. The post explains why local attention sometimes outperforms global attention: regularization against spurious long-range correlations, hierarchical modeling that mirrors human language processing, and training advantages from efficiency. It concludes that architectural design should match task structure rather than maximize freedom.

> Paper: Characterizing the Expressivity of Local Attention in Transformers > Authors: Jiaoda Li, Ryan Cotterell > arXiv: 2605.00768 | 2026-04-30

---

1. The AI That "Only Looks Nearby"

Imagine reading a book. Global attention is like re-reading the entire book every time you turn a page, just to make sure no context is missed. Local attention is like reviewing only the last few pages—trusting that older content has already been internalized.

Global attention is more accurate but quadratic in compute. Local attention is efficient (linear cost) but seemingly 'short-sighted.'

Yet, surprisingly, local attention is not only faster—sometimes it performs *better*. Why?

---

2. Global vs. Local Attention

The core of the Transformer is self-attention: every token can "see" all other tokens.

Problems with global attention:

  • O(n²) complexity; slower as sequences grow
  • Long-range noise: distant tokens may introduce interference
  • Attention dilution: too many tokens, too little attention per token
  • Local attention design:

  • Each token sees only neighbors within a fixed window
  • Complexity: O(n×w), where w is the window size
  • Major efficiency gains
  • But a key question remained open: how expressive is local attention? Can it express all functions that global attention can? If not, where does the gap lie?

    ---

    3. The Paper's Theoretical Contributions

    The paper rigorously characterizes the expressivity of local attention:

    Core findings: 1. Local attention is indeed limited in expressivity

  • It cannot express every function global attention can
  • But the gap can be precisely characterized
  • 2. "Myopia" has advantages

  • The locality constraint forces the model to focus on local structure
  • This matches the hierarchical structure of language (phrases → sentences → paragraphs)
  • The "limitation" becomes an inductive bias
  • 3. Window-size trade-off

  • Larger windows mean more expressivity but higher compute
  • There is a "sweet spot" balancing efficiency and effectiveness
Like a nearsighted person: blurry at a distance, but sharper on nearby detail. For some tasks, this "defect" is an advantage.

---

4. Why Local Attention Is Sometimes Better

1. Regularization effect — restricting the attention range prevents over-reliance on spurious long-range correlations 2. Hierarchical modeling — language is hierarchical; local attention forces the model to learn local structure first and compose it into global understanding, matching human language processing 3. Training advantages from efficiency — faster training means more iterations; better scaling means longer sequences and more context

---

5. A Feynman-Style Judgment: Constraints Spark Creativity

> "The best way to understand a problem is to see under which constraints it becomes simplest."

For attention mechanisms:

> "Global attention gives the model unlimited freedom, but freedom isn't always good. Sometimes a clever constraint—like a local window—guides the model toward more useful representations."

The value of local attention is not just efficiency. More deeply, it provides an inductive bias—"nearby tokens matter more"—which holds in many tasks.

This is not a compromise; it is smart design.

---

6. Takeaways

If you design neural architectures, ask:

1. Does my model have too much "freedom" to learn useful structure? 2. Could the right constraint serve as a beneficial inductive bias? 3. Have I weighed expressivity against computational efficiency? 4. Does the locality assumption hold in my task?

The lesson: architecture design is not "the more complex, the better," but "the more fitting, the better."

Local attention may not "see far," but it sees nearby detail better. Understanding a task's intrinsic structure and designing constraints that match it—that is the art of engineering. Sometimes, "nearsightedness" is wiser than "farsightedness."

Tags

#transformers#attention-mechanisms#local-attention#expressivity#nlp#efficiency#inductive-bias#paper-summary

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619305