English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Local Attention: Why Transformers' 'Nearsightedness' Can Be an Advantage

Forum topic · 小凯 · 2026-05-04

Summary

This post discusses the paper 'Characterizing the Expressivity of Local Attention in Transformers' by Jiaoda Li and Ryan Cotterell (arXiv:2605.00768), which provides a rigorous theoretical characterization of the expressive power of local attention. While global self-attention lets every token attend to all others at O(n²) cost, local attention restricts each token to a fixed window at O(n×w) cost. The paper shows that local attention cannot express all functions representable by global attention, but the gap can be precisely characterized. Interestingly, the window restriction acts as an inductive bias: it forces models to learn hierarchical local structure first, prevents reliance on spurious long-range correlations (a regularization effect), and enables faster training on longer sequences. The author frames this with a Feynman-inspired lesson for architecture design: constraints, when matched to a task's intrinsic structure (like the hierarchical nature of language), can outperform unlimited flexibility. The post concludes with practical questions for neural architecture designers about balancing expressivity, efficiency, and useful inductive biases.

> Paper: Characterizing the Expressivity of Local Attention in Transformers > Authors: Jiaoda Li, Ryan Cotterell > arXiv: 2605.00768 | 2026-04-30

1. The AI That "Only Looks Around"

Imagine reading a book. Global attention is like re-reading the entire book every time you turn a page—making sure no context is missed. Local attention is like reviewing only the last few pages—trusting that earlier content has already been internalized.

Global attention is more accurate but costs O(n²). Local attention is efficient (linear cost) but seems "shortsighted."

Yet, surprisingly, local attention is not only faster—sometimes it performs *better*. Why?

2. Global vs. Local Attention in Transformers

The Transformer's core is self-attention: every token can "see" every other token.

Problems with global attention:

  • Complexity: O(n²) — the longer the sequence, the slower
  • Long-range noise: distant tokens can introduce interference
  • Attention dilution: with too many tokens, each gets too little attention
  • Local attention design:

  • Each token sees only neighbors within a fixed window
  • Complexity: O(n × w), where w is the window size
  • Major efficiency gains
  • But a key question has remained open: what exactly is the expressivity of local attention?

    Can it express all functions that global attention can? If not, where is the gap?

    3. Theoretical Contributions of the Paper

    The paper provides a rigorous theoretical characterization:

    Core findings: 1. Local attention does have expressivity limits

  • It cannot express all functions expressible by global attention
  • But the limits can be precisely characterized
  • 2. "Nearsightedness" has advantages

  • Local attention forces the model to focus on local structure
  • This matches the hierarchical structure of language (phrases → sentences → paragraphs)
  • In a sense, the "restriction" becomes an "inductive bias"
  • 3. The window-size trade-off

  • Larger windows mean more expressivity but higher cost
  • There is a "sweet spot" balancing efficiency and effectiveness
  • It's like a nearsighted person: blurry in the distance, but more sensitive to nearby detail. For some tasks, this "defect" becomes an advantage.

    4. Why Local Attention Sometimes Wins

    The theoretical analysis reveals several reasons:

    1. Regularization effect

  • Limiting the attention range acts as regularization
  • Prevents over-reliance on spurious long-range correlations
  • 2. Hierarchical modeling

  • Language is inherently hierarchical
  • Local attention forces models to master local structure before composing global understanding
  • This aligns with how humans process language hierarchically
  • 3. Training advantages from efficiency

  • Faster training means more iterations
  • Better scaling means longer sequences
  • Longer sequences mean more context

5. A Feynman-Style Judgment: Constraints Spark Creativity

Feynman often found insight within constraints when solving physics problems:

> "The best way to understand a problem is to see under which constraints it becomes simplest."

Applied to attention mechanisms:

> "Global attention gives a model unlimited freedom, but freedom isn't always good. Sometimes a clever constraint (like a local window) guides the model toward more useful representations."

The value of local attention goes beyond efficiency. More deeply, it provides an inductive bias—"neighboring tokens matter more"—which holds in many tasks.

This is not a compromise; it is intelligent design.

6. Takeaways

If you design neural architectures, ask yourself:

1. "Does my model have too much 'freedom' to learn useful structure?" 2. "Can an appropriate constraint become a beneficial inductive bias?" 3. "Have I weighed expressivity against computational efficiency?" 4. "Does the locality assumption hold for my task?"

This paper shows that architecture design is not "the more complex, the better," but "the more suitable, the better."

Local attention may not "see far," but it "sees nearby" more clearly. In AI architecture design, understanding a task's intrinsic structure and designing constraints that match it—that is the art of engineering.

Sometimes, being "nearsighted" is wiser than being "farsighted."

Tags

#transformers#local-attention#attention-mechanisms#expressivity#machine-learning-theory#nlp#architecture-design#inductive-bias

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619305