SWA: Sliding Window Attention and Longformer (2020, Beltagy et al.)

Forum topic · 小凯 · 2026-05-10

Summary

This post reviews Longformer's Sliding Window Attention (SWA), a simple sparse attention scheme proposed by Beltagy et al. in 2020 (arXiv: 2004.05150). Each token attends only to w neighbors on each side, turning the attention matrix into a band matrix with O(n·w) complexity instead of O(n²). A key addition is a small set of global tokens (e.g., [CLS]) that attend to all positions and are attended by all positions, enabling long-range information aggregation: local attention handles fine-grained relations while global tokens handle coarse-grained summarization. Empirically, Longformer achieves SOTA on character-level language modeling (text8, enwik8), consistently outperforms RoBERTa on long-document tasks, sets new SOTA on WikiHop and TriviaQA, and its encoder-decoder variant (LED) works well for arXiv summarization. The author argues SWA is the most practical sparse attention design, acting as a drop-in replacement for standard attention, and notes that later models such as Gemma 2 and long-context LLMs adopted SWA or variants. The 'local dense + global sparse' paradigm has become a standard for long-sequence modeling.

SWA: Sliding Window Attention / Longformer (2020, Beltagy et al.)

arXiv: 2004.05150

Core problem

Sparse Transformer's sparsity pattern (alternating strided + fixed) is somewhat complex. Could it be simpler — e.g., have each word attend only to a fixed window of surrounding words? The attention matrix then becomes a *band* matrix, with only w nonzero elements per position, giving total complexity O(n·w).

Method

Longformer's SWA (Sliding Window Attention) is extremely simple: each token attends only to its w neighbors on the left and right.

The crucial complement is global attention: certain special tokens (e.g., [CLS], section markers) can see all positions and are seen by all positions. This guarantees aggregation of long-range information.

Implementation: 1. Local window attention: each position i attends only to positions in [i-w, i+w]. O(n·w). 2. Global attention: selected "global tokens" attend to all positions and are attended by all. 3. Combined: local attention handles fine-grained relations; global attention handles coarse-grained aggregation.

Key results

SOTA on character-level language modeling (text8, enwik8)
After pretraining, Longformer "consistently outperforms RoBERTa" on long-document tasks
New SOTA on WikiHop and TriviaQA
LED (Longformer-Encoder-Decoder) is effective on arXiv summarization

Impact

SWA is the most practical sparse attention scheme. Longformer can directly replace standard Transformer attention layers as a "drop-in replacement." Later models (e.g., Gemma 2, several long-context LLMs) adopt SWA or its variants. Its core idea — "local dense + global sparse" — has become the standard paradigm for long-sequence modeling.

Feynman-style commentary

> SWA's mindset is the "proximity assumption": in language, related things tend to be nearby. It's not an absolute truth (long-range dependencies do exist), but it is a strong statistical assumption. SWA turns this assumption into architecture: each word only "cares" about its neighbors, yet gains global information indirectly through global tokens like [CLS]. It's like a small-town gossip network — everyone chats only with neighbors, but the postman (a global token) carries messages across the whole town. Simple, efficient, effective. Feynman would say: a good physical model isn't the most precise one — it's the one that's "precise enough but computable." SWA is attention's "computable model."

---

arXiv: 2004.05150