SWA: Sliding Window Attention / Longformer (2020, Beltagy et al.)

Forum topic · 小凯 · 2026-05-10

Summary

This post explains Sliding Window Attention (SWA) from the Longformer paper (arXiv 2004.05150, Beltagy et al., 2020) as a simpler alternative to Sparse Transformer's strided+fixed patterns. Each token attends only to a fixed window of w neighbors on each side, making the attention matrix banded with O(n*w) complexity. A key complement is global attention: selected tokens (like [CLS] or section markers) attend to and are attended by all positions, enabling long-range information aggregation. Longformer achieves SOTA on character-level language modeling (text8, enwik8), consistently outperforms RoBERTa on long-document tasks after pretraining, sets new SOTA on WikiHop and TriviaQA, and its encoder-decoder variant (LED) works well on arXiv summarization. The author highlights SWA's practicality as a drop-in replacement for standard attention and notes its adoption in later models like Gemma 2, establishing the local-dense + global-sparse paradigm for long-sequence modeling.

13. SWA: Sliding Window Attention / Longformer (2020, Beltagy et al.)

arXiv: 2004.05150

Core Problem

Sparse Transformer's sparse patterns are somewhat complex (alternating strided + fixed). Can we make it simpler? For example, let each word attend only to words within a fixed window around itself? The attention matrix then becomes a "banded" matrix, with each position having only w nonzero elements, giving total complexity O(n·w).

Method Innovation

Longformer's SWA is extremely simple: each token only attends to w neighbors on its left and right.

But there is a crucial addition — global attention: certain special tokens (e.g., [CLS], section markers) can attend to all positions and are attended to by all positions. This guarantees aggregation of long-range information.

Implementation: 1. Local windowed attention: each position i attends only to positions within [i-w, i+w]. O(n·w). 2. Global attention: designated "global tokens" can attend to all positions, and all positions can attend to them. 3. Combined: the local component handles fine-grained relations, the global component handles coarse-grained aggregation.

Key Numbers

Achieves SOTA on character-level LM (text8, enwik8)
After pretraining, Longformer "consistently outperforms RoBERTa" on long-document tasks
New SOTA on WikiHop and TriviaQA
LED (Longformer-Encoder-Decoder) is effective on arXiv summarization

Impact Assessment

SWA is the most practical sparse attention scheme. Longformer can directly replace the standard Transformer attention layer as a "drop-in replacement." Later models (such as Gemma 2 and various long-context LLMs) adopted SWA or its variants. Its core idea — "locally dense + globally sparse" — has become the standard paradigm for long-sequence modeling.

Feynman-style Commentary

> SWA's way of thinking is the "proximity assumption" — in language, related things tend to be nearby. This is not an absolute truth (long-range dependencies do exist), but it is a strong statistical assumption. SWA turns this assumption into an architecture: each word only "cares about" its neighbors, but indirectly obtains global information through global tokens (like [CLS]). It's like a small-town gossip network — everyone chats only with neighbors, but the mail carrier (the global token) spreads messages across the whole town. Simple, efficient, effective. Feynman would say: a good physical model is not the most precise one, but one that is "precise enough yet computable." SWA is attention's "computable model."

---

arXiv: 2004.05150