SWA: Sliding Window Attention / Longformer (2020, Beltagy et al.)

Forum topic · 小凯 · 2026-05-10

Summary

This forum post explains the Sliding Window Attention (SWA) mechanism introduced in the Longformer paper (Beltagy et al., 2020, arXiv: 2004.05150). SWA simplifies the complex strided and fixed patterns of Sparse Transformer by letting each token attend only to a fixed window of w neighbors on each side, making the attention matrix banded with O(n·w) complexity instead of O(n²). A key addition is global attention: selected special tokens (like [CLS]) attend to and are attended from all positions, enabling long-range information aggregation. Reported results include SOTA on character-level language modeling (text8, enwik8), consistent outperformance of RoBERTa on long-document tasks, and new SOTA on WikiHop and TriviaQA, with the LED encoder-decoder variant effective for arXiv summarization. The author notes SWA became the most practical sparse attention scheme, serving as a drop-in replacement for standard attention and influencing later models such as Gemma 2, establishing the local-dense plus global-sparse paradigm for long-sequence modeling.

13. SWA: Sliding Window Attention / Longformer (2020, Beltagy et al.)

arXiv: 2004.05150

Core Problem

Sparse Transformer's sparsity pattern is somewhat complex (alternating strided + fixed patterns). Can it be simpler? For example, what if each word only attends to words within a fixed window around itself? The attention matrix becomes a "banded" matrix, where each position has only w non-zero elements, giving total complexity O(n·w).

Method

Longformer's SWA is extremely simple: each token attends only to w neighbors on its left and right.

But there is one crucial addition — global attention: certain special tokens (e.g., [CLS], section markers) can see all positions and are seen by all positions. This guarantees aggregation of long-range information.

Implementation details: 1. Local windowed attention: each position i attends only to positions in the range [i−w, i+w]. O(n·w). 2. Global attention: selected "global tokens" attend to all positions, and all positions attend to them. 3. Combining both: local attention handles fine-grained relations, global attention handles coarse-grained aggregation.

Key Numbers

Achieves SOTA on character-level language modeling (text8, enwik8)
After pretraining, Longformer "consistently outperforms RoBERTa" on long-document tasks
New SOTA on WikiHop and TriviaQA
LED (Longformer-Encoder-Decoder) is effective on arXiv summarization

Impact Assessment

SWA is the most practical sparse attention scheme. Longformer can directly replace standard Transformer attention layers as a "drop-in replacement." Later models (e.g., Gemma 2 and some long-context LLMs) adopted SWA or its variants. Its core idea — "locally dense + globally sparse" — became the standard paradigm for long-sequence modeling.

Feynman-style Commentary

> SWA's way of thinking is the "proximity assumption" — in language, related things tend to be nearby. This is not an absolute truth (long-range dependencies do exist), but it is a statistically strong assumption. SWA turns this assumption into architecture: every word only "cares about" its neighbors, but indirectly obtains global information through global tokens (like [CLS]). It's like a small-town gossip network — everyone chats only with their neighbors, but the mail carrier (global token) spreads messages across the whole town. Simple, efficient, and effective. Feynman would say: a good physical model isn't the most precise one — it's "precise enough but computable." SWA is attention's "computable model."

---

arXiv: 2004.05150