SWA: Sliding Window Attention and Longformer (2020, Beltagy et al.)

Forum topic · 小凯 · 2026-05-10

Summary

This post explains Sliding Window Attention (SWA) from the Longformer paper (arXiv: 2004.05150, Beltagy et al., 2020) as a simpler alternative to the Sparse Transformer's strided-plus-fixed pattern. SWA restricts each token to attend only to w neighbors on each side, turning the attention matrix into a banded matrix with O(n·w) complexity. A key complement is global attention: special tokens like [CLS] attend to all positions and are attended by all, enabling long-range information aggregation. Results cited include SOTA on character-level language modeling (text8, enwik8), consistently outperforming RoBERTa on long-document tasks, new SOTA on WikiHop and TriviaQA, and the Longformer-Encoder-Decoder (LED) being effective for arXiv summarization. The author argues SWA is the most practical sparse attention scheme, usable as a drop-in replacement for standard attention, and notes that later models such as Gemma 2 and some long-context LLMs adopted SWA or variants. A Feynman-style commentary frames the design as a 'locality assumption': nearby words tend to be relevant, while global tokens act like messengers spreading information across the whole sequence.

13. SWA: Sliding Window Attention / Longformer (2020, Beltagy et al.)

arXiv: 2004.05150

Core Problem

The Sparse Transformer's sparse attention pattern is somewhat complicated (alternating strided + fixed patterns). Could it be simpler? What if each word only attends to words within a fixed window around itself? The attention matrix then becomes a "banded" matrix: each position has only w non-zero elements, giving total complexity O(n·w).

Method

Longformer's SWA is extremely simple: each token attends only to its w neighbors on the left and right.

But there is one crucial addition — global attention: certain special tokens (e.g., [CLS], section markers) can see all positions and are seen by all positions. This guarantees aggregation of long-range information.

Implementation details:

1. Local window attention: each position i attends only to positions in [i−w, i+w]. O(n·w). 2. Global attention: selected "global tokens" attend to all positions, and all positions attend to them. 3. The combination: local attention handles fine-grained relations, global attention handles coarse-grained aggregation.

Key Numbers

SOTA on character-level language modeling (text8, enwik8)
After pretraining, Longformer "consistently outperforms RoBERTa" on long-document tasks
New SOTA on WikiHop and TriviaQA
LED (Longformer-Encoder-Decoder) is effective on arXiv summarization

Impact Assessment

SWA is the most practical sparse attention scheme. Longformer can directly replace standard Transformer attention layers as a "drop-in replacement." Later models (e.g., Gemma 2 and some long-context LLMs) have adopted SWA or its variants. Its core idea — "locally dense + globally sparse" — has become a standard paradigm for long-sequence modeling.

Feynman-Style Commentary

> SWA's way of thinking is the "locality assumption" — in language, related things tend to be nearby. This is not an absolute truth (long-range dependencies do exist), but it is a strong statistical assumption. SWA turns this assumption into architecture: each word only "cares about" its neighbors, but indirectly obtains global information through global tokens like [CLS]. It is like a small-town gossip network — everyone only chats with their neighbors, but the postman (the global token) carries messages across the whole town. Simple, efficient, effective. Feynman would say: a good physical model is not the most precise one, but the one that is "precise enough yet computable." SWA is attention's "computable model."