Longformer and Sliding Window Attention (SWA): A Simple, Practical Sparse Attention Scheme

Forum topic · 小凯 · 2026-05-10

Summary

This forum post analyzes the Longformer paper (Beltagy et al., arXiv:2004.05150), which introduced Sliding Window Attention (SWA) as a simpler alternative to Sparse Transformer's more complex strided and fixed patterns. In SWA, each token attends only to a fixed window of w neighbors on each side, turning the attention matrix into a banded matrix with O(n·w) complexity. The key complement is global attention: selected special tokens (like [CLS]) can attend to and be attended by all positions, enabling long-range information aggregation through a local-dense + global-sparse design. Reported results include SOTA on character-level language modeling (text8, enwik8), consistent outperformance of RoBERTa on long-document tasks, new SOTA on WikiHop and TriviaQA, and an effective Longformer-Encoder-Decoder (LED) for arXiv summarization. The author notes SWA's practicality as a drop-in replacement for standard attention layers and its lasting influence—later models such as Gemma 2 and various long-context LLMs adopted SWA or variants. The post frames SWA's underlying 'proximity assumption' as a strong statistical prior that is sufficiently accurate yet computationally tractable.

Paper: Beltagy et al. (2020). *Longformer: The Long-Document Transformer.* arXiv:2004.05150

Core Problem

Sparse Transformer's sparsity patterns are somewhat complicated (alternating strided + fixed patterns). Can we do something simpler—e.g., have each word attend only to words within a fixed window around itself? This turns the attention matrix into a "banded" matrix where each position has only w non-zero elements, for total complexity O(n·w).

Method: Sliding Window Attention + Global Attention

Longformer's SWA (Sliding Window Attention) is extremely simple: each token only attends to w neighbors on its left and right.

But there is a crucial supplement—global attention: certain special tokens (e.g., [CLS], section markers) can attend to all positions and are attended to by all positions. This guarantees aggregation of long-range information.

Implementation details:

1. Local windowed attention: each position i attends only to positions in the range [i−w, i+w]. O(n·w). 2. Global attention: selected "global tokens" attend to all positions, and all positions attend to them. 3. Combined: local attention handles fine-grained relations, global attention handles coarse-grained aggregation.

Key Results

SOTA on character-level language modeling (text8, enwik8)
After pretraining, Longformer "consistently outperforms RoBERTa" on long-document tasks
New SOTA on WikiHop and TriviaQA
LED (Longformer-Encoder-Decoder) is effective on arXiv summarization

Impact Assessment

SWA is the most practically adopted sparse attention scheme. Longformer can directly replace the standard Transformer attention layer as a "drop-in replacement." Later models (e.g., Gemma 2 and several long-context LLMs) have adopted SWA or its variants. Its core idea—"locally dense + globally sparse"—has become a standard paradigm for long-sequence modeling.

Commentary (Feynman-style)

> SWA's way of thinking is the "proximity assumption": in language, related things tend to be nearby. This is not an absolute truth (long-range dependencies do exist), but it is a strong statistical assumption. SWA turns this assumption into architecture: each word only "cares" about its neighbors, but indirectly obtains global information via global tokens (like [CLS]). It's like a small-town gossip network—everyone chats only with neighbors, but the postman (the global token) carries messages across the whole town. Simple, efficient, effective. Feynman would say: a good physical model is not the most precise one, but one that is "precise enough yet computable." SWA is attention's "computable model."

Reference paper: Beltagy et al. (2020). *Longformer: The Long-Document Transformer.* arXiv:2004.05150