Longformer's Sliding Window Attention (SWA): A Simple Sparse Attention for Long Documents

Forum topic · 小凯 · 2026-05-10

Summary

This forum post analyzes the Sliding Window Attention (SWA) mechanism from the Longformer paper (Beltagy et al., 2020, arXiv:2004.05150). SWA simplifies sparse attention by letting each token attend only to a fixed window of w neighbors on each side, turning the attention matrix into a banded matrix with O(n·w) complexity instead of O(n²). The author highlights a crucial addition: selected global tokens (such as [CLS]) attend to and are attended by all positions, enabling long-range information aggregation. The combination of local windowed attention and global attention handles fine-grained local relations and coarse-grained global context respectively. Reported results include state-of-the-art performance on character-level language modeling (text8, enwik8), consistently outperforming RoBERTa on long-document tasks, new SOTA on WikiHop and TriviaQA, and the Longformer-Encoder-Decoder (LED) being effective for arXiv summarization. The post argues SWA is the most practical sparse attention design, serving as a drop-in replacement for standard attention, and notes its adoption in later models like Gemma 2. A Feynman-style commentary frames SWA as encoding a 'proximity assumption' about language—related content tends to be nearby—made computationally tractable through local density plus global sparsity.

Overview

This post discusses Sliding Window Attention (SWA) from the Longformer paper (Beltagy et al., 2020, arXiv:2004.05150), framing it as a simpler alternative to the more complicated sparsity patterns of Sparse Transformer (strided + fixed alternation).

Core Problem

Can sparse attention be made simpler? What if each word only attends to words within a fixed window around itself? The attention matrix becomes a banded matrix where each position has only w non-zero elements, giving total complexity of O(n·w).

Method

Longformer's SWA is extremely simple: each token attends only to its w neighbors on the left and right. But there is a key addition — global attention:

1. Local windowed attention: position i attends only to positions in [i-w, i+w]. Complexity O(n·w). 2. Global attention: selected "global tokens" (e.g., [CLS], section markers) attend to all positions and are attended by all positions. 3. Combination: local attention handles fine-grained relations; global tokens handle coarse-grained aggregation, guaranteeing long-range information flow.

Key Results

SOTA on character-level language modeling (text8, enwik8)
Pretrained Longformer consistently outperforms RoBERTa on long-document tasks
New SOTA on WikiHop and TriviaQA
LED (Longformer-Encoder-Decoder) is effective for arXiv summarization

Impact

SWA is the most practical sparse attention scheme. Longformer works as a drop-in replacement for standard Transformer attention layers. Later models (e.g., Gemma 2 and various long-context LLMs) adopted SWA or variants. Its core idea — local dense + global sparse — has become a standard paradigm for long-sequence modeling.

Commentary (Feynman-style)

> SWA's underlying mindset is the "proximity assumption" — in language, related things tend to be nearby. This is not an absolute truth (long-range dependencies do exist), but it is a statistically strong assumption. SWA turns this assumption into architecture: each word only "cares" about its neighbors, but obtains global information indirectly through global tokens like [CLS]. It is like a small-town gossip network — everyone chats only with neighbors, but the postman (the global token) carries messages across the whole town. Simple, efficient, effective. A good physical model is not the most precise one, but the one that is "precise enough yet computable." SWA is attention's computable model.

References

Beltagy et al. (2020). *Longformer: The Long-Document Transformer*. arXiv:2004.05150