Transformer Explained: Why Attention Is All You Need (Vaswani et al., 2017)

Forum topic · 小凯 · 2026-05-10

Summary

This forum post breaks down the 2017 landmark paper "Attention Is All You Need" (arXiv: 1706.03762v7) by Vaswani et al., which introduced the Transformer architecture. It explains why recurrent networks bottleneck machine translation through sequential processing and why CNNs need deep stacking for long-range dependencies. The Transformer solves this with self-attention, letting every token interact with all others in parallel. Three key design choices are highlighted: multi-head attention (8 heads learning different relation types), sinusoidal positional encoding that gives each position a unique fingerprint across frequencies, and residual connections with Layer Normalization for stable deep training. Benchmark results include 28.4 BLEU on WMT 2014 English-German (2 BLEU above the previous best, including ensembles), 41.8 BLEU on English-French, trained on 8 GPUs in 3.5 days, with 65M (base) and 213M (large) parameters. The post argues the paper's legacy is defining the entire LLM era—GPT, BERT, T5, LLaMA, and DeepSeek all build on Transformer—by proving recurrence and convolution are unnecessary when a general pairwise attention mechanism is combined with positional labels and residual pathways.

Transformer: Attention Is All You Need (2017, Vaswani et al.)

arXiv: 1706.03762v7

The Core Problem

In machine translation, an RNN acts like a word-by-word interpreter—each token must wait for the previous one to be processed. This sequential dependency makes training slow, and information from early tokens degrades over long sentences. CNNs allow some parallelism but require many stacked layers to capture long-range relationships. Is there a way for all tokens to "see each other" at once—parallel, fast, and without losing long-distance information?

The Method

The paper's answer is the attention mechanism: instead of passing information like a relay baton layer by layer, every token directly "talks" to all other tokens. Like a meeting where everyone speaks simultaneously, a softmax-based importance scoring system lets each token focus only on the most relevant others.

Three key architectural designs:

1. Multi-Head Attention (MHA): Each token's representation is split into 8 heads, each independently learning different relation types—syntax, semantics, coreference—like 8 experts analyzing the same text simultaneously. 2. Positional Encoding (PE): Since attention is position-agnostic, the paper uses sinusoidal/cosine encodings—waves of different frequencies give each position a "fingerprint" added to the token embedding. High frequencies distinguish nearby positions; low frequencies distinguish distant ones. 3. Residual Connections + LayerNorm: Information can take "shortcuts" (residuals) or undergo "deep processing" (attention/FFN), with LayerNorm preventing numerical explosion.

Key Numbers

WMT 2014 English→German: 28.4 BLEU, beating the prior best (including ensembles) by 2 BLEU
WMT 2014 English→French: 41.8 BLEU, new single-model SOTA
Training cost: 8 GPUs, 3.5 days—a fraction of the best prior models
Parameters: 65M (base), 213M (large)—tiny by today's standards, massive at the time

Impact

This paper defined the entire LLM era. GPT, BERT, T5, LLaMA, DeepSeek—every modern large model is built on the Transformer skeleton. It proved that "attention is enough": no recurrence, no convolution—pure attention plus feed-forward layers can do it all.

Feynman-Style Commentary

> The paper's true value isn't inventing attention—attention existed in earlier work. Its real contribution is proving one thing: when you design a sufficiently general "mutual attention" mechanism, add "position labels" and an "information highway" (residuals), recurrence and convolution become redundant. It makes you rethink whether deep learning's core is step-by-step processing or global association. The answer is the latter.

---

arXiv: 1706.03762v7