Transformer: Attention Is All You Need (2017, Vaswani et al.) — Explained

Forum topic · 小凯 · 2026-05-10

Summary

This post is a Chinese-language deep-dive analysis of the landmark 2017 paper "Attention Is All You Need" (arXiv: 1706.03762), which introduced the Transformer architecture. The author explains why recurrence was the bottleneck: RNNs process tokens sequentially, slowing training and degrading long-range information, while CNNs need stacked layers to capture distant dependencies. The Transformer solves this with self-attention, letting every token attend to all others in parallel. Three key designs are highlighted: multi-head attention (8 heads learning different relation types), sinusoidal positional encoding that gives each position a frequency-based fingerprint, and residual connections with LayerNorm for stable deep training. Reported results: 28.4 BLEU on WMT 2014 English-German (2 BLEU above prior best including ensembles), 41.8 BLEU on English-French as single-model SOTA, trained on 8 GPUs in 3.5 days, with 65M (base) and 213M (large) parameters. The post argues the paper defined the LLM era—GPT, BERT, T5, LLaMA, and DeepSeek are all Transformer-based—proving that attention plus feedforward layers suffice without recurrence or convolution.

Transformer: Attention Is All You Need (2017, Vaswani et al.)

arXiv: 1706.03762v7

The core problem

In machine translation, an RNN works like a word-by-word interpreter—each token must wait for the previous one. This sequential dependency makes training extremely slow, and information from early tokens becomes distorted by the time it reaches long-range positions. CNNs can parallelize better, but capturing distant relationships requires stacking many layers. Is there a way for all tokens to "see each other at once"—parallel, fast, and without losing long-distance information?

The method

The paper's answer is the attention mechanism: instead of passing information like a relay baton layer by layer, every token directly "talks to" all other tokens. Like a meeting where everyone speaks simultaneously, a softmax-based importance scoring system lets each token focus on the most relevant others.

Three key architectural designs:

1. Multi-Head Attention (MHA): Each token's representation is split into 8 heads, each independently learning different relation types—syntax, semantics, coreference, and so on—like 8 experts analyzing the same text in parallel. 2. Positional Encoding (PE): Since attention itself is position-agnostic, the paper uses sinusoidal/cosine encodings—different-frequency waves give each position a "fingerprint" added to the token embedding. High frequencies distinguish nearby positions; low frequencies distinguish distant ones. 3. Residual connections + LayerNorm: Information can take "shortcuts" (residuals) or go through "deep processing" (attention/FFN), with LayerNorm preventing numerical instability.

Key numbers

WMT 2014 English→German: 28.4 BLEU, beating the prior best (including ensembles) by 2 BLEU
WMT 2014 English→French: 41.8 BLEU, single-model SOTA
Training cost: 8 GPUs for 3.5 days—a fraction of the previous best models
Parameters: 65M (base), 213M (large)—tiny today, huge at the time

Impact

This paper defined the entire LLM era. GPT, BERT, T5, LLaMA, DeepSeek—the skeleton of every modern large model is the Transformer. It proved that "attention is all you need": no recurrence, no convolution—pure attention plus feedforward layers can do it all.

Feynman-style commentary

> The paper's true value is not inventing attention—it existed in earlier work. Its real contribution is proving one thing: when you design a general enough "mutual attention" mechanism, add "position labels" and an "information highway" (residuals), recurrence and convolution become redundant. It makes you rethink: is the core of deep learning "step-by-step processing" or "global association"? The answer is the latter.

---

arXiv: 1706.03762v7