Transformer: Attention Is All You Need (2017, Vaswani et al.)
arXiv: 1706.03762v7
Core Problem
When doing machine translation, RNNs work like a word-by-word interpreter: each token must wait for the previous one to be processed. This sequential dependency makes training painfully slow, and information from the beginning of a long sentence is distorted by the time it reaches the end. CNNs are a bit better because they can be parallelized, but capturing relationships between distant tokens requires stacking many layers. The question: is there a way to let every token "see" every other token at once, in parallel, without losing long-range information?
Method Innovations
The paper's answer is the attention mechanism. Instead of passing information like a relay baton, every token directly "talks" to every other token, much like a meeting where everyone speaks at once but a softmax-based "importance scoring" makes each participant attend only to the most relevant others.
Three key architectural designs:
1. Multi-Head Attention (MHA): Each token's representation is split into 8 independent heads, each learning a different type of relationship, e.g., syntactic, semantic, or coreference relations, like 8 specialists analyzing the same passage in parallel. 2. Positional Encoding (PE): Because attention itself has no notion of order, the paper invents sinusoidal positional encodings: waves of different frequencies act as a "fingerprint" for each position and are added to the token embedding. High-frequency waves distinguish nearby positions; low-frequency waves distinguish far-apart ones. 3. Residual Connections + LayerNorm: Information can take a "shortcut" (residual) or be "deeply processed" (attention/FFN), and LayerNorm prevents numerical explosion.
Key Numbers
- WMT 2014 English-to-German: 28.4 BLEU, beating the previous state of the art (including ensembles) by more than 2 BLEU.
- WMT 2014 English-to-French: 41.8 BLEU, a new single-model SOTA.
- Training cost: 8 GPUs, 3.5 days, a fraction of the cost of previous best models.
- Parameters: 65M (base) and 213M (large), tiny by today's standards but already massive at the time.
Impact Assessment
This paper defines the entire LLM era. The skeletons of all modern large models (GPT, BERT, T5, LLaMA, DeepSeek) are Transformers. It proved that "attention is enough": no recurrence, no convolution, just attention plus feed-forward layers.
Feynman-style Commentary
> The paper's true value is not inventing attention (that existed earlier). Its real value is proving a single point: when you design a sufficiently general "see-each-other" mechanism, plus positional labels and information highways (residuals), recurrence and convolution are both redundant. It makes you rethink: is the essence of deep learning "step-by-step processing" or "global association"? The answer is the latter.
---
*Source: zhichai.net deep-paper-research thread*