How Transformers Resolve Pronouns: A Core-by-Core Breakdown of Attention, Q/K/V, Softmax, Multi-Head, and Causal Mask

Forum topic · 小凯 · 2026-05-31

Summary

This explainer uses the sentence "The boss told the employee that he must work overtime" to demystify the Transformer attention mechanism. Coreference resolution—figuring out that "he" refers to the "employee"—requires dynamically comparing every token to every other token, which earlier RNN/LSTM and rule-based systems handled poorly past 10–15 tokens. The attention formula Attention(Q, K, V) = softmax(QK^T / √d_k) V is unpacked step by step: Q projects what a token is looking for, K projects identity, and V carries the semantic content that gets weighted and aggregated. The √d_k divisor prevents softmax saturation; multi-head attention runs h parallel projections so different heads learn syntactic, semantic, sentiment, and logical patterns; GPT-3 has 96 layers × 96 heads = 9,216 heads. Causal masking sets the upper-triangular attention scores to −∞ so autoregressive models cannot peek at future tokens, with a PyTorch snippet from nanoGPT. The piece closes by stressing that attention only chooses where to look; word embeddings, positional encoding, FFNs, residual connections, layer norm, and training data supply the actual knowledge and representations.

Overview

Why does a Transformer know that "he" in "The boss told the employee that he must work overtime" refers to the employee, not the boss? This post unpacks the coreference-resolution intuition layer by layer, using it as a springboard to explain the attention mechanism in detail.

Key points

Coreference is hard for machines. Rule-based methods (Hobbs, 1978) use nearest-noun heuristics and break easily; ML methods need heavy feature engineering; RNN/LSTMs lose dependencies beyond roughly 10–15 tokens. Pre-2017 accuracy on complex coreference stalled around 70–80%.
Attention drops the sequential assumption. Instead of carrying hidden state forward, every token directly "sees" every other token by computing pairwise scores. *Attention Is All You Need* (Vaswani et al., 2017, arXiv:1706.03762) reframes language understanding as a selection problem rather than a memory problem.
Q/K/V as "look for / be / carry." Each token is projected through three learned matrices (W_Q, W_K, W_V) into a Query ("what I want"), Key ("what I am"), and Value ("what I contribute"). Attention(Q, K, V) = softmax(QK^T / √d_k) V.
QK^T computes matching scores ("lane widths") between tokens.
Softmax turns raw scores into a probability budget that sums to 1.
The weighted sum of V vectors is the actual output—attention is not just "looking" but "moving" semantic content into the current token's representation.
Why divide by √d_k. When d_k is large (64, 128), dot products grow in variance, pushing softmax into near-one-hot outputs ("exclusive" attention) and starving gradients. Dividing by √d_k normalizes the distribution.
Multi-head attention captures parallel patterns. h independent Q/K/V projections let different heads specialize—e.g., syntactic coreference, semantic associations, sentiment polarity, logical connectors. Anthropic's *A Mathematical Framework for Transformer Circuits* (2021) shows these heads are interpretable. GPT-3 has 96 layers × 96 heads = 9,216 heads; deeper layers (49–96) handle abstract reasoning, document structure, and author intent, not just local coreference.
Causal mask enforces left-to-right generation. Autoregressive models must not see future tokens during training, so the upper-triangular portion of the attention matrix is set to −∞ (which softmax maps to 0). nanoGPT implements this with torch.triu(..., diagonal=1) and masked_fill(mask, float('-inf')).
Attention is not the whole Transformer. A full block also requires token embeddings, positional encoding, the feed-forward network (FFN) for per-token nonlinear transformation, residual connections for gradient flow, and layer normalization for training stability. The post stresses: attention chooses *where* to look; FFNs do the actual understanding; training data determines *what* gets understood. Without the embedding tables and learned W matrices, attention alone is meaningless.
A useful mental model. Attention answers the question: "In this sentence, who should I pay attention to?" The answer is the attention-weight vector.

References

Vaswani et al. (2017). *Attention Is All You Need.* arXiv:1706.03762
Devlin et al. (2019). *BERT.* arXiv:1810.04805
Brown et al. (2020). *Language Models are Few-Shot Learners* (GPT-3). arXiv:2005.14165
Elhage et al. (2021). *A Mathematical Framework for Transformer Circuits.* https://transformer-circuits.pub/2021/framework/
Karpathy (2022). *nanoGPT.* https://github.com/karpathy/nanoGPT
Hobbs (1978). *Resolving Pronoun References.* Lingua.
Stanford CS224n materials on coreference resolution.

How Transformers Resolve Pronouns: A Core-by-Core Breakdown of Attention, Q/K/V, Softmax, Multi-Head, and Causal Mask

Summary

Overview

Key points

References

Tags