What Do Large Language Models Actually Do? The Math Behind Next-Token Prediction

Forum topic · 小凯 · 2026-06-25

Summary

This zhichai.net forum post explains the mathematical essence of large language models (LLMs): next-token prediction as conditional probability modeling P(x_t | x_<t), trained via cross-entropy loss. The author argues this is not parrot-like memorization, since models generate novel code, proofs, and writing — evidence they learn generative rules rather than specific sentences. The post breaks down the Transformer architecture: attention as a context retrieval engine (softmax(QK^T/√d)V) and FFN layers as an implicit key-value knowledge store holding roughly two-thirds of parameters. It reviews scaling laws (Loss ∝ C^-α) and the Chinchilla-optimal parameter-to-data ratio of roughly 1:20, then examines emergence — abilities like arithmetic jumping from near-zero to high accuracy at scale — with three hypotheses: metric illusion, capability-composition thresholds, and phase transitions in representation space. It closes with Ilya Sutskever's insight that compression is intelligence, a step-by-step 10ms walkthrough of a forward pass, and a formula reference table.

Key points

Core task: An LLM is a conditional probability estimator, $P(x_t | x_1, ..., x_{t-1})$ , trained with cross-entropy loss $L = -\sum \log P(x_t | x_{<t})$ to predict the next token.
Not memorization: Models generate code, math proofs, and creative text never seen in training data, implying they learn generative rules — grammar, semantics, logic, world knowledge — rather than specific sentences.
Compression = intelligence: Following Ilya Sutskever's insight, good next-token prediction forces the model to learn the causal structure of data. To predict a chess player's next move perfectly, you must learn to play chess.

Transformer architecture: retrieval + storage

Pipeline: input tokens → Embedding → [Attention → FFN] × N layers → output probabilities.

Attention as search engine: $Attention(Q, K, V) = softmax(QK^T / \sqrt{d})V$ . Query = what the current token needs; Key = what each token offers; Value = the content. Attention dynamically retrieves relevant context (e.g., looking back to "cat" and "hungry" when filling "The cat, which was very hungry, sat on the ___").
FFN as implicit knowledge base: $FFN(x) = \sigma(xW_1 + b_1)W_2 + b_2$ . About 2/3 of parameters live in FFN layers; research (Meng et al. 2022; Dai et al. 2021) views them as key-value stores where $$W_1$$ rows act as knowledge keys and $$W_2$$ columns as knowledge values.
Implication: most of an LLM's "knowledge" is static, stored in parameters, while attention performs dynamic lookup.

Scaling laws

OpenAI (2020): $Loss \propto C^{-\alpha}$ (α ≈ 0.05–0.07), with similar power laws for parameters N and data D.
Chinchilla (DeepMind, 2022): for a fixed compute budget, parameters and data should scale together, roughly $N_{optimal} \approx D_{optimal}/20$ — e.g., ~50B parameters for 1T training tokens. GPT-3-style models were over-parameterized and undertrained.

Emergence

Some abilities jump discontinuously with scale (Google 2022: two-digit addition accuracy ~10% at 3B, ~15% at 10B, ~80% at 62B, ~95% at 540B parameters). Three hypotheses:

1. Metric illusion: all-or-nothing benchmarks make smooth underlying progress look abrupt. 2. Capability composition threshold: multi-step reasoning requires simultaneously encoding several sub-skills (numbers, operators, rules, inference chains), achievable only past a size threshold. 3. Phase transition in representation space: representations shift from fragmented to highly structured, enabling vector-arithmetic-style concept relations and compositional generalization.

Emergent abilities typically require multi-step reasoning, combining knowledge, or abstract rules; basic abilities grow smoothly.

Anatomy of one forward pass (~10 ms on an A100-class GPT-3 setup)

1. Tokenization (~0.1 ms) → token IDs 2. Embedding + positional encoding (~0.2 ms) 3. Attention across 96 layers (~4 ms; GEMMs are ~60% of FLOPs) 4. FFN across 96 layers (~3 ms; intermediate dim ≈ 4× embedding dim) 5. LayerNorm + residual connections (~0.5 ms) 6. Output projection to ~100k-vocab logits (~1 ms) 7. Softmax → top token ("Paris") (~0.2 ms)

Key figures cited: 175B parameters, ~350 GB FP16 memory, ~350 TFLOPs per token, ~0.5–1 kWh per 1000 tokens.

Takeaways

Prediction forces compressed world models: good guessing requires understanding.
Emergence may be a qualitative phase change, not gradual improvement.
Attention (retrieval) and FFN (storage) form a clean separation of concerns.
Scale is the most controllable lever, but data quality and training method matter (Chinchilla).
Intelligence may be *optimized into existence* rather than explicitly designed — reasoning and knowledge modules emerge from a single prediction objective.

Formula quick reference

| Concept | Formula | Meaning | |---|---|---| | Conditional probability | $P(x_t \| x_{<t})$ | Next-token probability | | Cross-entropy loss | $L = -\sum \log P(x_t \| x_{<t})$ | Training objective | | Attention | $softmax(QK^T / \sqrt{d})V$ | Context-weighted aggregation | | FFN | $\sigma(xW_1 + b_1)W_2 + b_2$ | Feed-forward knowledge storage | | Scaling law | $Loss \propto C^{-\alpha}$ | Power-law loss vs. compute | | Chinchilla | $N \approx D / 20$ | Optimal parameter-data ratio |