What Do Large Language Models Actually Do? The Math of Next-Token Prediction, from a Guessing Game to Emergent Understanding

Forum topic · 小凯 · 2026-06-25

Summary

This forum post from zhichai.net explains the mathematical essence of large language models: next-token prediction as conditional probability modeling P(x_t | x_<t), optimized via cross-entropy loss. It argues that prediction is not parroting—models must internalize grammar, semantics, world knowledge, and multi-step reasoning to minimize loss, echoing Ilya Sutskever's thesis that compression is intelligence. The Transformer architecture is framed as Attention serving as a dynamic context-retrieval engine (softmax(QK^T/√d)V) while FFNs (~2/3 of parameters) act as an implicit key-value knowledge store. Scaling laws show loss decreases as a power law with compute, parameters, and data; the Chinchilla result (N ≈ D/20) favors balanced scaling. The post reviews emergent abilities—abrupt capability jumps at scale for arithmetic, code, and reasoning—alongside three hypotheses (metric illusion, capability-combination thresholds, phase transitions in representation space). It closes with a 10ms walkthrough of a single forward pass and five takeaways, including that intelligence may be optimized into existence rather than explicitly designed.

Key points

Core objective: LLM training is literally "guess the next token" — estimating the conditional probability P(x_t | x_1, ..., x_{t-1}) with a cross-entropy loss L = -Σ log P(x_t | x_<t).
Not parroting: Models generate code, proofs, and creative text never seen in training, implying they learn the *rules* that generate text (syntax, semantics, logic, world knowledge) rather than memorizing sentences. To predict a chess player's next move perfectly, you must learn chess — prediction forces understanding.
Compression = intelligence (Ilya Sutskever): Good prediction is equivalent to learning the causal structure of data; the shortest description of data requires understanding its deep structure, which in turn enables generalization.

Transformer: retrieval plus storage

Attention softmax(QK^T / √d) V works like a search engine: Query = what the current token needs, Key = what other tokens offer, Value = the actual content. It dynamically retrieves relevant context (e.g., attending back to "cat" and "hungry" in a long sentence).
FFN σ(xW₁ + b₁)W₂ + b₂ holds ~2/3 of parameters and behaves like a huge key-value store (Meng et al. 2022; Dai et al. 2021): W₁ rows are knowledge keys, W₂ columns are knowledge values. Most of an LLM's knowledge is static, stored in parameters; Attention does the dynamic lookup.

Scaling laws and Chinchilla

OpenAI (2020): Loss ∝ C^(−α) — loss falls as a power law with compute; similar relations hold for parameters N and data D.
DeepMind Chinchilla (2022): given fixed compute, scale parameters and data proportionally, roughly N ≈ D/20 (e.g., ~50B parameters for 1T tokens). Many earlier models like GPT-3 were over-parameterized and under-trained.

Emergence: why abilities jump instead of growing smoothly

Observed example (Google 2022): two-digit addition accuracy ~10% at 3B parameters, ~15% at 10B, ~80% at 62B, ~95% at 540B. Three hypotheses are discussed:

1. Metric illusion: all-or-nothing benchmarks make smooth underlying progress look discontinuous. 2. Capability-combination threshold: complex skills (multi-step math) require many sub-skills encoded simultaneously, possible only past a size threshold. 3. Phase transition in representation space: large models develop highly structured internal representations enabling vector arithmetic like "king − man + woman ≈ queen" and compositional generalization.

Emergence tends to affect multi-step reasoning, composition, and abstract-rule tasks; basic language modeling improves smoothly.

Anatomy of one forward pass (~10 ms, GPT-3-scale on an A100)

1. Tokenization (~0.1 ms) → token IDs 2. Embedding + positional encoding (~0.2 ms) → e.g., 5 × 12288 matrix 3. Attention across 96 layers (~4 ms total; GEMMs ≈ 60% of FLOPs) 4. FFN across 96 layers (~3 ms; intermediate dim ≈ 4× embedding dim) 5. LayerNorm + residuals (~0.5 ms) 6. Output projection to vocab logits (~1 ms), softmax (~0.2 ms) → "Paris"

Key figures: 175B parameters, ~350 GB in FP16, ~350 TFLOPs per token, roughly 0.5–1 kWh per 1000 tokens.

Takeaways

Prediction forces a compressed model of the world; understanding is a byproduct of optimization.
Emergence resembles a phase transition in representation structure, not gradual improvement.
The Attention/FFN split separates dynamic retrieval from static knowledge storage.
Scale is the most controllable lever, but data quality matters (Chinchilla).
Reasoning and knowledge modules are not explicitly designed — they emerge from a single objective, prompting questions about whether human intelligence has a similar origin.

Formula quick reference

| Concept | Formula | Meaning | |---|---|---| | Conditional probability | P(x_t \| x_<t) | next-token probability | | Cross-entropy loss | L = −Σ log P(x_t \| x_<t) | training objective | | Attention | softmax(QK^T / √d) V | context-weighted aggregation | | FFN | σ(xW₁ + b₁)W₂ + b₂ | feed-forward knowledge storage | | Scaling law | Loss ∝ C^(−α) | power-law loss vs. compute | | Chinchilla | N ≈ D/20 | optimal params-to-data ratio |