This is lesson 09 of the *ai-engineering-from-scratch* open course (58k stars on GitHub, authored by Rohit Ghumare). The lesson is a Learn-type module, ~60 minutes, with lesson 06 (probability) as a prerequisite. Its goal: take apart CrossEntropyLoss() — the number you minimize every day — and show it has units.
Surprise, entropy, and cross-entropy
Information theory puts a precise price tag on "surprise": a 50% event is worth 1 bit, a 1/6 event 2.58 bits, a 0.001 event 9.97 bits, a certainty 0 bits. The formula is one line — the negative log. For a coin biased 99% heads: heads is worth 0.0145 bits, tails 6.64 bits — a 400x gap on the same coin.
Entropy is average surprise over a distribution: a fair coin is worth 1 bit per flip; the biased coin only 0.08 bits on average. Engineering reading: entropy is irreducible uncertainty you cannot squeeze out.
Cross-entropy: the world plays by true distribution P, you pay per your model's quote Q; the average extra surprise is cross-entropy. For classification, P is one-hot, so the formula collapses to the negative log of the predicted probability of the true class. Course Step 3 numbers: logits [2.0, 1.0, 0.1] → softmax [0.659, 0.242, 0.099]; true class is 0, so loss = 0.417 nats. A nat is just a bit with a different base: 1 nat = 1.4427 bits, and PyTorch stores everything in nats.
KL divergence
Cross-entropy minus entropy equals KL divergence — the overpayment. Since the true distribution is fixed during training, minimizing cross-entropy and minimizing KL are the same thing: you push the model's distribution toward the true one.
Course Step 2: a good model [0.6, 0.25, 0.15] vs true [0.7, 0.2, 0.1] overpays only 0.033 bits; a bad model [0.1, 0.1, 0.8] overpays 1.87 bits. KL is also asymmetric — the author measured the same pair: 1.8651 bits one way, 2.0193 bits the other. It is not a distance; each direction wastes differently.
Mutual information
Mutual information is how much uncertainty about one variable is removed by knowing another. Independent variables give zero; the course's example matrix [[0.45, 0.05], [0.05, 0.45]] yields 0.531 bits. This is what feature selection should look at, and it is stronger than Pearson correlation: it captures any statistical dependency, and it is symmetric and non-negative — two properties KL lacks.
Label smoothing and why cross-entropy wins
Label smoothing replaces hard labels [0, 0, 1, 0] with [0.025, 0.025, 0.925, 0.025]. In information-theoretic terms it raises the target's entropy from zero to positive. It helps because perfectly matching one-hot targets requires infinite logits; a soft target frees the model from pushing to infinity.
Three views on why cross-entropy is the standard classification loss:
- Information theory: it measures how many extra bits your model wastes as an encoder versus the true distribution.
- Maximum likelihood: negative log-likelihood is literally cross-entropy; minimizing it maximizes data likelihood.
- Gradients: with softmax, the gradient is prediction minus target — one clean line.
Verification audit
The clean part
Every number was re-run and matched exactly: biased-coin entropy 0.0808 bits, fair die 2.585, good model cross-entropy 1.1896, bad model 3.0219, KL 0.0328 and 1.8651, mutual information 0 and 0.5310. The label-smoothing loss identity matched to six decimal places. Sixty minutes of material with zero formula-level fluff.Four flaws found
1. A missing word in "you can't compress below entropy". The word *lossless* is dropped — lossy compression (JPEG) runs below entropy all the time, paying with distortion. Also, the lossless bound is asymptotic: a 0.08-bit entropy coin still costs at least 1 bit per flip with any single-symbol code (a 12x gap), unless you block-code a thousand flips together. Huffman single-symbol only guarantees ≤ entropy + 1.
2. Silent interface bugs. cross_entropy aligns inputs with zip, which silently truncates: a 3-element true distribution with a 2-element model distribution quietly returned 0.9159 bits. The entropy function doesn't check normalization — [0.5, 0.5, 0.5] (sum 1.5) returned 1.5 bits, breaking the just-learned "max entropy of a binary variable is 1 bit" rule. information_content(1.5) returned 0.0, silently treating an invalid probability as certainty. A function measuring uncertainty that is itself uncertain about input validity — a common teaching-code flaw, but worth naming here.
3. Step 4's identity demo has no auditing power. "Cross-entropy equals negative log-likelihood" is shown by running the same formula twice — same softmax, same negative log, difference exactly 0 (not even float noise). A real verification needs cross-implementation: hand-written NLL vs torch.nn.CrossEntropyLoss.
4. An unsourced figure: "GPT-2 perplexity ≈ 30". The paper's numbers are WikiText-103 zero-shot: 37.50, 23.33, 17.48, 16.34 across the four sizes. "About 30" names neither size nor dataset. The direction is right — modern models reach single digits — but this 30 has no provenance.
Slogan audit
The course's framing of training as a communication problem holds up, and it later lays out all three hats (communication, likelihood, gradients). The sneakier line is the colloquial gloss on mutual information: "observing something never loses information." I(X;Y) ≥ 0 is true, but the data processing inequality says something else: observation doesn't lose, *processing* does — for X → Y → Z, I(X;Z) ≤ I(X;Y). The classic feature-engineering illusion that "more features can't hurt" is fed by this phrasing: junk features contribute zero plus compute cost, and passing through a lossy model can dilute useful signal.
Exercise 2, worked
Logits [5.0, 2.0, 0.5], true class 1. Softmax: [0.9426, 0.0469, 0.0105] — the model puts 94% confidence on class 0 and only 4.7% on the true class. Loss = 3.06 nats; perplexity = e^3.06 = 21.3 — on a three-option question, the model is as confused as choosing among twenty-one. What logits give zero loss? No finite numbers do: the true class's logit gap must tend to infinity. Zero loss is an asymptote, not a destination.