This is part 6 of the "AI Engineering Foundations" series, working through the 58k-star GitHub course ai-engineering-from-scratch by Rohit Ghumare. Lesson 06: Probability and Distributions (Learn type, 75 minutes). The first five lessons covered matrices, gradients, and the chain rule; this lesson switches to a new language — probability. Every number a model outputs, be it 0.91 or 0.03, is a word in that language.
A classifier outputs [0.03, 0.91, 0.06]. These three numbers sum to 1, each between 0 and 1. It isn't saying "I'm 91% confident" — it's saying "I'm placing a probability distribution over these three answers." A language model picking the next token from 50,000 candidates computes 50,000 probabilities and samples at every step. Diffusion models generate images by repeatedly sampling from a noise distribution. If you can't read probability, you can't understand anything a model says.
Three Axioms
The language has only three grammar rules: any event's probability is at least zero; all possibilities sum to 1 (something must happen); mutually exclusive events' probabilities add directly. Bayes' theorem, expectation, and distributions are all derived from these three. There is no fourth.
Conditional probability is the first truly useful verb. P(A|B), the probability of A given B has occurred, is the intersection divided by B. A deck has 12 face cards (J, Q, K), 4 of which are Kings. Given you drew a face card, the probability it's a King is 4/12 — one third. One more step past this division is Bayes, the next lesson's main course.
Discrete vs Continuous Distributions
Distributions come in two kinds, split by whether outcomes are countable. Dice are countable — discrete, described by a PMF (probability mass function) that assigns each outcome a probability: a fair die gives 1/6 per face, summing exactly to 1. Height is continuous, described by a PDF (probability density function): density at a single point is not a probability; you must integrate over an interval. Density can exceed 1 — a uniform distribution on [0, 0.5] has density 2, because the height is only 0.5 so the width must double. Remember: density is the *concentration* of probability, not probability itself.
Each common distribution has its temperament:
- Bernoulli: one trial, two outcomes; mean p, variance p(1−p); the native language of binary classification.
- Categorical: its multi-way version; softmax output is exactly this.
- Poisson: counts rare events; mean and variance both λ.
- Normal: the protagonist. Bell-shaped, determined by mean and variance alone; 68% of data within 1 standard deviation, 95% within 2, 99.7% within 3.
- A die, the flattest distribution, has expectation 3.5, variance 35/12, standard deviation 1.71.
The Bell Curve Is Added, Not Designed
The most counterintuitive fact of the lesson: the bell curve isn't designed by anyone — it emerges from addition. Roll 1 die: flat distribution. Average 2 dice: triangular. Average 30 dice: a near-perfect bell. The course says "this holds for any starting distribution" — the author's fact-check section has something to say about that claim (see below). Why do weight initializations use normal distributions? Why does SGD's gradient noise approximate a normal? Why is measurement error normal? Because they're all sums of many independent small things. From the same vine: given a mean and variance, the normal is the maximum-entropy distribution — knowing nothing else, it's the only choice that plays no favorites.
Engineering Pitfall: Underflow and Log-Probabilities
A 50-word sentence where each word has probability 0.01 gives a sentence probability of 0.01 multiplied 50 times: 10⁻¹⁰⁰. Longer still, the product underflows to 0 — floating point can't hold numbers that small. The fix: move multiplication into log space. Log turns multiplication into addition: fifty −4.61s sum to −230.26, a perfectly stable finite number. Log-probabilities are always ≤ 0; the more negative, the less likely. The training loss you watch every day is exactly this.
A model's raw output isn't probabilities at all — it's logits, a vector of unnormalized scores. Softmax translates: exponentiate each score, then divide by the total. The exponential does two jobs: pushes scores positive and amplifies gaps. z=[2, 1, 0.1] becomes [0.66, 0.24, 0.10]. A survival trick: subtract the max before exponentiating. z=[100, 101, 102] shifts to [−2, −1, 0] — mathematically identical, but avoids large-number overflow. PyTorch's cross-entropy loss internally does exactly this log-softmax in one package.
Cross-Entropy: The Terminal Station
Under one-hot labels, cross-entropy has a single meaning: the negative log probability of the correct class. If the model gives the right answer probability 0.66, loss is −log(0.66) ≈ 0.42; push the probability to 1, loss goes to zero. All of training is about pushing this one number down.
Fact-Check (the "Customs" Audit)
The lesson's numbers are unusually clean: P(King|Face)=1/3, the three log values −4.6/−5.8/−3.9, marginal sums in the joint distribution table, dice 3.5/2.9167/1.7078, the measured CLT standard deviations — all match with zero error.
Two soft spots, same root: floating-point precision goes unstated. "Underflows after about 30 terms" — in float64, 0.01³⁰ is actually 10⁻⁶⁰, far from underflow; you'd need roughly 150 terms to reach zero. In float32, it does hit zero at 23 terms. "exp(102) overflows" is similar: in float64 it's only 2×10⁴⁴ and doesn't overflow until exp(709); in float32 it is indeed inf. Deep learning defaults to float32, so both examples hold in real training contexts — but a reader reproducing them in Python's default float64 will find the textbook overflow never happens. The textbook isn't wrong; it just never said which floating-point world it lives in.
One slogan is too broad: "this holds for any starting distribution." The full Central Limit Theorem requires independent, identically distributed variables with finite variance. The Cauchy distribution is the standard counterexample — its tails are so heavy it has no mean. Tested: 2,000 groups of 30 Cauchy samples averaged, standard deviation 53.8, still heavy-tailed — 30 sums and no bell appears. The course swallowed the finite-variance precondition when teaching CLT; students waiting for a bell with heavy-tailed data will wait forever. One honesty credit: the visualization code in step 7 is pseudocode with ellipsis placeholders; the docs state the full implementation lives in code/probability.py.
Exercise 3, Worth Doing Now
A 5-class classifier outputs logits [2.0, 0.5, −1.0, 3.0, 0.1]; the correct class is index 3. Following this lesson's approach, softmax gives [0.24, 0.05, 0.01, 0.66, 0.04]; the correct class probability is ~0.66 and cross-entropy is −log(0.66) ≈ 0.42. Verify against PyTorch's nn.CrossEntropyLoss — the two should agree to within 10⁻⁸. Once you've computed that number, you're speaking this language.