This is lesson 07 of the "AI Engineering Foundations" series, working through the open-source course ai-engineering-from-scratch (58k stars on GitHub, written by Rohit Ghumare), one lesson at a time. Today's topic: Bayes' theorem — a Build-type lesson, 75 minutes. The previous six lessons covered matrices, gradients, and probability; this one answers an older question: how do you update your beliefs after seeing evidence?
The screening-test surprise
You take a medical test and the report shows a positive mark. The doctor says the test is 99% accurate. Most people conclude they're almost certainly sick. The actual math is startling: if the disease affects 1 in 10,000 people, the probability you're actually sick after a positive result is under 1%. The other 99% of positives are false positives from healthy people. It's not that the test is bad — the disease is just so rare that the pool of false positives is a hundred times larger than the pool of patients.
The formula
Bayes' theorem follows in two steps from the definition of conditional probability. Four quantities, one equation:
- Prior: how you bet before seeing evidence
- Likelihood: how common the evidence is, assuming the hypothesis is true
- Evidence: the total probability of seeing this signal across all possible worlds
- Posterior: your updated bet
- Zero probability for unseen words kills the whole product. Fix: Laplace smoothing — add 1 to every word count so nothing is zero.
- Underflow from multiplying many small numbers (multiplying 0.01 fifty times gives 10⁻¹⁰⁰). Fix: work in log space, turning products into sums.
Plugging in the medical example: prior 0.0001, likelihood 0.99, denominator including false positives from healthy people — posterior comes out to 0.98%. The same formula works elsewhere: an email containing the word "lottery" pushes spam probability from 30% to 95.5%.
Naive Bayes in code
The lesson implements a naive Bayes spam classifier: a training set of 12 sentences (5 spam, 7 normal), learning per-word probabilities for each class; a new message is classified by multiplying word probabilities and picking the higher-scoring class. "Naive" refers to the word-independence assumption — clearly wrong ("New" and "York" always co-occur), yet surprisingly effective in practice, because classification only needs ranking, not calibrated probabilities.
Two classic pitfalls are covered:
MAP and conjugate priors
Two more substantial pieces:
1. MAP (Maximum A Posteriori) estimation: MLE only pleases the data; MAP additionally places a prior belief on parameters (e.g., "parameters should be small"). A Gaussian prior is exactly L2 regularization; a Laplace prior is L1 — the regularization terms you add to your loss every day are priors in Bayesian dialect. 2. Conjugate priors: a Beta prior with Bernoulli data yields a Beta posterior, with update rules as simple as addition — prior Beta(a, b), observe s successes and f failures, posterior Beta(a+s, b+f). Today's posterior is tomorrow's prior — the prototype of online learning. A coin shows 7 heads 3 tails on day one (mean 0.667); day two adds 5 heads 5 tails, giving posterior Beta(13, 9) and pulling the mean back to 0.591. As evidence accumulates, beliefs move on their own.
Bayesian A/B testing lives in the same language: give each variant a Beta posterior, draw 100,000 Monte Carlo samples from each, and count how often B beats A — that fraction is the probability that "B is better."
Fact-check results
The numbers check out: medical example 0.98%, lottery example 95.5%, Beta sequence means 0.667 and 0.591, A/B posterior means 0.051 and 0.066 — hand calculations match the computed results. The code runs: predictions for four test messages match sklearn's MultinomialNB line by line.
But three issues surfaced:
1. One number doesn't match its own data. At the end of the A/B section, the course claims Bayes gives you a direct statement: "the probability B is better is 97%." The actual computation gives 92.5% (Monte Carlo fraction 0.9252 over 100k samples). This matters because the course's own decision rule requires P(B>A) > 0.95 to ship variant B. At 92.5%, the data fails the bar — by the course's own rule, the verdict is "keep collecting," not "ship B." The 97% is borrowed sample phrasing, not the result for this dataset.
2. "The same algorithm" needs an asterisk. The from-scratch version tokenizes with split(), producing a 44-word vocabulary; CountVectorizer's default regex only keeps tokens of two or more characters, giving 43 words — the word "a" is dropped. Test sentences like "you won a free prize" therefore receive different evidence sets in each version. Classification results happen to agree because "a" is nearly neutral evidence. But the tokenizer isn't part of the algorithm — it's part of the interface, and interfaces hide decisions: today the dropped token is a harmless article, tomorrow it may not be.
3. How things actually fail. The text says that with smoothing=0, a single unseen word "kills the entire product." But the course's own code runs in log space, so the failure mode differs: math.log(0) raises a ValueError on the spot, pointing directly at that word. In multiplication the failure is a silent zero; in log space it's a loud crash. The failure mode itself is honest — same temper as the Value class's exponentiation from the previous lesson: errors happen in plain sight.
A wording habit worth flagging
The benefits list for Bayesian A/B testing includes the slogan: "check results anytime without inflating the false positive rate — no peeking problem." The condition sits in a neighboring table in the same section: "given a well-chosen prior and a correctly specified model." The condition is real — with a skewed prior or a wrong model, peeking still bites — but the slogan omits it. Same pattern elsewhere: "just update two numbers" is only true for conjugate priors; general posteriors require MCMC, which the course itself concedes three paragraphs later. Slogan first, condition later — reading this course means getting used to supplying the qualifiers yourself.
Exercises worth doing now
Exercise 1: after two independent positive results, what's the probability you're sick? Not 0.98% squared, and not 99%. Use the first result's 0.98% as the new prior and run the formula again: the answer is 49.5%. The first test pulls you from one-in-ten-thousand to one-in-a-hundred; the second takes you to a coin flip.
Exercise 4: 10 coin flips, 7 heads. With a Beta(2, 2) prior, the MAP estimate is 0.667, while MLE gives 0.7. The prior pulls your estimate three centimeters toward 0.5. Stop on those two numbers.