AI Engineering Foundations 15: Is a 0.02 Gap Improvement or Noise? Statistics for ML, Audited

This is part of the "AI Engineering Foundations" series, working through the 58k-star open course AI Engineering from Scratch (MIT license) lesson by lesson. Today: Lesson 15, Statistics for Machine Learning — a Build-type lesson, 120 minutes, prerequisite being the probability and Bayes lessons.

0.87 vs 0.89 — Do You Dare Ship It?

A concrete scenario: you trained two models, A scores 0.87 on the test set, B scores 0.89. You ship B. Three weeks later, production metrics drop. Why? Because that 0.02 gap may not be an improvement at all — it may be noise. The test set is too small, variance is too high, or both. You packaged random fluctuation as a feature and shipped it.

This lesson is about doing that math before you ship. Let's go through the core tools.

Descriptive statistics. Mean, median, and mode all answer "where is the middle." For [1, 2, 3, 4, 1000], the mean is 202 while the median is 3 — the mean gets dragged by the large value while the median stays put. When they diverge, the distribution is skewed. Spread is measured by variance and IQR; percentiles expose the tails. In latency monitoring, P50 is the typical user's experience while P99 is the long tail, often several times the median. In safety-critical applications, a model with bad P99 may be worse than one with mediocre averages.

Correlation. Pearson measures linear relationships; Spearman measures monotonic ones. For y = x³, a perfectly monotonic curve, Pearson gives less than 1 (it only recognizes straight lines), while Spearman gives 1 (it ranks first). The covariance matrix is the family portrait of correlations: diagonal entries are variances, off-diagonals are pairwise covariances. The PCA from the previous lesson decomposes exactly this matrix.

Hypothesis testing. The logic: state a null hypothesis of "no difference," then ask — if there truly were no difference, how unusual is my data? That probability is the p-value. If p < 0.05, you call the difference significant. But the course repeatedly stresses a point worth recording verbatim: a p-value is NOT the probability that the null hypothesis is true. It is the probability of data this extreme, assuming the null is true. The direction is reversed — the most misread line in statistics.

T-tests compare means, chi-square tests compare frequencies — these are standard parts. The real protagonists are two new tools.

Effect size. With enough data, even trivial differences become significant. 0.9234 vs 0.9237, a 0.03% gap over a million samples, produces a lovely p-value — but is it worth switching models? Cohen's d divides the difference by the pooled standard deviation, giving a sample-size-independent measure of "how big is this": 0.2 small, 0.5 medium, 0.8 large. P-values tell you whether it's real; effect size tells you whether it matters. Report both.

Bootstrap. Its claim is bold: with no distributional assumptions, for any statistic — median, F1, the AUC difference between two models — I can build you a confidence interval. The method is brute force: resample your data with replacement, compute the statistic once; repeat a thousand times, sort the results, trim to the middle 95%. Done. Its backing is compute, not formulas.

One practical corollary of the Central Limit Theorem deserves its own mention: mini-batch gradient descent works because the average gradient over a batch converges to the true gradient. The lesson also offers a rare self-warning — CLT does not save heavy-tailed distributions; the Cauchy distribution (infinite variance) is not covered. This caveat was missing when Lesson 06 of this series was audited; this lesson patched it itself.

Passing Through Customs

As usual, all code was run and every number checked. The clean parts are solidly clean: t and chi-square p-values are computed via hand-written regularized incomplete beta functions, matching scipy to sixteen decimal places with zero error; the chi-square example in the text (120 vs 80, chi-square = 8, df = 1) gives p = 0.00468, and the text's "less than 0.005" is precise; the multiple-comparison demo produced 1/20 uncorrected false positives and 0/20 after Bonferroni, exactly matching the predicted "expected 1"; power simulations of 0.32 and 0.84 nearly match theoretical values of 0.32 and 0.85. Textbook-grade implementation.

But this lesson has a strange pattern: it teaches things more accurately than it uses them.

Finding 1: the example teaching statistical significance is itself not significant. The text says: model A accuracy 0.9234, model B 0.9237, one million test samples, p = 0.001, statistically significant. Plug these into a standard two-proportion test: z = 0.80, p = 0.425. To actually get 0.001, the gap would need to be four times larger (0.9234 vs 0.9246 territory). The numbers are salvageable: if the models were compared pairwise on the same test samples — B fixed 300 problems A got wrong, breaking none — a McNemar test would give an absurdly small p. The problem: the example never states which framing it uses. Independent reading: not significant. Paired reading: wildly significant. And by its own narrative (two models, same test set), the paired framing applies. In a section about "you only know if you compute," the example numbers were never computed. Same disease as Lesson 07's "97% chance B is better."

Finding 2: a code demo fails live, instructively. The "statistical vs practical significance" demo wants to show: small samples can't detect small effects; large samples detect them but the effect is trivial. Actual output: small-sample group (n=30, true effect 0.01 SD) p = 0.028 — significant — with Cohen's d of 0.58, a medium effect. Large-sample group (n=100k) p = 0.002, d = 0.014. The demo's first half flopped — it meant to show "can't detect" but instead showed a false positive happening live: 30 samples happened to draw a 0.58-SD random difference. This is precisely a live case of error #8 in the lesson's own ten-item list (small test sets without variance reporting) and a perfect contrast with error #4 (confusing statistical and practical significance). Change the seed, and the story changes script.

Finding 3: a recommendation collides with math. The text says: in ML experiments you usually only have 5-fold or 10-fold cross-validation, so non-parametric methods like the Wilcoxon signed-rank test are often more appropriate than t-tests. Problem: with 5 folds, the two-sided Wilcoxon p-value has a mathematical floor of 2/2⁵ = 0.0625 — greater than 0.05. That means winning all five folds — B wins every single one — still cannot reach significance. Six folds barely crosses the line (0.031). Verified with scipy: five all-positive differences, p = 0.0625, unmoved. Five-fold users wanting Wilcoxon should bump to six folds first.

Finding 4: silent degradation in one function. The demo data is 15 distinct numbers; the mode function runs on it and prints "Mode: 12." But 12 is not the most frequent value — every value appears once, so no mode exists. Internally the function treats all values as "tied for first" and, after sorting, returns the smallest. 12 is the minimum. The text itself says "the mode is rarely meaningful for continuous data," but the code has no guard: no repeats, no error, no warning — it quietly becomes a min(). Fourth sighting of this no-guard interface family in the series.

Finding 5: unstated conventions bury a trap. The Pearson formula is written as r = numerator / (n·s_x·s_y), with s the standard deviation. Which standard deviation? Two pages earlier the lesson just taught Bessel's correction: sample variance divides by n−1, not n. A student plugs in the freshly learned sample standard deviation, and r shrinks systematically — by exactly (n−1)/n. For the ten-fold cross-validation data, r comes out 10% low. The course's own code uses the population convention (divide by n), which matches standard Pearson — no bug there — but the formula carries no label, and the conventions silently switch between adjacent sections. The same happens with covariance: the text's formula divides by 1/n, the code's covariance function defaults to (n−1), the covariance-matrix function hardcodes (n−1), and Pearson internally uses 1/n — four places, three conventions. On the same five numbers, the text's formula gives variance 2.0 while the code gives 2.5 — a 25% discrepancy. Fortunately, correlation coefficients are unaffected since conventions cancel between numerator and denominator; everyone who reads covariance values directly is affected.

Finding 6: bootstrap — the text teaches one thing, the code runs another. The text's model-comparison procedure is clear: resample test-set indices, compute both models' metrics on the same resampled indices, store the difference. That's the paired approach, preserving the natural structure of "both models see the same data." The code implements something else: two independent resamplings per model. On paired data, the independent version widens the difference confidence interval by 1.89x (measured: paired width 0.016, independent width 0.031) — a pure loss of power. Teaches paired, codes independent, numbers off by nearly double.

Three small items. First, the text says training loss distributions are "left-skewed by easy samples, with mean far below median" — the direction is reversed. Loss is bounded below by 0; easy samples pile near 0 while hard samples drag the tail rightward. That's right skew, mean above median — the same shape as the income distribution the course itself gets right (right-skewed, mean pulled up by the wealthy). Right for income, reversed for loss. Second, sign inconsistencies in demo output: the observed difference prints as B−A (+5.73) while Cohen's d and the t-statistic use A−B (−0.56, −2.46); readers must reconcile directions themselves. Third, Bonferroni's footnote is attached in the wrong place: the text says it "is valid when tests are independent" — in fact Bonferroni controls the error rate under arbitrary dependence (it is a pure union bound); what actually needs independence is the earlier 1 − 0.95²⁰ = 0.64 formula, whose prerequisite the text omits.

A closing compliment: the CLT section includes a three-item "do not" list — don't expect it to make your data normal, don't expect it to save infinite-variance heavy tails, don't apply it to correlated data. The third is especially rare; the Lesson 06 audit flagged that the slogan "holds for any starting distribution" lacked exactly this qualifier. A course patching its earlier lessons in its later half is a first for this series.

Compute First, Then Speak

Compressing this lesson into one line: a point estimate is an assertion; a confidence interval is the bandwidth you attach to it. Next time you say "B is better than A," say the 0.02 together with its confidence interval. With only ten folds, run paired differences through Wilcoxon — and remember six folds is the floor. For gaps under 0.02, compute Cohen's d before calling a meeting. Want to see the non-significant example yourself? Plug 0.9234 and 0.9237 into a two-proportion test: z = 0.80 — that number will finish the sentence this lesson left unsaid.

---

*Original course: ai-engineering-from-scratch, Lesson 15: Statistics for Machine Learning (Rohit Ghumare, MIT license). Earlier installments on the author's homepage.*