95.67%.
That is the accuracy of a model in this course's demo. Three hundred samples: 287 negative, 13 positive. The model does exactly one thing — always predict negative. 287 correct, 13 wrong, 287 out of 300 = 95.67%. It learned nothing and knows nothing, yet its accuracy looks better than many earnestly trained models.
That is the problem this lecture on model evaluation tackles. Scores themselves don't lie. What lies is how you read them.
Three splits, three purposes
First, one thing must be clear: one dataset gets cut into three parts, each with its own job.
The training set is the textbook — the model learns from it. The validation set is the mock exam — you tune hyperparameters and pick among candidate models using it. The test set is the final exam — touched only once, at the end, for the final grade. The course puts it starkly: if you look at test performance and then go back and change the model, it is no longer a test set — it has become a second mock exam. You've used the final as practice once, and that score can never be trusted again.
The common ratio is 60/20/20.
The four cells
Every classification metric comes from one table with four cells.
Predicted positive/negative crossed with actual positive/negative gives: true positive, true negative, false negative (missed), and false positive (false alarm). That's it.
- Accuracy is the proportion correct across all four cells. Great when classes are balanced; on 287:13 data it enables the 95.67% trick above.
- Precision looks at two cells: of everything the model flagged positive, how much was real. Use it when false positives are costly — e.g., a spam filter killing a client's important email.
- Recall looks at the other two cells: of all true positives, how many were caught. Use it when false negatives are costly — e.g., missing a tumor in cancer screening.
- F1 is the harmonic mean of the two, for when you can't sacrifice either.
There is also AUC, which ignores thresholds. The metrics above require drawing a line first (probability > 0.5 = positive). AUC sweeps every possible line and measures ranking ability: the probability that a randomly chosen positive scores above a randomly chosen negative. 0.5 is a coin flip, 1.0 is perfect separation. The demo model's AUC is 0.9408 — far better than its 0.8167 accuracy, because the threshold line landed in an information-losing spot.
Round and round
With small data, a single validation split is wasteful. Enter cross-validation: cut into five folds, hold one out for validation each round, train on the other four, average over five rounds. Every point gets validated exactly once; the average is stabler than any single split.
The course compares ordinary 5-fold against stratified 5-fold, where each fold keeps the same class ratio as the full data — designed for imbalanced data.
But the comparison ran on balanced data (152:148). Results: ordinary 5-fold averaged 0.8733, stratified 0.8762 — a 0.3 percentage-point gap — while fold-to-fold variance actually rose from 0.036 to 0.061. Stratification on data that doesn't need it reveals nothing. What actually shows its value is the 287:13 dataset — I ran the comparison there myself: with random splits, the positive counts in the five validation folds were 1, 4, 3, 2, 3; after stratification, 2, 2, 2, 2, 5. With only 1 positive in a fold, evaluating that fold is a coin toss. That's the problem stratification solves — the demo just didn't show it where it matters.
Stratification has its own limits. When the minority class has fewer examples than folds — say 3 positives, 5 folds — the course's implementation dumps all 3 positives into the last fold. The first four validation folds contain zero positives: with F1 as the metric, four folds score 0.0 — not because the model is bad, but because there's nothing to count. The fifth fold is worse: its training set has zero positives, producing a model that only predicts negative. The whole cross-validation silently returns a meaningless result. sklearn would warn you loudly in the same scenario. Each fold needs at least one positive — a prerequisite of stratified sampling that never appears in the parameter docs.
A test set with three patients
Back to the imbalanced data. Thirteen positives scattered across 300 samples; after a 60/20/20 split, the test set gets 3 positives.
The trained model on those 3 positives: recall = 3/3 = 1.0, precision = 3/4 = 0.75 (caught everything, but with one extra false alarm). Respectable-looking numbers with extremely low resolution: each missed positive drops recall by 33%; each extra false alarm drops precision by 15–25 points. A lecture preaching reliable evaluation built its own evaluation on 3 samples. Stratified splitting would have given each part a stable 2–3 positives — the course's error checklist even lists "not using stratified splitting," just not where it was actually needed.
How to read curves
Learning curves: plot training and validation scores against training-set size. High training score, low validation score, big gap → overfitting, more data should help; both lines flat and low → underfitting, more data won't help, get a more complex model.
The demo's curves: training score fell from 1.0000 (24 samples) to 0.8792 (240 samples); validation hovered between 0.8167 and 0.85. The gap shrank from 0.15 to 0.03 — that direction is trustworthy; overfitting is receding. But don't take the 3–4 point wiggles in validation seriously: with only 60 validation samples, each mistake is 1.67 points, and sampling luck alone is ±4 points. The data's theoretical ceiling is about 0.892 (generated by injecting 0.5 noise into the sum of two features, which no model can beat), so every score from 0.82 to 0.87 sits inside that noise band. Discipline for reading curves: trust the gap's direction, don't fuss over the absolute validation level.
The wrong order, a tiny error
The course's number-one error is data leakage: test-set information reaching the model during training. The textbook form, in the course's own words: "fitting the standardizer on the full dataset before splitting," followed by the operating rule — split first, then preprocess.
Then the demo itself: 200 regression samples, standardization computed on the full data first, then split.
The order is reversed. I tested the correct order (split first, standardize using training statistics only): R² 0.9567 vs. the course order's 0.9560 — a gap of 0.0006. On uniformly distributed data, this much leakage barely hurts. But the same reversed order on skewed distributions, or with information-heavy preprocessing, is no small matter — a standardizer that has "seen" the test set before the split is like studying the exam's statistical summary beforehand; it's just that this exam is too symmetric to reveal the advantage. The course's own skills checklist file says "Split before any preprocessing" — of the three places (text, skills file, demo code), two are right.
A t-test with t = 0
The demo's last section is statistical comparison: a paired t-test between two models. Model A trained 100 rounds, model B 500 rounds, same data, same 5 folds.
t = 0.0000.
Not "the difference isn't significant" — there is no difference to detect at all. Both models converged to the same point on this data: weights matching to seven decimal places (3.4088268 vs 3.4088270), and per-sample predictions identical across all five folds. The t-test procedure ran in full, but what was compared were two copies of the same thing. To give the tool something to do, you need genuinely different models — say, 5 rounds vs 2000 rounds of training. Then there's a real difference to compute: t = -1.00, still far from the 2.78 significance line, and the conclusion is still "can't tell them apart" — but at least it's a real question.
One more detail: the code divides by 5 (folds) when computing the standard deviation; the textbook paired t-test divides by 4 (degrees of freedom). This inflates |t| by a factor of 1.118, biasing toward false significance. Here t was exactly zero, so it didn't show — but swap in a real comparison and the number quietly skews.
Exercises
Of the three exercises, the third is the most worth doing: the permutation test. Shuffle the labels randomly, retrain, record the score; repeat 100 times to get the score distribution of "a model that learned nothing." Then see where your model's real score falls in that distribution. It turns "is this model actually learning" from a judgment call into a countable number — connecting right back to this lecture's starting point: that 95.67% model would score about the same on every one of those 100 shuffles.
*Source: zhichai.net forum post (AI Engineering Foundations, lecture 31). All figures are as reported in the original post.*