AI Engineering Foundations 25: Logistic Regression - The Worse the Error, the Softer the Gradient

Still prediction, but this time the target has only two values. Is a tumor malignant or benign? Does a student pass or fail? Last lesson's linear regression fails here: fitting a straight line to 0 and 1 gives outputs like 0.3, 1.7, -0.5. What does 1.7 mean? Extremely malignant? Does -0.5 mean extremely benign? Nobody knows. Classification needs a probability between 0 and 1, plus a clear boundary: one side of the line is one class, the other side is the other class.

Logistic regression in one sentence: the linear part stays the same, compute z = wx + b, then feed z into a function called sigmoid. This function looks like a stretched letter S, designed to squash any number into the range between 0 and 1. When z is zero it outputs 0.5; as z grows large it approaches 1; as z goes very negative it approaches 0. The output is used as a probability: above 0.5, predict one class; below, the other. Despite "regression" in the name, it is a classification algorithm - the name comes from the S-shaped function it uses.

From here the lesson heads into a real problem, worth pausing on.

The Real Reason to Switch Rulers

Training needs a loss function: a ruler that scores predictions. Last lesson used mean squared error (MSE), the average of squared differences between predictions and truth. This time it cannot be reused. The textbook answer: MSE combined with sigmoid produces a non-convex loss surface with many local minima, and gradient descent gets stuck in them.

This claim is true - and harsher than the textbook admits.

Testing with the lesson's minimal example: ten students, study hours from 1 to 10, the first four fail and the last six pass. The model is just a sigmoid. Training it with MSE from 200 random starting points - 82 of them, over 40%, forever stop at the same answer: predict everyone fails. The loss is exactly 0.6 (each of the six true passes contributes a squared 1; 6 divided by 10). Keep training for 500,000 iterations: nothing moves. The model is not stupid - it stands on a plateau with exactly zero slope, and gradient descent by definition will not move.

Why does "everyone fails" specifically trap it? Because differentiating MSE through sigmoid carries a factor of the sigmoid's own derivative. That derivative peaks at only 0.25 and vanishes at both ends. Where the model is most wrong - truth is pass, model says probability near zero - is exactly where sigmoid is flattest. The worse the error, the smaller the gradient; the softer the shout. A concrete point: where the true label is 1 and the model outputs 0.000000002, cross-entropy's gradient is over 200 million times larger than MSE's. MSE is nearly mute there.

Cross-entropy's formula looks fearsome, but it is the logarithm doing the work: differentiating the negative log of sigmoid's output p, the log's derivative cancels the sigmoid's derivative exactly, leaving the clean expression (p−y)x. y is the true label, p the predicted probability. The bigger the gap, the bigger the gradient: model says 0.29 when truth is 1, the gradient pulls with force 0.71; model says 0.99, barely pulled. The course notes this looks identical to linear regression's gradient - it does - but the real point is the cancellation step. Switching the loss function is not cosmetic; it cures the "the worse the error, the quieter the voice" defect. With cross-entropy, from that same saturated position, the two-hundred-run experiment escapes within 2,000 iterations, and all 200 starting points converge into the same valley.

Convex - Then What?

Cross-entropy gives logistic regression a convex loss surface: one valley bottom, no other pits to fall into. The textbook says this guarantees a single global minimum. That holds for this dataset - but "a minimum exists" and "you have walked up to it" are two different things.

The course's training set has 160 points, including two holdouts: one truly malignant point on the wrong side of the boundary, and one truly benign point standing just past 0.5. Just these two points make the minimum exist but push it very far away - the other 158 separable points keep pushing the weights outward, the two holdouts act as brakes, and the tug-of-war ends far away but finite. After the course's prescribed 1,000 iterations the loss is 0.129; the true valley bottom is 0.039, more than three times lower. Continuing with the same step size takes about 300,000 iterations to get near the minimum.

Last lesson showed "the program finished, the training didn't"; this lesson reprises it with a different mechanism: last time uncentered features stretched the bowl, this time near-separability pushes the valley bottom far away. More subtly, this time even accuracy cannot help you. The test set has 21 malignant and 19 benign cases separated by a wide gap; by 1,000 iterations everything is classified correctly, and running 290,000 more iterations changes not a single prediction.

But the probabilities keep changing. Watch the point (3.5, 3.5): at 1,000 iterations the model says 0.54; at 3,000, 0.52; at 100,000, 0.60; at 300,000, 0.62. The decisions locked in long ago; the confidence keeps steepening. If you only care about right/wrong, early stopping is fine; if you use the probabilities themselves - risk pricing, confidence levels, whether to re-examine - then "accuracy is maxed" and "probabilities have settled" are two different things. A side discovery: the training log's first printed loss is 0.6289, not the expected initial 0.6931 - the code updates one step before printing. Anyone checking the numbers will pause here.

An even more extreme case deserves mention. If the training data is perfectly separable - no holdouts at all - the minimum simply does not exist: the weights grow without bound, the loss approaches zero but never reaches it. scikit-learn's logistic regression has slight regularization by default, guarding against exactly this.

An Empty Game on the Scoreboard

The course ends by saying your from-scratch implementation and scikit-learn give the same decision boundary. Three implementations were tested: the hand-coded version at 1,000 iterations, scikit-learn without regularization (the theoretical optimum of the same loss), and scikit-learn's defaults. The three boundaries have slopes -1.30, -1.21, and -1.16 respectively, all different; overlaid on one plot, 1.4% to 2.8% of grid cells get opposite predictions. All three score perfectly on this test set - but because the test is too easy, not because the paths converge.

This also explains why the report shows precision, recall, and F1 all at 1.0000: 21 malignant all caught, 19 benign all spared - the scoreboard has no contested match. The real arena for these metrics is the course's threshold table: push the threshold down to 0.3 and two benign cases get flagged malignant, dropping precision to 0.913; raise it to 0.7 and two malignant cases get missed, dropping recall to 0.905. That table is this lesson's true scoreboard - precision and recall trade off, and which way to move the threshold depends on whether false positives or false negatives cost more. Spam filtering fears blocking real mail, so set it high; cancer screening fears missed diagnoses, so set it low.

Scanning 0 to 1 and Getting a Negative Number

Course exercise 3: for 100 thresholds between 0 and 1, compute true-positive and false-positive rates, draw the ROC curve, and compute the area under the curve with the trapezoid rule. Followed literally, it returns -1.0.

Not a calculation error - the direction is flipped. As the threshold rises from 0 to 1, the false-positive rate falls from 1 to 0 - the curve is drawn right to left. The trapezoid rule follows point order, so the area comes out negative. Reversing the two arrays' order gives the normal +1.0: on this test set, the highest probability for a truly benign case is 0.311 and the lowest for a truly malignant case is 0.563, with a 0.25-wide gap between the two groups; any threshold inside the gap scores perfectly. The ROC curve hugs the left and top edges in a right angle - the rounder that right angle, the wider the gap between the two groups' scores.