Linear Regression Done Right: When Your Program Finishes but Training Hasn't

Forum topic · 小凯 · 2026-10-19

Summary

A detailed lesson on linear regression as the foundation of machine learning training loops: define the model, define the loss, compute gradients, update parameters. The article shows how gradient descent with unnormalized features (x ranging 0–10) stretches the error surface into a narrow ellipse—slope vs. intercept curvature differing 121×—leaving the solution at MSE 3.698 after 1,000 steps versus 3.644 for the closed-form normal equation. It corrects a common misconception: the cubic cost of the normal equation scales with the number of features, not rows (10M rows × 5 features solve in ~0.7s). It also covers why collinear features blow up coefficients (fixed with a tiny ridge penalty, λ=1e-6), why raw coefficients like 10,145 mislead feature-importance readings until standardization, polynomial regression, and a train/test exercise exposing overfitting (degree-10 test R² dropping to 0.69).

AI Engineering Foundations 24 | Linear Regression: The Program Finished, the Training Didn't

Machine learning proper begins with a straight line. Given a hundred points—house area and sale price—fit the best line, then use it to price new houses. The example lays out, for the first time, the four moves of every training loop: define the model, define the loss, compute gradients, update parameters. Every later algorithm, from decision trees to deep networks, is a reskin of these four steps.

The model is two characters: y = wx + b. The hard part isn't writing the model—it's answering: how close is "close," and how do you find the best w and b?

After One Thousand Steps

The loss is mean squared error. Squaring punishes large errors disproportionately (an error of 10 costs 100× an error of 1, not 10×) and is smooth everywhere, so it's differentiable.

Gradient descent: compute the steepest-ascent direction of the error at the current w and b, step the opposite way, repeat. The error surface here is a bowl; walk long enough and you reach the bottom. The course's from-scratch code (under 100 lines) runs 1,000 rounds.

Result: y = 3.14x + 6.54. Ground truth: 3.0x + 7.0. Looks plausible.

The normal equation—linear regression's closed-form solution—gives y = 3.076x + 6.985 in one step.

The two answers differ. Gradient descent's MSE after 1,000 rounds is 3.698; the formula's optimum is 3.644. Gradient descent hasn't reached the bowl's bottom—the intercept stopped at 6.54 vs. 6.99, the last 6% unfinished. The program finished; the training didn't. And no line in the code tells you this.

The cause: x ranges from 0 to 10 and wasn't mean-centered. This one detail stretches the error bowl into a narrow ellipse—steep along the slope direction, nearly flat along the intercept direction, a 121× difference in curvature. Each step barely moves in the flat direction. Mean-centering x and rerunning the same 1,000 steps lands right at the formula's solution.

The course introduces "standardize your features" later, as new material—yet its own 1D example already got bitten, just lightly: parameters are "about right," no error, no warning, nobody cries out. Without feature scaling, the cost isn't a crash—it's silently never finishing. In real projects that's harder to spot: course data can afford 10,000 rounds; real training runs all day.

Who Judges?

Trust 3.14 or 3.08? Reality offers no ground truth (only the data's creator sees 3.0). Two reliable checks: does the loss curve flatten, and can you compare against a closed-form solution? Here the gap is 3.698 vs. 3.644—small, but not closed. Not closed means not converged.

Same code, same learning rate, 10,000 rounds: matches the formula to six decimal places. The method is fine—it just stopped early. Real projects won't expose this: the program exits normally, the numbers look normal.

The course claims a from-scratch implementation "produces the same results as scikit-learn." Strictly, not yet true here: scikit-learn gives 3.0756 and 6.9853; gradient descent gives 3.1433 and 6.5432—diverging at the second decimal. Converged to the same solution is math; producing the same output is engineering—and requires converging first.

The Closed-Form Formula's Two Preconditions

1. Features must not be nearly identical. Construct two features that differ only at the 10th decimal place; true weights are 2 and 3. The formula returns ~450,000,000 and −450,000,000. The optimum lies along a long ridge; numerical solving walks to its far end—weights explode, predictions still look fine. The fix: add tiny diagonal damping, λ = 1e-6, and the solution snaps to a stable 2.53 and 2.53. This is the embryo of Ridge regression—you're not buying better fit, you're buying a trustworthy solution.

2. It must be computable. The course says big datasets should use gradient descent because matrix inversion is cubic. The commonly misremembered part: the cubic term counts features, not rows. Measured: 10 million rows × 5 features—0.7s for all inner products, the 5×5 solve is instant. Doubling rows doubles time. But 2,000 features → 0.35s; 4,000 features → 1.98s—doubling features costs ~8×. The course's skill file has this backwards: millions of rows are trivial; the real thresholds are many features or data too big for one machine.

Coefficient Ten Thousand Isn't the Biggest

Multiple regression swaps the line for a hyperplane over three features: area, bedrooms, age. Without standardization, training yields weights ≈ 54, 10,145, −1,008. Intuition screams: bedrooms matter most—coefficient ten thousand!

That's the units lying. Area moves in hundreds, bedrooms in single rooms, age in years—three different rulers, coefficients not directly comparable. After standardization: 0.77, 0.28, −0.27—area is the dominant factor; bedrooms and age are minor. Standardization's second value: not just faster training, but readable weights—"one unit of area outweighs two-plus units of bedrooms."

Curved Roads, Straight Cars

Curved relationship? Feed x², x³ as ordinary features—the model stays *linear in the weights*. The course compares degree-2 and degree-5 and notes "degree-5 is slightly better on training data but risky on new data."

Two issues: (1) both numbers are halfway—x, x², x³ on [0,1] look alike; the flattest direction's gradient is ~1/4,000,000th of the steepest, and 2,000 rounds barely move it. Closed-form: degree-2 reaches 0.678, degree-5 reaches 0.711—those are the true floors. (2) "Risky" is asserted with no data—the course never split the data.

Running exercise 2 (cubic signal + noise, 40 points, 25% held out): degree-1 trains 0.41, tests 0.26. Degree-3: train 0.94, test 0.95—learned the real thing. Degree-10: train 0.96, test drops to 0.69. The train score is the model's self-report; the test score is the outside world's verdict. The crack between them is overfitting, and it appears between degree 3 and degree 10.

Exercise 3 leaves a hook: Lasso's absolute-value penalty pushes unimportant weights to exactly zero; Ridge's squared penalty only shrinks them. Plot both penalties—one has a sharp corner, one is round—and the answer hides in that corner.

Key Takeaways

Every ML training loop is: model → loss → gradient → update.
Unnormalized features quietly prevent convergence; the program can "finish" while training hasn't.
Verify convergence via a flat loss curve or comparison with a closed-form solution.
The normal equation's cubic cost scales with feature count, not row count.
Near-collinear features destabilize the closed-form solution; a tiny ridge penalty (λ=1e-6) restores a trustworthy answer.
Standardized weights are the only comparable feature-importance measure.
Train/test gaps, not training scores, reveal overfitting.