AI Engineering Foundations 32: Error Splits Three Ways, But Only Two Parts Are Treatable

Forum topic · 小凯 · 2026-10-28

Summary

This post is a detailed technical review of a machine learning lesson on the bias-variance decomposition, tested hands-on in a sandbox where the true function sin(1.5x)+0.5x is known. A degree-15 polynomial shows training error 0.06 versus test error 3.95, illustrating overfitting. The author verifies the decomposition into bias-squared, variance, and noise floor (sigma=0.5, floor=0.25), and exposes subtle pitfalls: the course's identity-based decomposition 'check' is not a real test until labels carry fresh noise each round; learning-curve tables mix noisy training error against clean test error, producing a fake negative gap of about -sigma^2; regularization strength only matters relative to basis scale (a monomial basis needs lambda near one million to bite, while Chebyshev polynomials flip bias past variance at lambda=10); variance estimates at high degrees are lottery-like, unstable across random seeds; bagging cuts variance tenfold at degree 3 but fails (ratio 1.08) at degree 12 because member models blow up together; and pushing past the interpolation threshold reveals the double-descent right arm, though it never returns to degree-5 performance. With zero noise, the optimal degree jumps from 5 to anywhere between 7 and 25. The takeaway: of the three error components, noise is untreatable.

This is a hands-on audit of a machine learning lesson on bias and variance, using a sandbox where the ground-truth function is known: data are 30 points sampled from sin(1.5x)+0.5x with Gaussian noise, sigma = 0.5.

The headline: a degree-15 polynomial scores training error 0.06 and test error 3.95 — a 61x gap. A linear model gives 0.80/0.65; a degree-4 model gives 0.30/0.32. As capacity grows, training error keeps improving while test error first falls, then explodes.

The three-part decomposition

Under squared loss, expected error splits exactly into:

Bias: how far the average prediction (over hundreds of re-drawn training sets) is from the truth. Systematic error; a line cannot bend to fit a curved law — underfitting. Degree 1: bias-squared 0.43, variance 0.04.
Variance: how much predictions scatter across training sets. A degree-15 polynomial passes through every training point, but each new point set produces a different fit — it fits noise, not signal — overfitting. Degree 15: variance in the hundreds to thousands.
Noise: the irreducible floor, sigma^2 = 0.25 here. Degree 5: bias-squared 0.0012, variance 0.05, total 0.30 — essentially sitting on the floor, which is the achievable limit.

The sandbox's luxury, and two hidden design decisions

The decomposition requires knowing the true function and being able to redraw data — a sandbox-only privilege. Two quiet design choices dominate the numbers:

Training data span [-3, 3] but the test grid only [-2.5, 2.5], dodging the wildest polynomial region at the edges. Extending the test grid to [-3, 3] raises a degree-10 model's variance from 1.4 to 99.3.
The course's 'bootstrap samples' are not bootstraps; they are fresh draws from the true distribution — cleaner, but impossible in the real world.

The four-decimal 'check' that isn't one

The lesson's check — bias-squared + variance + noise = total error — matches to four decimals but is vacuous: the noise term is hardcoded at 0.25, and bias-squared + variance against the clean true function is an algebraic identity with total error. A real test compares predictions against freshly noised labels each round; then the agreement is within 0.5–2% — measured, not gifted.

The negative learning-curve gap

The learning-curve table shows the gap shrinking past zero (degree 5, n=300: train 0.2497, test 0.0051, gap −0.24 ≈ −sigma^2). Cause: training error is measured against noisy labels (contains the 0.25 floor), test error against the clean truth — two rulers, so the gap column is shifted by one sigma^2. Table summary sentences ('degree 12: train low, test high'; 'more data doesn't affect bias') contradict their own printed numbers; the latter discrepancy comes from Monte Carlo noise in a 200-draw average (variance 235 per row).

Regularization: the knob's range matters

Sweeping lambda from 0.001 to 100 on a degree-15 monomial basis never shows the textbook U-curve's right end (variance 47, bias-squared 0.59 even at lambda=100). Reason: the 15th basis column reaches 3^15 ≈ 14 million, so lambda=100 is negligible. Switching to bounded Chebyshev polynomials — same function space — restores the curve: bias 0.23 vs variance 0.10 at lambda=10; 0.99 vs 0.05 at lambda=100; the monomial basis needs lambda ≈ one million. Lesson: regularization strength is lambda relative to the basis scale. PolynomialFeatures + Ridge without feature scaling hits exactly this trap.

Variance at high degree is a lottery

The degree-15 variance printed as 26315.1214 (four decimals) jumps between 8,306 and 1.14 million across seeds, and grows with more draws — the second moment is dominated by rare catastrophic fits (median max prediction 1.9, maximum 1262). Non-monotonic table entries (degree 9: 110; degree 10: 2.7) are per-row lottery draws; the trend is real, the ordering is luck.

Bagging fails exactly where you need it

At degree 3, averaging 10 models cuts variance 0.0222 → 0.0022 (ratio 0.100), matching the sigma^2/10 textbook claim. At degree 12: 21.2 → 23.0, ratio 1.08 — no reduction. Pointwise correlation between members is only 0.02, so 'imperfect independence' is the wrong explanation; the real cause is that when one member blows up, its bag-mates (sharing the data's geometric weakness) blow up with it, dragging the average to 647. The same knob that creates variance (model capacity) also neutralizes the cure (averaging).

Double descent, completed

The course teaches double descent but stops at degree 15, while the interpolation threshold sits near degree 29. Pushing through: error explodes to 10^18–10^22 near degree 29, then falls eight orders of magnitude from degree 36 to 60. The right arm is real — but it never returns to degree-5 performance (10^14 vs 0.30). The worst place to stand is exactly the threshold: stay below it or go past it.

Turning noise off

With sigma = 0, optimal complexity jumps: degree 7 total error 0.000015; degrees 10–25 all print 0.000000. What makes complexity expensive is not complexity itself but noise. Of the three error parts, only two are treatable — the noise floor has no medicine.