This is part 4 of the AI Engineering Foundations series, working through the 58k-star open-source course ai-engineering-from-scratch by Rohit Ghumare, one lesson at a time. Today's Lesson 04: Calculus for ML — derivatives and gradients (Learn-type, 60 minutes). The matrices and eigenvalues built up in the first three lessons all get used here.
The main line
Training a network means turning millions of weight knobs. The course opens with a nice slogan: *the derivative tells you which way is downhill, and that's all a neural network needs to learn*.
That's only half the story. Direction is free. Step size is not the derivative's job.
- The derivative is the local exchange rate at a point: for
f(x)=x², the derivative at x=2 is 4 — nudging x right by 0.01 raises y by about 0.04. - Hold all other variables fixed and vary one: partial derivative. Collect all partials into a vector: gradient. The gradient points uphill, so training walks against it — gradient descent:
w ← w − lr · dL/dw, repeated millions of times. - A neural network is a chain of functions: input, linear, activation, linear, loss. The chain rule says the derivative of a composition is the product of the segment derivatives. Backpropagation is that rule multiplied from output back to input, summing contributions where paths fork and rejoin. That one line is the mathematical core of all deep learning.
- 'Derivatives tell you downhill, that's enough' — the course itself undercuts it around line 100: too large a learning rate overshoots. Direction doesn't price the step; curvature does.
- 'Local minima are rarely a practical problem in high dimensions' — mainstream consensus, but it hides the second half: what *is* common is saddle points and ill-conditioned curvature. Both are covered later in this very lesson.
The new character: the Hessian
The gradient gives slope; the Hessian (matrix of second partial derivatives) gives curvature. At a critical point where the gradient is zero, look at its eigenvalues: all positive → bowl bottom; all negative → hilltop; mixed → saddle. Eigenvalues, introduced in Lesson 03 to describe how a transformation distorts space, return here as terrain classification.
Why curvature matters: Newton's method rescales the gradient by the inverse Hessian — small steps on steep ground, big steps on flat ground, one step to the bottom of a quadratic. The cost: N parameters require an N×N matrix; 1M parameters means 1 trillion entries. So in practice people use Adam — running averages of each parameter's gradient mean and variance, a cheap approximation to second-order information. Taylor series makes the connection explicit: first-order approximation (loss as a straight line) is gradient descent; second-order (loss as a parabola) is Newton's method. All gradient-based optimization is locally treating the loss as a polynomial and walking to the approximate minimum.
Verified numbers and two failed demos
All code was run and every number checks out: the x² derivative table, partial derivative 2x+3y, chain rule 6(3x+1), saddle Hessian eigenvalues one positive one negative, 1M parameters → 1 trillion entries. But:
Demo 1 (Step 7). The lesson demonstrates second-order Taylor approximation using sin at 0, printing order1 and order2 columns. Both columns come out identical (0.1000 vs 0.1000, 2.0000 vs 2.0000). Reason: sin is odd, so the second-order term is 0.5 · f''(0) · h² and f''(0) = −sin(0) = 0. The second-order term is identically zero — the demonstration never happens. The code and formula are correct; the chosen function kills the demo. Use cos to see the second-order term at work.
Demo 2 (Step 8) — the more interesting one, unnoticed by the course. Training y = 2x+1 from scratch, learning rate 0.01, 200 epochs, prints w=2.08, b=0.73. Truth is 2 and 1. The missing 0.27 on b is not a matter of more epochs — it's ill-conditioned curvature. The Hessian is 2·[[11,3],[3,1]], with eigenvalues ≈23.7 and ≈0.34: the w-direction slope is 69× steeper than the b-direction. The learning rate is capped by the steep direction's safety limit, so the slow direction crawls 69× slower. Verified: 400 epochs → b=0.86, 1000 epochs → b=0.98, 2000 epochs → 0.9994. The irony: the lesson just taught Hessian and Newton rescaling, then its own experiment is a living specimen of pathological curvature — without connecting the dots. Running it, you'd think 'just train longer.' Wrong: the bottleneck is curvature, not epochs. The fixes are all taught in the course: momentum (Exercise 3 asks you to add it), Adam, or one Newton step.
One loose table entry
Adam is described as a 'diagonal Hessian approximation.' It tracks the first and second moments of the gradient — the uncentered variance. High variance does not imply high curvature; gradient noise inflates it too. Fine as popularization, not as a definition.
Slogan audit
Exercises worth running
Exercise 3 uses x⁴−3x²: two symmetric bowls at ±1.225, depth −2.25, local maximum at the origin. Comparing with/without momentum here lets you watch momentum carry you across the shallow pit. Also worth running: bump Step 8 to 2000 epochs and watch b grind from 0.73 to 0.9994. Curvature doesn't block you — it just taxes your time.