AI Engineering Foundations 14: Norms and Distances — Change the Ruler, Change Your Nearest Neighbor

This is part of the "AI Engineering Foundations" series: working through, verifying, and reviewing the open-source course AI Engineering from Scratch (58k GitHub stars, MIT license), one lesson at a time. Today: Lesson 14, Norms and Distances — a Build-type, 90-minute lesson; prerequisites are the earlier linear algebra lessons.

Your "find similar" code is really a choice of ruler

A concrete scenario: you write one line of code that fetches the ten most similar documents to a user's question from a vector database. You call similarity_search with cosine. You probably didn't think hard about it — everyone uses cosine.

That unexamined choice defines what "similar" means in your entire system. KNN classifiers, recommendation engines, clustering, loss functions, regularization — they all stand on the ruler you pick. Change the ruler, and the nearest neighbor changes.

Norms measure a vector's "size":

• L1 (Manhattan): sum of axis-wise moves. From (1,1) to (4,5): 3 + 4 = 7 blocks.
• L2 (Euclidean): the straight line. 3-4-5 triangle, answer is 5.
• L∞: only the largest component difference — "the dimension where we differ most differs by 4."

Same two points, three rulers, three answers: 7, 5, 4. Nobody is wrong.

As p → ∞, the Lp norm converges to the max component — not a convention but a limit: higher powers give the large components more say, and at infinity the small ones drown. Hence the unit ball deforms with p: a diamond at p=1 (corners on axes), a circle at p=2, a square at p=∞. Remember these shapes — regularization will test you on them later.

Cosine measures something else: angle. Two documents where one is twice as long with identical proportions: L2 says they're far apart, cosine says they're identical — magnitude is noise, direction is meaning. That's why it dominates NLP. The dot product is un-normalized cosine; once embeddings are normalized, they're exactly the same.

One level deeper: Mahalanobis distance measures "how anomalous you are within the data's shape." Height and weight are strongly correlated: 6'2" at 180 lbs is normal, 5'0" at 180 lbs is anomalous — Euclidean can't tell them apart, Mahalanobis factors in the covariance matrix (whiten first, then measure). The course demo: moving 3.84 units along the correlated axis gives Mahalanobis distance 0.98; moving 3.16 units perpendicular gives 3.86. Euclidean says the first is farther; Mahalanobis says the second is four times more anomalous. Same data, two conclusions.

Then the non-vector rulers:

• Jaccard for sets: intersection over union — IoU in segmentation models.
• Edit distance for strings: kitten→sitting takes three operations, computable via dynamic programming on a matrix.
• KL divergence for distributions — but the course spends a full section stressing: it's asymmetric, violates the triangle inequality, it's a divergence, not a distance.
• Wasserstein also handles distributions, and it *is* a true metric, remaining finite even for completely non-overlapping distributions — where KL jumps to infinity and gradients die. That's why WGAN replaced it.

Finally, regularization, the buried thread of this lesson. Add the L1 norm (Lasso) to the loss and weights become exactly zero in groups; add squared L2 (Ridge) and all weights shrink but none hit exactly zero. Why? Back to those three shapes. When the constraint region is a diamond, the loss contours (ellipses) most likely hit a corner — corners sit on axes, and being on an axis means some weights are zero. Circles have no corners; any contact point is smooth, both weights survive. L1's sparsity and L2's smoothness aren't mysticism — they're geometry.

Customs inspection

As usual, I ran the code and checked every number. Most of this lesson's numbers are clean: all six edit-distance cases correct (algorithm→altruistic is 6, python→pytorch is 4); all five Wasserstein cases match hand-computed CDF values; the ordering L∞ ≤ L2 ≤ L1 held in five random trials; the KL asymmetry demo is real (0.368 vs 0.511); normalized dot product equals cosine to six decimal places.

But two demos promised something in their titles and didn't deliver.

First: "same data, different metrics, different nearest neighbors." The code builds 8 points and prints nearest neighbors under four metrics. Actual results: L1 picks Point 0, L2 picks Point 0, cosine picks Point 0, L∞ also picks Point 0. All four metrics agree completely. The demo even contains a "metrics disagree" printout — but its trigger (all four selecting different points) never fired, so it silently skipped.

Second, even more direct: "distance metric changes the prediction." Nine training points in three classes, query at (2.8, 2.8), k=3 KNN under four metrics. Actual: all four predict class C. Not one flipped.

Does the phenomenon not exist? It does. I swept 121 query positions over the same training set: at 42 positions the four metrics disagree. The most dramatic is (1.0, 0.5): L1 votes A, L2 votes C, cosine votes B — three metrics, three answers. Move the query from (2.8, 2.8) to (1.0, 0.5) and the title delivers. The course's query happened to sit 0.28 from C's center, a comfort zone where every ruler names the same neighbor. Exercise 3 asks students to "find a dataset where all four metrics disagree" — something the course's own demo failed to do, turning it into homework. Incidentally, it's solvable: a minimal four-point example with query (1,1) gives L1→(1,3), L2→(2.95,1.3), cosine→(5,5), L∞→(1.95,2.9). Four metrics, four answers, no overlap.

Second finding: a double standard. The course kicks KL out of the distance club with clear justification: asymmetry, no triangle inequality. Both check out. But apply the same gate to the course's own cosine distance: 1 minus cosine. Symmetry passes; the triangle inequality fails. A counterexample uses the course's own example: a=(1,0), b=(0,1), c along the angle bisector. cos_dist(a,b)=1 while cos_dist(a,c)+cos_dist(c,b)=0.293+0.293=0.586. The direct path costs more than the detour — triangle inequality violated. This is a classical result: 1−cosine isn't a metric; the true angular metric needs arccos divided by π. Nobody in engineering cares — vector databases use it and it works as a ranking signal — but the course is strict with KL and lenient with cosine: it uses the metric axioms to expel KL while cosine distance, failing the same exam, sits in every "when to use" table with zero warning. Also, the body opens by asserting "any distance function between two vectors can be written as the norm of their difference" — cosine distance precisely cannot; it's half the squared norm of the difference after normalization, not a norm.

Third finding: unguarded interfaces — the third occurrence of this family in the series. kl_divergence pairs distributions with Python's zip, silently truncating mismatched lengths: KL([0.9,0.1] ‖ [0.5]) doesn't error — it computes 0.529. Worse, it doesn't check normalization: KL([0.9,0.1] ‖ [0.9,0.2]) returns −0.069, a negative KL. Mathematically KL is non-negative by Gibbs' inequality; feed it two unnormalized "distributions" and it quietly returns a mathematically impossible number. Meanwhile wasserstein_1d does assert on unequal bin counts (good) but doesn't check total mass: [0.5,0.5,0,0] vs [0,0,0.5,0.9] — right side totals 1.4 — still yields 2.4. One is "the function name says distributions, any input computes"; the other is "half-guarded."

Fourth finding: a mechanism mismatch in the regularization demo. The body teaches sparsity via contours hitting the diamond's corners — correct, standard picture. The demo implements something else: each step pushes every weight toward zero by a fixed step, 50 steps, total budget 0.1×50=5. The largest initial weight magnitude is only 2.995, so the budget exceeds everyone's norm: all 10 weights die, 10/10 zeros. That demonstrates annihilation, not sparsity (sparsity means some zeros and some non-zeros). And a pure decay process has no loss function holding any weight back — a different mechanism from the taught "contours meeting corners." Change steps from 50 to 10 (budget 1.0) and the same code instantly becomes genuinely sparse: 8/10 zeroed, with 1.404 and −2.995 surviving — small weights die, big weights live. That *is* Lasso behavior. The L2 demo has a display issue: weights multiply by 0.8 each step, so after 50 steps they're 1.4×10⁻⁵ of the initial value; three-decimal printing shows all ten as 0.0 — the "never exactly zero" selling point is hidden by its own print precision; you need the zero-check row (threshold 10⁻¹⁰) to see none actually hit zero.

Three small accounts. The Mahalanobis demo prints "r ~ 0.8"; the measured correlation is 0.923 — the constructed data's slope 0.8 got reported as a correlation coefficient, in a lesson that just taught covariance. The glossary says KL "measures extra bits when using Q to encode P," but the code uses natural logs throughout and the demo itself prints nats — bits would require log2; two unit conventions in one lesson. The task table lists KL divergence in the "best distance" column right after a section explaining it isn't a distance; casual readers will memorize it as a distance. One harmless extra: the loss function table writes Hinge loss as max(0, margin − d), where d is never defined.

Finally, credit where due — Exercise 4 is well designed: Wasserstein([0.5,0.5,0,0], [0,0,0.5,0.5]) = 2.0 (both mass lumps move two cells), while Wasserstein([0.25,0.25,0.25,0.25], [0,0,0.5,0.5]) = 1.0 (half the uniform mass is already in place). Same destination, wider starting spread, less work to move. And to see "change the ruler, change the neighbor" with your own eyes: change the course's query from (2.8,2.8) to (1.0,0.5) and run the four KNNs — L1 says A, L2 says C, cosine says B. That one-line change is this lesson's entire thesis, live.

---

*Original lesson: ai-engineering-from-scratch, Lesson 14 "Norms and Distances" (Rohit Ghumare, MIT license). Full translation archived.*