This is a technical review of Lesson 11 of the open-source course *ai-engineering-from-scratch* (by Rohit Ghumare, ~58k GitHub stars), part of a Chinese blog series on AI engineering foundations. The lesson covers Singular Value Decomposition (SVD) as a 120-minute build-style module, framed as "lifting the hood" on PCA from Lesson 10.
What the lesson teaches
- Why not eigendecomposition? Eigendecomposition only works on square matrices with a full set of linearly independent eigenvectors. SVD's slogan: any shape, any rank, unconditionally.
- Geometry: any matrix acts as rotate → stretch → rotate. A sphere passed through the matrix becomes an ellipsoid; singular values are the ellipsoid's axis lengths. Per-direction bookkeeping:
A v_i = σ_i u_i. - Outer-product form:
A = σ_1 u_1 v_1^T + σ_2 u_2 v_2^T + …. Truncating to the top k terms is provably optimal (Eckart–Young theorem): the best rank-k approximation. Error is the (k+1)-th singular value in spectral norm, or the root-sum-of-squares of the tail in Frobenius norm. - Applications: image compression (store U_k, Σ_k, V_k; claimed 14.6% of original storage at k=50 for an 800×600 image), recommender systems (latent-factor user/movie profiles, filled via dot products; production variants include FunkSVD and ALS), LSA in NLP, denoising (cut where singular values cliff), and the pseudoinverse for stable least-squares.
- Numerical caution: forming A^T A squares the condition number (10^6 becomes 10^12); modern algorithms work on A directly.
- Punchline: PCA *is* SVD on centered data — principal components are the right singular vectors V, explained variance is σ²/(n−1). sklearn's PCA calls SVD internally.
- From-scratch SVD matches NumPy's singular values
[3.4038, 2.2974, 1.0119, 0.4588]with max deviation 2.2e-16; reconstruction error 0. - Pseudoinverse via three routes (manual SVD,
lstsq,pinv) all give[1.5, 1.6667], matching the analytic normal-equations solution. - LSA toy matrix: cos(cat, dog) = 0.999, cos(fish, ocean) = 0.998; cross-cluster pairs all below 0.07.
- Arithmetic claims checked: 480,000 pixels, k×1401 storage, 2.9%/14.6%/29.2% ratios, 10^6 → 10^12 condition squaring.
What the reviewer verified (passing)
Four issues found
1. Image compression demo contradicts the text (biggest problem). The text claims natural images compress well (rank-50 truncation "looks nearly identical," saving 85%), but the demo uses np.random.randn(200, 300) — pure Gaussian white noise, the canonical worst case for low-rank structure. Its spectrum is flat (s1=31.0 vs s200=3.4). The course's own output: k=50 gives 67.2% error and 41.8% storage. Swapping in a rank-5 smooth image makes the promise true (k=5: error 1e-15, storage 4.2%). Also: the demo matrix is 200×300, not 800×600; the 14.6% figure doesn't transfer across sizes; and image = np.random.seed(42) assigns None to image (masked by the next line).
2. Warning vs. implementation conflict. The text says to always prefer np.linalg.svd(A) over np.linalg.eig(A.T @ A), yet the from-scratch implementation's loop body computes A_residual.T @ A_residual — the exact path it warns against. On benign matrices the cost is invisible (2.2e-16), but at condition number 10^8: the A^T A route returns 1.133e-8 for a true 1e-8 smallest singular value (13% relative error); direct SVD returns 1.000e-8 (error 1.8e-9). Exercise 1 has students compare both routes but with default random matrices the gap is zero — the experiment that exposes the harm requires a deliberately ill-conditioned matrix, which the exercise never mentions.
3. Denoising demo ignores its own rule. The method is "keep above the cliff." The data's cliff is after s1: 44.9 vs next 5.5, so k=1 is correct (86.4% improvement); the course code uses k=5 (58.5% improvement) — the extra 4 directions are pure noise. The "clean signal" is an outer product of two vectors, so true rank is 1 (s2/s1 = 1.4e-16).
4. Minor conceptual slip. The opening says eigendecomposition "requires a full set of linearly independent eigenvectors" — that is the condition for diagonalizability, not for the Jordan form's existence. A textbook shorthand, noted for the record.
Slogan audit
The lesson opens: "SVD works on any matrix. Any shape. Any rank. Unconditional." The first half is true; the second half hides in the ledger. Existence of the decomposition is unconditional; its *benefits* are conditional — written in the decay rate of the singular values, a property of the data, not of the decomposition. The course's own noise-matrix demo is the best evidence: it has a perfect SVD, yet rank-50 truncation still leaves 67% error.
Exercise 5's hidden answer
Build a rank-3, 50×40 clean matrix, add noise, sweep k from 1 to 40, find the optimal truncation rank by error against the clean matrix. At the course's four noise levels (0.1, 0.5, 1.0, 2.0) the optimal k is always 3 and the curve is flat — the noise floor for a 50×40 pure-noise matrix sits around 13.4·σ, and the matrix's third singular value (28.5) is only swamped at σ ≈ 2.1, just above the course's highest level. Push further: at σ=4 the optimal k drops to 1; at σ=8 it is 0 — the best approximation is the zero matrix, with error frozen at 78.36. That number is not a coincidence: it equals the root-sum-of-squares of the three clean singular values (56.41, 46.32, 28.51), a live demonstration of the Key Terms entry that the Frobenius norm "equals the root-sum-of-squares of all singular values."