This is lesson 10 of the *AI Engineering Foundations* series, working through the open-source course ai-engineering-from-scratch (58k GitHub stars, by Rohit Ghumare). Lesson 10 covers dimensionality reduction — a Build-type lesson, 90 minutes, with lesson 03 (eigenvalues) as a prerequisite. It turns eigenvectors from a fingerprint into a wrench: 784-dimensional data, rotated, compressed into dimensions you can actually see.
Each sample is 784 numbers — impossible to draw. But a handwritten "7" doesn't need 784 independent numbers: stroke angle, crossbar length, overall tilt; the rest is noise. Dimensionality reduction finds the small manifold where the data actually lives.
Why compression is necessary
Three counterintuitive facts about high-dimensional space:
- Distance fails: pairwise distances between random points collapse to the same value; nearest-neighbor search loses meaning.
- Volume hides in corners: in a 100-dimensional unit cube, the inscribed sphere occupies a ~10^-70 fraction of the volume; data spreads to the corners.
- Sample requirements explode: keeping 10 sampling bins per dimension, 2 dimensions need 100 samples, 20 dimensions need 10^20 — a factor of 10^18 more. You can never collect enough; you must reduce.
PCA in five steps
Subtract the mean, compute the covariance matrix, eigendecompose, sort by eigenvalue, project. The covariance matrix is the co-movement table between features; eigendecomposition finds mutually orthogonal principal directions, with the first axis aligned to the fattest spread of the data. Each axis's captured variance is recorded in the explained variance ratio. The course's example: PC1 captures 47.3%, PC2 25.1%, and four components accumulate to 92.5% — the drop between axes is the shape of the redundancy.
Three ways to choose k: reach 90–95% cumulative variance, look for the elbow, or sweep k against downstream accuracy.
Kernel PCA, t-SNE, UMAP
PCA only rotates axes; it cannot bend. If the data lies on a circle, no straight line separates it. The kernel trick (SVM's old partner) projects data into a higher-dimensional feature space where PCA is performed — the kernel matrix does all the work, no explicit coordinates needed. Concentric circles go in, two linearly separable rings come out.
For visualization: t-SNE preserves only neighborhoods — it computes pairwise probabilities in the original space (near = high probability), then finds a 2D layout where that distribution still holds. Intra-cluster distances are meaningful; inter-cluster distances are not. UMAP is similar but uses approximate nearest-neighbor graphs for speed, and cluster relative positions carry more meaning. Rule of thumb: preprocessing and compression → PCA; seeing structure → t-SNE or UMAP.
Reconstruction error
Project to k dimensions and back, compare with the original; the mean squared error is what you lost. PCA has a clean identity: reconstruction error = sum of discarded eigenvalues / dimension. t-SNE has no inverse transform — it only shows.
The customs inspection: what checks out
This lesson is unusually clean. All code runs verbatim: synthetic circle data compressed from 3D to 2D captures 99.93% variance (the third dimension was a linear combination of the first two plus noise — PCA identifies it exactly); t-SNE on 5,000 MNIST samples runs in 42.7 s with median same-digit distance 18.1 vs 65.0 for different digits (3.6x separation); UMAP installs and runs.
Highlight: Step 4's cross-validation design. Handwritten PCA vs sklearn — explained variance ratios agree to six decimal places, coordinates differ by at most 2.66e-15, machine precision. The course uses the odd-looking abs(abs(a)-abs(b)) comparison: eigenvector signs are free (the same solution times -1 is the same solution), so naive subtraction gets fooled by sign flips (raw difference measured at 8.36). Absolutes first — that is real cross-implementation validation, respecting degrees of freedom.
The reconstruction identity was also verified. On MNIST with 50 components back to 784 dimensions: MSE 746.0020; the sum of the discarded 734 eigenvalues divided by 784 gives 746.1512 — a 0.02% gap. Why? np.cov divides by n-1, the identity assumes population variance (divide by n); multiplying by 4999/5000 makes them match exactly. A degrees-of-freedom bookkeeping note, harmless pedagogically.
Four things to intercept
1. The distance-ratio table lacks a census page. The curse-of-dimensionality table shows ratio ~5.0 in 2D → ~1.02 in 1000D. The numbers reproduce (median over dozens of seeds with 5 random points in a uniform cube: 4.94, 1.70, 1.19, 1.05). But the point count is unstated, and the ratio depends heavily on it: with 100 points, the 2D ratio is 134, not 5. "~5.0" is a property of 5 points, not of 2D space. The trend is real (it converges to 1); the four numbers have no provenance.
2. t-SNE is not "default O(n²)". The course's code uses sklearn, whose TSNE defaults to method='barnes_hut', documented as O(N log N); O(N²) is the price tag of method='exact'. One word wrong — and ironically, that's why 5,000 samples finish in 43 seconds.
3. The k-sweep is a hump, not a plateau. The course claims accuracy flattens well before 784 dimensions, and the flattening point is the operating point. Running its code verbatim: k=10 → 81.5%, k=30 → 89.1%, k=50 → 89.7% — then it turns around: k=100 → 85.5%, k=200 → 87.0%, while raw 784 dims is only 87.7%. A hump peaking at 50. Meanwhile variance rises monotonically: 49.4%, 73.6%, 82.9%, 91.8%, 96.9%, 100%. Caveat: beyond k=100 convergence warnings flood the screen — max_iter=1000 doesn't finish in high dimensions, so under-convergence contributes to the downturn. But the "plateau" narrative fails under the course's own code and parameters. Variance ranks variance, not answers.
This cracks Exercise 3's premise too: build a 50-feature dataset with 5 informative features, run PCA, and see whether the explained-variance curve identifies the data as "actually 5-dimensional." With the official make_classification recipe, the first ten components' explained variance ratios are 0.080, 0.056, 0.028, 0.027, 0.026, 0.025 — component 6 is as fat as component 3; there is no elbow. Five components capture only 21.8% of variance; reaching 90% takes 42 components. The reason: "informative" in make_classification means class separability, not variance — the 45 pure-noise features each contribute ~2% variance, flattening the curve. "The data is 5-dimensional" holds in the information metric; in the variance metric it is 50-dimensional. PCA only sees variance. The best way to teach this exercise is to let students crash into the failure themselves.
4. Gamma, the unwritten knob. The concentric-circle example is correct: standard PCA leaves the inner and outer rings fully overlapping on PC1 (threshold accuracy 51%, a coin flip), while RBF kernel PCA separates them. But γ is never discussed: with the same RBF kernel, γ=5 gives perfect separation (100%); γ=1 gives 50.5% (still a coin); γ=10 collapses back to 58%. Too small a γ — kernel too wide, all points look equally near; too large — kernel too narrow, every point plays with itself. The course gives exp(-γ‖x−y‖²) and never says how to choose γ — which alone decides success or failure.
The slogan audit
The course opens with: high-dimensional data has structure; find the right angle and you can see it. Under the hat hides an unspoken clause — in PCA, the "right" angle is *defined* as the maximum-variance angle. The maximum-variance direction is not necessarily the direction you care about. The k-sweep hump, Exercise 3's flat curve, and 0.95 variance requiring 148 dimensions while 50 dimensions deliver only 82.9% are all footnotes to the same sentence. If what you want to see is class structure, you need t-SNE's eyes — or turn γ up to 5.