This is episode 03 of the AI Engineering Foundations series. I'm working through *ai-engineering-from-scratch*, an open-source course with 58k stars on GitHub — 523 lessons, from linear algebra all the way to agent clusters. One lesson, one post: full translation saved first, then the lesson retold in my own words.
When you read about PCA, the first line is "compute the eigenvectors of the covariance matrix." Read about model stability, and the first line is "check that all eigenvalues have modulus less than 1." Read data-augmentation code and you see apply_random_rotation(). All of these share one prerequisite: you know what a matrix does to space. Without that, they're just incantations.
The first two lessons laid the foundation: lesson one said matrices are *actions*, lesson two had you type those actions out by hand. This lesson zooms out: what shape does the whole space become under the action? And a sharper question — is there any direction the action *cannot* bend?
Four basic actions of the space machine
In 2D, every linear transformation is a 2×2 matrix. The two columns of the matrix are the new addresses of the original basis vectors [1,0] and [0,1]. Once you know where the basis vectors went, every other point's destination follows. This is the cheapest shortcut in the whole subject.
Four basic parts. A rotation matrix moves points along circular arcs — distances and angles untouched. A scaling matrix stretches along each axis independently: x doubled, y halved, and the unit square becomes a rectangle. A shear matrix pushes one axis proportionally to the other, turning a rectangle into a parallelogram. A reflection is a mirror: x flips, y stays.
Actions can be chained, and chaining is matrix multiplication. But order matters — not a courtesy warning, an iron law. Take the point (1,0): rotate 90° then scale by (2, 0.5) and you get (0, 0.5). Scale first, then rotate, and you get (0, 2). Same start, same two actions, four-fold different landing point. Mathematically S @ R and R @ S are different matrices — matrix multiplication is not commutative. Every time you write a @ b versus b @ a in a framework, you're choosing between these two paths.
Directions that can't be twisted
Hit most vectors with a matrix and their direction changes. Rotate 45° and an eastward arrow now points northeast. But some matrices carry special directions: the direction stays, only the length scales. Those directions are eigenvectors; the scaling factors are eigenvalues. Plain language: this is the transformation's fingerprint.
The course's core example: matrix [[2,1],[1,2]]. Along direction [1,1] it stretches by 3: A @ [1,1] = [3,3], still on the original direction. Along [1,-1] it does nothing: A @ [1,-1] = [1,-1]. Every other direction in the plane is a blend of these two axes. The matrix looks like four numbers, but its personality is two axes.
How do you compute eigenvalues? Solve the characteristic equation. For a 2×2 matrix [[a,b],[c,d]], the eigenvalues are the roots of λ² − (a+d)λ + (ad−bc) = 0. The trace and the determinant each make one appearance. The course builds this solver from scratch in ~20 lines, matching NumPy exactly.
Then the course plants three threads. PCA: principal components are eigenvectors of the covariance matrix — directions with large eigenvalues are where data spreads widest; keep those, cut the rest. Recurrent network stability: eigenvalue moduli above 1 explode, below 1 vanish — vanishing and exploding gradients in one sentence. Spectral clustering: eigenvalues of the Laplacian cut graphs — the L = D − A from lesson 21's graph theory ties back here.
The determinant also gets an upgrade. Last lesson it was a flattening detector (det = 0 means area crushed to zero). This lesson gives the full picture: det is an area scaling factor. Rotation: det = 1, area unchanged. Scaling (2,3): det = 6, area ×6. Reflection: det = −1, area unchanged, orientation flipped. Shear: det = 1, skewed but not lost. det = 0 means the space is compressed into a lower dimension — information destroyed, transformation irreversible.
Through customs
I ran all the course code. Rotating (1,0) by 45° gives (0.7071, 0.7071); the composite S @ R = [[0,−2],[0.5,0]] with eigenvalues 3 and 1; the five determinants 1, 6, 1, −1, 0; exercise answer [[4,2],[1,3]] has eigenvalues 5 and 2 — all checks pass. Even the floating-point details are honest: cos(π/2) evaluates to 1.22e-16, not exact zero; printed to two decimals it reads as zero.
But this time I caught a place where the signature is broader than the claim. The course's glossary states "a shear matrix has determinant 1." That only holds for single-axis shear. The code's function is shearing_2d(kx, ky), returning [[1, kx], [ky, 1]] — it shears both axes at once. Its determinant is 1 − kx·ky, not identically 1. Tested with kx = ky = 1: the determinant is zero, the whole plane is flattened onto the line y = x, and points (1,1) and (3,−1) both map to (2,2) — the transformation is irreversible. The course itself only demos shearing_2d(1, 0), dodging the landmine. But a student who casually plays with the parameters per the signature will flatten the plane with their own hands. Fun coincidence: this landmine is the best possible teaching aid for this lesson — det = 0 means space collapse, a ready-made live example.
Two other loose spots:
1. The text says eigenvectors are "directions a matrix only scales, never rotates," without noting some matrices have no *real* such directions at all. Rotation matrices are the case: a 45° rotation has eigenvalues 0.7071 ± 0.7071i — pure complex, no real direction survives pointing where it started. The course code actually handles the complex branch; the conceptual text never spells it out.
2. "Eigenvalue modulus above 1 explodes" is the linear-systems criterion. In real recurrent networks, explosion is governed by the spectral radius of the recurrent Jacobian, with activation derivatives in between — sigmoid's derivative peaks at 0.25, which is exactly the mathematical source of the earlier lessons' claim that ReLU beats sigmoid. Fine to compress to one sentence in an intro course, but you should know what got compressed away.
Slogan audit. This lesson's opener is "a matrix is a machine that reshapes space." That claim holds up — more honest than lesson one's "it's all matrix math." What's hidden is the word linear: this machine only does what linear maps can express. Last lesson you hand-wrote relu(W @ x + b); that bias b is a translation, translations don't preserve the origin, so they're not linear transformations and the matrix machine can't produce them. The course says every transformation a network applies to data is "either one of these operations or a composition of them" — within the linear domain that's fully true: any linear transformation factors into rotation × scaling × rotation. But every dense layer carries that extra b; restating it as "matrices handle stretching, biases handle relocation" is more complete.
Exercise 2 is worth doing by hand. Matrix [[4,2],[1,3]]: trace 7, determinant 10, characteristic equation λ² − 7λ + 10 = 0, factoring to (λ−5)(λ−2) — a two-minute job. Then cross-check with the course's from-scratch solver, and again with np.linalg.eig: three sources agreeing before you sign off. Exercise 3 is this lesson's final exam: three transformations chained on eight points of a circle, verifying the composite matrix's determinant equals the product of the three determinants — rotation 1 × scaling 1.2 × shear 1, so you should guess the answer 1.2 before computing. Next lesson we leave linear algebra for calculus: matrices tell you what shape space was kneaded into; gradients tell you which direction to knead for the best return.
---