AI Engineering Foundations 12: Tensor Operations — Shape Errors That Raise Are Cheap; the Silent Ones Are Expensive
This is part 12 of the "AI Engineering Foundations" series, working through the open-source course ai-engineering-from-scratch (58k GitHub stars, MIT license): full translation, formula-by-formula audit, then a plain-language retelling. Today's topic: tensors.
You build a transformer, run it, and it crashes: mat1 and mat2 shapes cannot be multiplied (32x768 and 512x768). You transpose, and the error becomes Expected 4D input (got 3D). You add an unsqueeze, and something else explodes. This isn't stupidity — a transformer chains together dozens of reshapes, transposes, and broadcasts; one wrong axis cascades. The course puts it more bluntly: some shape errors never raise at all. They silently broadcast along the wrong dimension, sum over the wrong axis, and produce perfectly-shaped garbage. This article explains the machinery and actually tests how quiet the "silent" failures are.
What a tensor is
A multidimensional array plus one discipline: uniform dtype. The number of dimensions is the rank; each dimension is an axis; the shape is the tuple of axis sizes. A batch of 32 color images at 224×224 is (32, 3, 224, 224) — four axes: batch, channel, height, width. Multi-head attention with 12 heads also has four axes: (batch, heads, seq_len, head_dim).
One term needs defusing on the spot: a tensor's rank is not the linear-algebra rank of a matrix. The matrix [[1,2],[2,4]] has linear-algebra rank 1 (row two is twice row one, linearly dependent), but as an array its rank is 2 (two axes). The course glossary itself flags this — rare honesty in a textbook.
Memory: strides
Arrays live in memory as a single line. Strides tell you how many elements to skip along that line when you advance one step along an axis. A (3, 4) matrix with strides (4, 1): next row skips 4 elements, next column skips 1.
This explains why transpose is cheap: it moves no data, just swaps strides so the same memory is read differently. It also explains the NCHW vs NHWC war: PyTorch defaults to channels-first, TensorFlow to channels-last, and a mismatched layout is not just slow — it's often silently slow.
Broadcasting
How do differently-shaped tensors add? Three rules: align from the right; each dimension pair must be equal or one of them must be 1; the lower-rank side gets 1s prepended on the left. Example: (8, 1, 6, 1) + (7, 1, 5) → pad the latter to (1, 7, 1, 5) → result (8, 7, 6, 5). All size-1 axes are stretched to match the other side. This lets a bias vector (D,) add directly onto a (B, T, D) batch in one shot.
But note the scope: broadcasting requires shapes to be *compatible*, not *correct*. (B, 1) - (B,) is perfectly legal and yields (B, B) — a B×B matrix. Compute cross-entropy with a label one dimension off, and your loss quietly becomes B×B. No error, no warning, training keeps running.
Einsum
Einsum gives each axis a letter. One rule: letters appearing in inputs but not in the output are summed.
i,i->— dot product (i summed, empty output)i,j->ij— outer product (no letter vanishes)ij,jk->ik— matmul (k vanishes)bhtd,bhsd->bhts— attention scores (d vanishes; Q times K)
(2, 8, 64) → split heads (2, 4, 8, 16) → scores (2, 4, 8, 8) → softmax and multiply V back to (2, 4, 8, 16) → merge heads (2, 8, 64) → output projection. Write every shape on paper first — that's the course's "shape tracker": state the contract before writing code.There's also a clean cost model: one einsum contraction costs the product of all index sizes. bij,bjk->bik with B=32, I=128, J=64, K=128 is 32×128×64×128 = 33,554,432 multiply-adds. Verified — the number is correct.
Customs inspection
All the course's code runs, and every number checks out. But three things get stopped at the border.
First: an error message pointing to a nonexistent method. The hand-rolled Tensor class, on shape mismatch, tells you Use broadcast() first. — but no broadcast method exists anywhere in the class (confirmed via hasattr: False). More awkward: the text explicitly teaches that adding a bias (D,) to a batch (B,T,D) requires unsqueezing to (1,1,D) first — yet (2,2,3) + (1,1,3) still raises a Shape mismatch. The standard operation taught in the text doesn't work on the course's own class. That capability is Exercise 2's homework: write broadcast_to yourself, and only then does the function named in the error message come into existence. An error pointing at a phantom API — a first.
Second: "transpose moves no data" contradicts the course's own implementation. The claim is true in NumPy, and the course's demo indeed uses NumPy to show strides going from (24,8) to (8,24). But the Tensor class students build in the Build It section implements permute by allocating a new array, copying element by element, and recomputing strides for the new shape. reshape copies too. Result: the scratch class is always contiguous — the non-contiguous tensor concept the text teaches doesn't even exist in the class you built. The shape semantics are simulated correctly; the memory mechanics are a different system. You cannot use your own class to verify one of the text's most important claims.
Third: the "broadcasting doesn't copy data" framing. True for the inputs: in a * b, neither a nor b is actually copied. But intermediate results are fully materialized. The course's broadcast-based pairwise distance — subtracting (M,2) from (N,2) — genuinely creates an (M, N, 2) intermediate array. Measured at M=N=2000: 64 MB intermediate, versus 32 MB for the final (M,N) output — the intermediate is twice the result. Extrapolating to M=N=10000: 1.6 GB intermediate, 0.8 GB output. scipy's cdist produces the same answer with O(M+N) memory. Broadcasting saves input copies, not peak memory — two concepts the text's "without copying data" collapses into one.
Two minor notes: the text's NLP example uses (16, 128, 768) while the companion demo runs (16, 512, 768) — inconsistent but both legal; and step 7's code snippet uses an undefined K (present in the full code, dropped in the excerpt).
Slogan audit
The course opens with "tensors are the universal language between data and deep learning." True — but with half a sentence hidden: this language has shape syntax and no semantic vocabulary. (32, 3, 224, 224) carries no information that "this is a cat"; which axis is channels and which is spatial depends on conventions like NCHW or NHWC. The moment data becomes a tensor, semantics stay outside the door. This is the deep reason shape errors are deep learning's most common daily bug: axis semantics are the only contract in the whole system that is never automatically validated — it lives entirely in the programmer's head. (The claim "shape errors are the most common bug" is given without a source — an experiential assertion. But the sentence right after it is the most valuable in the piece: some shape errors don't raise and silently produce garbage. Tested: (B,1) - (B,) yields (B,B), and even with Python warnings escalated to error level, it passes with zero warnings.)
Exercise 2 is this lesson's real diploma. Out of the box, the scratch class raises errors invoking a function that doesn't exist yet; finish broadcast_to, make addition auto-broadcast, and the error message tells the truth for the first time. Test cases are ready-made: (3, 1) plus (1, 4) should give (3, 4). Walk the three broadcasting rules on paper first, and you'll know which axes stretch and how many copies the data makes.