Unsupervised Learning Without Answer Keys: A Hands-On Audit of K-Means, DBSCAN, and GMM
Hospitals hold millions of medical records that no one has ever labeled disease by disease. E-commerce sites store millions of user sessions with no customer segments ever drawn. Security teams watch massive network logs where no one has annotated which entries are attacks.
Up to the previous lesson, every model we built was waiting for a standard answer. Starting here, the answers are gone.
Unsupervised learning deals with exactly this situation: you don't tell the algorithm what to look for; you let it stare at the data until structure emerges on its own. Supervised learning is studying a textbook with an answer key; unsupervised learning is staring at raw data until you start seeing the pattern.
The cost is obvious: with no standard answer, there is no accuracy to compute. You need a different set of tools to judge whether the structure the algorithm found actually makes sense.
The Compass: K-Means
K-Means is the workhorse of clustering, simple enough to state in one sentence: guess K centers, assign each point to the nearest center, move each center to the average position of its members, repeat until nothing moves.
The course provides a from-scratch implementation of about forty lines. On three well-separated Gaussian blobs, it converges on the fourth iteration, with centroids landing at [5.0, 8.05], [2.12, 1.98], and [8.11, 3.16] — the true centers are [5,8], [2,2], [8,3], all within sampling error. I cross-checked against scikit-learn's official implementation: inertia 190.2212, matching to six significant figures. The hand-written implementation is solid.
Inertia deserves an on-the-spot definition: the sum of squared distances from each point to its centroid; smaller means tighter clusters. You must choose K yourself, and two common helpers exist:
- Elbow method: increase K from 1 to 6; inertia keeps dropping, but at some point the drop suddenly slows and the curve bends like an elbow — that's your K.
- Silhouette score: each point asks "am I close to my own kind, or to the neighbors?" Being much closer to your own kind earns points, on a scale from -1 to +1.
- Textbook hand-written GMM: 52% accuracy — a coin flip.
- K-Means, ironically: 68%.
- scikit-learn's GMM with full covariance: 92%.
On this data both helpers agree: the elbow is at K=3 and the silhouette peak is at K=3.
Everything went smoothly — because the data was round, well-separated, and K was known. Remove any one of those three conditions and things get interesting.
The Moon Crash Site
The course builds a deliberately tricky dataset: two interleaving crescent moons, one hundred points each, noise 0.1. Then two contenders take the stage.
K-Means's fate was foreshadowed by the course: it splits wrong. The two centroids each cut a moon in half — K-Means only holds a compass, and the moons look like two blobs that should be separated by a straight line. Silhouette: 0.49. Printed conclusion: "K-Means splits moons poorly."
The DBSCAN section is titled "non-spherical clusters" — it's supposed to be this algorithm's moment to shine — and the text states "DBSCAN can find clusters of arbitrary shape."
The actual output: Found 1 clusters, 0 noise points.
One cluster. The two moons were welded together, with zero noise points left over.
I fed the same parameters to scikit-learn's DBSCAN: identical result, 1 cluster. So it's not a bug in the hand-written implementation — it's the parameters. The course set the neighborhood radius eps to 0.3, and I measured all 200 points pairwise: the closest pair of points between the two moons is 0.169 apart. A 0.3-radius neighborhood ball covers that pair, and the chain reaction spreads until the moons merge.
How narrow is the boundary? I scanned eps downward: 0.28, 0.25, 0.22 — all one blob. 0.215: still welded. 0.210: separated — two clusters, zero noise, 99.5% agreement with the ground truth. Between correct and welded, the window is under 0.01 wide; the course's 0.3 sits forty percent away from the boundary.
More painful still: the moon-generation function returns the true labels in a variable literally named moon_labels. The main program never touches it. Even a single glance at it would have exposed the 1-cluster disaster immediately. Skipping evaluation with unsupervised methods is forgivable — but the ground truth was right there in hand.
This is the price of "not having to choose K." DBSCAN indeed doesn't make you pick a cluster count, but it moves the decision to eps. A wrong K gives you visibly over- or under-segmented results; a wrong eps can fail silently like this — no error, no protest, one line of Found 1 clusters, and nobody notices.
The Ruler Also Has a Bias
Beyond the crash, there's something more subtle.
DBSCAN with eps=0.21 clusters the moons correctly, 99.5% accuracy. Its silhouette score? 0.33.
The wrongly-split K-Means scores 0.49.
The correct answer scores lower than the wrong one. This isn't the moons' fault — it's the ruler's. The silhouette measures "tight within, far between," a standard that inherently favors compact, round blobs. The correct moon clustering naturally has large average intra-cluster distances — moons are elongated; even a moon's head is far from its own tail. Measure a non-round correct answer with a round-favoring ruler, and you get a low score.
The course teaches the silhouette as a universal K-selection tool without mentioning this bias. On spherical data it's excellent — the K=3 peak of 0.75 points precisely and steadily. On moons, it tells you the wrong answer is better than the right one. The tool isn't broken, but every tool has a stance. Know where it stands before you use it.
Soft Assignments and the Round Lid
The third contender is GMM, introduced movingly: K-Means does hard assignment — every point belongs to exactly one cluster — while GMM does soft assignment, giving each point a probability for every cluster. And the text states verbatim: "GMM can model elliptical clusters, unlike K-Means which only does spherical ones."
The ellipses are real — just not in the textbook's code.
Look at the implementation: each Gaussian's shape is defined by a single variance — one number. A Gaussian shaped by one variance has circular iso-probability contours. To draw ellipses, each direction needs its own stretch, which requires a full covariance matrix, not a single number.
I built an exam to verify: two elongated elliptical blobs, one at 45 degrees, one at 135 degrees, axis ratio 10:1, tails overlapping. Results:
The elliptical capability isn't a lie — it's an upgrade left uninstalled. The EM algorithm, soft assignments, probabilistic membership — the skeleton is all real. Swap each Gaussian's single variance number for a covariance matrix and ellipses appear immediately. The course leaves this as an exercise; the capability claimed in the text only exists once the exercise is done.
Trees and the Points Nobody Wants
Two supporting characters remain.
Hierarchical clustering builds bottom-up: every point starts as its own cluster; repeatedly merge the two closest clusters; the entire merge history is recorded as a tree, the dendrogram. Want K clusters? Cut the tree at the right level — no need to pre-select K, but memory grows with the square of the point count. The course covers four linkage measures (single, complete, average, Ward), then leaves Ward as an exercise — there's no hierarchical clustering code in the main implementation.
Anomaly detection comes free with clustering. Toss three outliers — [20,20], [-5,-5], [15,0] — into the three blobs, and DBSCAN's noise labels catch all three. GMM's approach: check each point's probability under all Gaussians; low everywhere means weird. K-Means: check distance to the nearest centroid. Exercise 3 asks you to compare DBSCAN's and GMM's anomaly lists — where the two methods disagree is often where the data has the best stories.
The Truth Hidden in the Exercises
Exercise 1 asks you to implement K-Means++: pick the first centroid randomly, then draw each new centroid with probability weighted by squared distance to existing centroids — the farther, the more likely. I wrote it and compared. Six blobs, 360 points, 30 random seeds: K-Means++ converges in 9.9 iterations on average, random initialization in 13.2 — a quarter faster.
But what about solution quality? On this data, random initialization averages inertia 763.9; K-Means++ actually gets 783.3. Worst case: 915.6 vs 1084.6 — random is even more stable. K-Means++'s theoretical guarantee is in expectation — on average it's not bad, but no single run is guaranteed better. On well-separated data, random centroids converge to good solutions anyway; what truly needs K-Means++ is data where a bad initialization falls into a pit.
If you want to watch the moons crash yourself: change the textbook DBSCAN moon section's eps from 0.3 to 0.21 and rerun. Found 1 clusters becomes Found 2 clusters, zero noise, 99.5% agreement with ground truth. Then change it to 0.215. Between one blob and two moons lies a gap of 0.005.