AI Engineering Fundamentals 33: Twenty-One 60% Judges Combine into an 82% Verdict
In 1785, French mathematician Condorcet pondered a jury problem: if each judge is only 60% likely to be correct, can adding more judges and voting raise the probability of a correct verdict?
This is computable. With 21 mutually independent judges, each correct with probability 0.6, the probability that the majority votes correctly is a binomial sum — about 82.6%. With 101 judges, it rises to 97.9%.
The course textbook prints these two numbers as 74% and 84%. I recomputed with the binomial distribution: 21 judges at 60% gives 82.6%, not 74%; 101 judges at 60% gives 97.9%, not 84%. The 84% figure is actually the answer to a different problem — 101 people each with 55% accuracy yields exactly 84.4%. The book printed both numbers too low; the second looks like a case of members at 55% misremembered as 60%.
The numbers are misprinted, but the principle stands: a group of barely-passing judgments, as long as they err differently, can vote their way to an excellent result. This lesson is about how to construct such a group of judgments, how to combine them, and where the cost of combining lies.
The Prerequisites Are Harder Than You Think
The key to majority voting working is "erring differently." The book mentions in passing: if all models make the same mistakes, combining them helps not at all. How hard is that condition? Let's test it.
Build 21 classifiers, each individually 60% accurate, but give their errors a common factor — a parameter ρ controls the degree of error correlation. ρ=0 means everyone errs independently; ρ=1 means all models err identically. Take a few intermediate values:
- ρ=0: majority voting hits 82.6%, +22.5 points over a single model.
- ρ=0.5: drops to 63.5%, only +3.5 points of gain.
- ρ=0.8: 61%, only +1 point.
This is why "just add more models" is easier said than done. If models read the same data, learn the same patterns, and make the same mistakes, they are still one model. All the effort ensemble methods expend is essentially about manufacturing *different* errors: swapping samples (bagging), swapping features (random forests), making later models focus on earlier mistakes (boosting), swapping model families (stacking).
Bagging: Train on Resampled Data, Average Out the Jitter
Bagging is the most straightforward approach. Draw a same-sized new sample with replacement from the training set, repeat N times, train N models, and average predictions. Sampling with replacement has a mathematical property: a bootstrap sample covers about 63.2% of the original data, leaving 36.8% untouched. These "out-of-bag" samples come free as a validation set — no extra data held aside.
Each model overfits on its own sample, but in different directions, and averaging cancels the overfitting out. So bagging treats variance — the jitter in predictions.
How much can it actually suppress? In the theoretical best case, averaging 20 fully independent models should cut variance to 1/20 of a single model's. I ran an experiment on the course's data generator: repeat training 40 times, each time a single depth-5 tree plus a 20-tree bagging ensemble, and compare prediction variances. Result: single tree variance 0.764, bagging 0.337 — a ratio of 0.44.
Twenty trees cut variance by a bit more than half, not 95%. The reason is again that ρ: bootstrap samples share roughly 60% of the original data, so trees' errors are correlated, and averaging only removes the uncorrelated part. This is bagging's real discount — the book doesn't mention it.
By the way, one print line in the course demo reads "Variance reduction: +0.113" — it looks like a variance improvement, but it's actually the accuracy difference: single tree 0.750, bagging 0.863, difference 0.113. An accuracy difference is not variance. Two quantities got mixed up.
AdaBoost: Make the Next Model Focus on the Missed Questions
Bagging is every model for itself; boosting is a relay. AdaBoost works like this: start with equal weights on every sample, train a weak model (usually a decision stump); compute its weighted error rate — the lower the error, the more say (α) this model gets in the final vote; then increase the weights of misclassified samples, so the next model naturally focuses on those hard cases. After dozens of rounds, the final prediction is a weighted sum of all models.
The formula is short. The weight-update line is w × exp(-α·y·h(x)): correctly classified samples shrink in weight, misclassified ones swell.
AdaBoost has a beautiful mathematical guarantee: the training error has an upper bound that decays exponentially with rounds. The guarantee is real — I ran 300 rounds on data with noise level 0.35: training accuracy rose from 0.816 to 0.991, and the error upper bound tightened to 0.158. The exponential decay delivered as promised.
What about the test set? The best score came at round 14: 0.863. At round 300: 0.850. Wobbling around 0.85 from start to finish.
That's the boundary of that piece of mathematics: the theorem governs the training set. Training error falling and test score rising are two different things. The book's discussion of overfitting points the right way — too many rounds, and the model starts grinding on samples it cannot classify, some of which are pure noise. But to see it with your own eyes, don't watch the test curve (80 test samples; one miss swings 1.25 percentage points, so the curve is naturally jagged) — watch the widening scissors gap between training and test performance.
Gradient Boosting: Building on the Residuals
Gradient boosting generalizes boosting to arbitrary losses. Its idea is even more direct: after training the current model, compute each sample's residual (true value minus prediction), and have the next tree fit the residuals. Every tree builds on the errors left by all previous trees. Under squared loss, the negative gradient of the loss function happens to be exactly the residual, so "fitting the negative gradient" here means "fitting the errors themselves."
One step in the course's pseudocode looks intimidating: each round uses line search to find the optimal step size γ. The implementation skips it entirely. I computed it for them: under squared loss, γ comes out to 1.0000000000 every round — not a step more, not a step less. Running 100 rounds with γ gives test error 0.372; without γ, 0.373. Skipping it is correct — this step is for other loss functions and is dead code under squared loss.
The learning rate (shrinkage) controls how much each tree contributes. The course demo scanned a table: learning rate 0.5 with 20 trees, error 0.376; 0.1 with 100 trees, 0.307; 0.05 with 200 trees, 0.297; 0.01 with 500 trees, 0.465. Below the table sits a summary: "Smaller learning rates need more trees but often generalize better."
The word "often" can't hold up its own table. The smallest learning rate happens to be the worst row. And there's a simpler explanation: the learning rate times tree count equals 10 for the first three rows and 5 for the last — the smallest row isn't learning slower for stability; it simply wasn't given enough total steps, 500 trees haven't finished climbing the hill. Learning rate is indeed an anti-overfitting tool, but tuning it too small costs real trees.
Stacking: Teaching a Model Whom to Trust
Stacking takes a different approach: feed several base models' predictions as new features to a meta-learner, which learns "given this input, which model should I trust." The forest is good over here, the SVM over there — the meta-learner learns the division of labor.
There's an easy pitfall here, and the course avoids it: meta-features must be generated via cross-validation. If base models predict on data they trained on, and those predictions train the meta-learner, you're handing the meta-learner an inflated report card. The course splits the training set into five folds, and each fold's predictions come from base models that never saw it.
But the course demo slips elsewhere: the chart shows random forest, SVM, and logistic regression as the three base models; the exercises suggest logistic regression, decision tree, and k-nearest neighbors; the actual code deploys — the same decision tree at depths 3, 5, and 7.
Different depth doesn't mean different errors. Three same-family models reading the same data make highly overlapping mistakes; the meta-learner gets three nearly identical feature columns with little room to learn. The measured results agree: the best single base model scored 0.825; stacking stacked up 0.838 — squeezing out 1.3 percentage points. Diversity must come from different errors, not different depths.
There's also a bug in the meta-learner's few lines of code. It uses tanh for output, and its gradient is written as "error times features" — missing the derivative of tanh, breaking a link in the chain rule. Near zero weights, tanh's derivative is exactly 1, so the gradients match and it's invisible; away from the origin, the wrong gradient is 5.6 times the true one. The demo survives because in this toy the model mostly sits in the linear region. If you reuse this code to fit stronger nonlinearity, its steps will be too big exactly where they should be small.
Before Trusting a Table, Read What It Says
At the end, the course runs four models side by side: hand-written AdaBoost 0.870, sklearn's AdaBoost 0.880, random forest 0.890, gradient boosting 0.910. A 1-point gap between the hand-written version and production implementations is reasonable.
But another comparison table deserves mention. Single tree 0.870, bagging 0.880, AdaBoost 0.870. AdaBoost ties the single tree. Beneath the table, the summary still reads: "bagging reduces variance, boosting reduces bias." The first half has 1 point of support; the second half has zero on this table — boosting didn't reduce a single point of bias in this demo. The conclusion itself isn't wrong (with different data or a different base learner it can reduce bias); this table just can't carry it. The same code in another demo lifts bagging from 0.750 to 0.863 — that's where the conclusion belongs.
One smaller slip too: in the gradient boosting table, the 100-tree row prints error 0.307 and the 200-tree row 0.279, yet the closing summary reads "GBM 100 MSE: 0.279" — pasting the 200-tree number onto the 100-tree label. A misread line while copying its own table.
Practical Checklist
For real work on tabular data, the course's roadmap is practical. First shot: LightGBM or XGBoost with default parameters. If that fails, tune four knobs — tree count, learning rate, depth, minimum leaf weight. If you're still half a point short, build stacking from three to five genuinely different models. Cross-validate throughout. Neural networks occasionally tie tuned gradient boosting on tabular data, rarely beat it — that experience doesn't need re-examining anytime soon.
XGBoost's speed and stability come from engineering: regularization penalties on leaf weights, second derivatives of the loss, native missing-value handling, column sampling, cache-friendly data structures. The algorithm is gradient boosting; every point of speed comes from the implementation.
Back to the opening jury. What does a vote among 21 judges at 55% yield? 67.9%. Now you have the binomial distribution — you can compute this number yourself, and along the way verify whether the textbook's two underprinted numbers are misprints, or swapped parameters nobody mentioned.
---
*Source: rohitg00/ai-engineering-from-scratch (MIT License), Phase 2 Lesson 11, Ensemble Methods. The Condorcet probabilities, variance ratios, and gradient checks in this article were independently recomputed from the course code and formulas.*