AI Engineering Foundations 30: One Good Feature Beats a Thousand Data Points

You spent a week tuning hyperparameters and the model is still the same model. You switched to a fancier algorithm and the score didn't budge. Then someone changed how the raw data is represented, and a plain logistic regression beat you.

This isn't a joke—it happens every day in classical machine learning. How you represent data often matters more than which algorithm you pick. A house-price model fed "area, number of bedrooms" will beat one fed raw address strings, no matter how sophisticated the latter's learner is. Algorithms can only eat what you feed them.

Turning raw data into a shape models can easily digest is called feature engineering. This post walks the whole pipeline from scratch: how to transform numbers, fill missing values, encode categories, vectorize text, and remove redundant features. Every step runs, and every step has a landmine someone already stepped on.

Transformations first

Raw numbers rarely go straight into a model. The most common problem is scale: area is in the thousands, bedrooms in single digits. Distance-based algorithms like KNN and SVM will look only at area and ignore bedrooms—not because they're biased, but because the magnitudes are too far apart. So either squeeze all features into 0–1 (min-max scaling) or convert them to mean 0, variance 1 (z-score standardization).

This lesson's example data is 200 hypothetical houses: area from 500 to 5,000 sq ft, age from 0 to 50 years. One detail worth a look: in the course's demo matrix, area uses z-score while age uses min-max—two scales coexisting in the same dataset. The lesson text itself says "put features on the same range," yet its own example has two ranges. Fed to a distance-based model, that's a hazard in itself.

The log transform is another remedy, for right-skewed distributions. Income, population, word counts—most of the mass crowds the low end while a few extreme values drag out a long tail. Take a log and multiplicative relationships become additive; the tail flattens. But the medicine must match the disease: this lesson's area is uniform from 500 to 5,000, not skewed at all. Forcing a log actually twisted it into a left skew (skewness went from 0.02 to -0.64). On genuinely right-skewed data (log-normal, skewness 5.26), the log brought skewness down to a clean 0.01. Oddly, the course's demo code computes the log-transformed area, and then the column is never used again—the prescription was written for a patient who didn't have the disease.

What about binning? Cutting continuous age into bands like 0–10, 10–20 years. It suits one specific case: when the feature-target relationship is step-like, not smooth. Age relates linearly to price in this data ($1,000 depreciation per year), so binning is pure loss. Quantified: using mutual information with "is high-priced" as the yardstick, continuous age scores 0.036; after 5 bins only 0.007 remains—nearly 80% of the information got chopped away. Binning trades information for simplicity; the trade only pays off on step relationships.

Missing values: impute, but at a cost

Real data has holes. Dropping rows is easiest but only works when missingness is rare and random. More common is imputation: median for numeric columns (more robust than the mean against outliers), mode for categorical.

Imputation has a cost worth measuring yourself. In this data, 20 age values are missing and filled with the mean, 25.2. Afterward the column's variance shrinks by 10%, and the 20 identical values pile up into a spike at the exact center of the distribution—values that were each different numbers are now one number. The model will conclude that houses aged 25.2 are unusually common.

The course also teaches adding an "was this missing?" indicator column before imputing. The reasoning is right—missingness itself can carry information (a sensor failing on hot days ties missingness to faults). But this step can't be demonstrated on this data: the missingness is purely random (decided by dice at generation time). Mutual information between the indicator and "is high-priced" comes out at exactly 0.0000. The column was added, computed, and yielded zero. The course isn't wrong; this data's missingness has no story. In the real world, ask "does missingness have a cause?" before adding that column.

Categories: three encodings, three temperaments

Models need numbers, so categories must be translated. One-hot is the most direct: three neighborhoods become three columns, 1 in the matching one. The drawback is explosion—1,000 categories means 1,000 columns.

Label encoding turns categories into integers: downtown=0, far suburb=1, near suburb=2. The problem is that integers imply order, so the model assumes near suburb > far suburb > downtown—an order given by the dictionary, not the world. The course says it "only suits tree models," which omits half a legitimate use: when the categories are genuinely ordinal, label encoding is the correct answer. The lesson's own age bins (0–4) are ordinal, and bin 0 really is newer than bin 4.

The third is the most interesting: target encoding. Replace each neighborhood name with its average price. This borrows "the answer" as a feature—often very effective, but full of traps. The course warns about the most famous one: don't compute the means on the full dataset, or the test set's answers leak into training features and your score is fake. Compute on the training set only.

But there's another trap the course doesn't mention, and its own demo data steps on it. The true price gaps are clear: downtown +50k, near suburb +10k, far suburb +0. The encodings from 200 samples? Downtown 228k—correct. But far suburb 190k, near suburb 188k—inverted. Near suburb should be 10k more expensive; the encoding makes it 2k+ cheaper.

Why? With ~67 samples per group, within-group sampling noise in average price is around 9,000—the same magnitude as the true 10,000 gap. One draw is enough to flatten or flip a small gap. Re-run with 20,000 samples and the order is restored: downtown 233k, near suburb 192k, far suburb 179k. Target encoding is sensitive to small-group noise; in high-cardinality settings with many categories and few samples each, its ordering isn't necessarily trustworthy. Textbooks warn you about leakage; guard against this too.

A note on its smoothing mechanism: the encoding formula weights the group mean against the global mean by sample size. A category with 1 sample gets 91% global mean in its encoding—rare categories barely receive their own signal. That's the price of overfitting protection; know it before you use it.

Text: count words, then weight them

Vectorizing text starts with counting words. Step two is TF-IDF: a word appearing often in this document but rarely in others gets high weight. For "large modern house with pool," "house" appears in 1 of 5 documents, weight 0.32; "with" appears in 3, weight only 0.10. Stop words automatically sink without a maintained list.

Two things worth knowing. First, under the course's formula, a word appearing in every document gets weight exactly zero—zero-discrimination words don't count, which is a nice, crisp property. Second, checking a hand-rolled version against scikit-learn is a shock: "house" scores 0.32 by hand, 2.61 from the library—8x apart. Neither is wrong; three conventions differ: sklearn's IDF adds 1 at the end as a floor, its TF doesn't divide by document length, and it applies a normalization by default. Matching the formula to the implementation is just the passing grade; whose conventions align with the ecosystem's is the real question when comparing implementations.

Polynomial features: a function that takes a parameter and does nothing

The course's polynomial_features function signature says degree=2. Passing 3 or 5 returns exactly the same list as degree 2—only squared and cross terms, no cubics. The parameter is accepted; the work isn't done. If you actually need x³, this function will silently hand you a pile of squares. Running third-party code with non-default arguments is the cheapest health check there is.

Feature selection: two filters, zero work done

More features isn't always better. Ten good features usually beat ten good features plus ninety noise features—noise gives the model room to "memorize coincidences" on the training set.

The course implements two filters, then demos them on its own data. The result is dramatic: the variance-threshold filter takes 7 columns in, outputs 7. The correlation filter: 7 in, 7 out. Nothing removed.

Why can't the variance filter remove anything? The area column has been z-scored—z-score's definition sets variance to 1. A filter using variance as its criterion, facing a matrix where variance was artificially unified to 1, can't be moved by a 0.01 threshold. Variance filtering is meant to catch "nearly constant columns"; it should work on the original scale. Feed it a column that's 99% one value (variance 0.00025) and it correctly drops it immediately—the filter isn't broken, it's positioned wrong.

Same for the correlation filter. The highest pairwise correlation among the seven columns is around 0.5 (natural negative correlation between one-hot columns), far from the 0.9 cutoff. What does real-world redundancy look like? The same quantity computed twice by two methods. Feed "area" and "log(area+1)" together: correlation 0.97, and the filter deletes on sight. This data is too clean to leave the filter any prey.

So although this section's demo output literally prints "7 features kept," the thing that actually did work is the mutual-information table beside it: area scores 0.3886 against "is high-priced," far ahead; age 0.036; all three neighborhood columns under 0.016; the missing indicator roughly zero. To watch feature selection actually act, feed "area" and "log(area)" together to the correlation filter—at a 0.9 threshold it will precisely remove the latter: not one column more, not one less.

---

Exercise: change target encoding to the leave-one-out version (each row's encoding computed from means excluding itself), and compare it to the naive version on train and test sets. It took 20,000 samples to stabilize the ordering of three neighborhoods—what would you see with 100 categories and two or three samples each?

*Source: Feature Engineering & Selection, ai-engineering-from-scratch (Rohit Ghumare, MIT), Phase 2 Lesson 8.*