normalization

Normalization, in the feature-preprocessing sense, is rescaling numeric inputs so that one feature's raw units don't silently dominate a model just because its numbers happen to be bigger — distinct from batch normalization and layer normalization, the activation-rescaling techniques inside a neural network's layers covered elsewhere on this site. A dataset with age ranging 0 to 100 and salary ranging 0 to 200,000 feeds a K-nearest-neighbors distance calculation where a 10,000-dollar salary gap swamps a 10-year age gap, even if age matters just as much to the actual problem, purely because salary's numbers are roughly 2,000 times larger. Scaling both features onto a comparable range before training is what stops that distortion, and the two standard ways to do it are handled, and validated, identically.

Standardization — z-score scaling — subtracts the mean and divides by the standard deviation, landing most values roughly between negative three and positive three with no fixed ceiling. It's the safe default for most algorithms because the mean and standard deviation aren't thrown off by a handful of outliers nearly as badly as a minimum or maximum value is, and it preserves zero, which matters for sparse data where most feature values genuinely are zero. Min-max normalization instead subtracts the minimum and divides by the full range, squeezing everything into a fixed zero-to-one interval — the right choice when an algorithm expects bounded input, such as image pixel values rescaled from the 0–255 range down to 0–1 before they reach a neural network.

Tree-based models — random forests, gradient-boosted trees — don't need any of this, and scaling them is wasted effort. A decision tree splits on one feature at a time by asking whether a value is above or below some threshold, so it never compares the magnitude of age against the magnitude of salary directly; multiplying every salary value by 1,000 doesn't change where the best split point falls, only its label. Distance-based methods (K-nearest-neighbors, support vector machines, k-means clustering) and gradient-based methods (neural networks, logistic regression, linear regression with regularization) are the ones that genuinely need scaled inputs, because both literally compute magnitudes across features and get skewed by whichever feature happens to use bigger raw numbers.

When outliers are severe enough to distort even the mean and standard deviation, robust scaling swaps in the median and interquartile range instead. Standardization still shifts somewhat when a handful of extreme values are present — a single erroneous age entry of 500 pulls the mean and standard deviation along with it — and robust scaling sidesteps this by centering on the median and dividing by the spread between the 25th and 75th percentiles, both of which are far less sensitive to a handful of extreme values than a mean or a min/max range. scikit-learn's `RobustScaler` implements exactly this trade: slightly less common as a default, but the right tool whenever a dataset is known to carry data-entry errors or genuinely extreme values that shouldn't be allowed to compress everything else.

The leakage rule here is identical to the one that governs cross-validation and a held-out validation set, because it's the same mistake wearing a different name: whatever mean, standard deviation, minimum, or maximum gets used to scale a feature has to be computed from the training data alone, then applied unchanged to the test data, or the test set's own statistics quietly inform the transformation that produces its own score — turning an honest measurement into one that was partly computed from the answer key.