principal component analysis

Principal component analysis finds the directions in a dataset along which the data actually varies, ranked from most to least, by computing the eigenvectors of the data's covariance matrix and ordering them by their eigenvalues — the eigenvector with the largest eigenvalue is the first principal component, the single direction along which the data spreads out the most. One of the clearest demonstrations of what this buys you is genetic: Novembre and colleagues' 2008 Nature paper analyzed roughly 200,000 genetic markers across 1,387 Europeans, and when they plotted each person's position on just the first two principal components of that genetic data, the resulting scatter plot reconstructed a recognizable map of Europe — no geographic coordinate was ever given to the algorithm, yet the two axes PCA found on its own aligned closely with north-south and east-west geography. The mechanism behind that result runs in a fixed sequence.

Each principal component comes with a number telling you exactly how much it's worth keeping. An eigenvalue divided by the sum of all eigenvalues gives that component's explained variance ratio — the fraction of the dataset's total spread captured along that one direction — and plotting these ratios in descending order produces a scree plot, the standard tool for deciding how many components to keep: when explained variance drops off sharply after the first few components and then flattens into a long, low tail, that elbow is usually where the real signal ends and noise begins. What makes the Europe result striking is exactly this number: each of those first two genetic principal components individually explained a small fraction, often under a percent, of the total genetic variance in the dataset, yet that sliver was precisely the sliver that encoded geography — a reminder that "small variance explained" does not mean "unimportant," only "not the dominant source of spread."

The other famous demonstration of the same math is faces, not genomes. Turk and Pentland's 1991 Eigenfaces method computed the principal components of a dataset of face images and showed that a new face could be reconstructed to a reasonable approximation using only a few dozen of those components — roughly 40 eigenfaces captured the bulk of the variance present across a face database — turning a high-dimensional image-matching problem into one of comparing a short list of numbers, which was fast enough to make real-time face recognition practical on early-1990s hardware. The same insight that makes Eigenfaces work (most of a face's variation lives in a small number of directions) is exactly why PCA shows up as a preprocessing step before slower non-linear methods, as covered in dimensionality-reduction.

PCA's sensitivity to the scale of each feature means skipping standardization quietly breaks the whole method. Because PCA chases directions of maximum variance, a feature measured in units that happen to produce bigger numbers — income in dollars next to age in years — will dominate the first principal component for no reason other than its units, exactly the same distortion covered under normalization; this is why PCA implementations in practice are almost always run on standardized (z-scored) features rather than raw ones, and why skipping that step is one of the most common silent mistakes in applying PCA to real data.

The genetics result is the cleanest illustration of what PCA actually promises: it does not know what geography is, does not know what a face is, and has no concept of meaning at all — it only knows how to find the directions along which a dataset spreads out the most, and hands you those directions with no guarantee they will correspond to anything interpretable. That they turned out to trace the outline of a continent in one case, and the structure of a human face in another, is not PCA understanding geography or faces — it's a reminder that the dominant axes of variation in real-world data are very often not arbitrary, and PCA is simply the tool honest enough to show you whatever axis that happens to be.