linear regression

Linear regression fits a straight line through data by finding the single slope and intercept that minimize the sum of squared differences between the line and every actual point — the least squares method behind it was developed independently by Adrien-Marie Legendre (1805) and Carl Friedrich Gauss (who claimed priority back to 1795), and Gauss used it to one of history's most dramatic effects in 1801, predicting where the newly discovered asteroid Ceres would reappear from behind the sun after astronomers had lost track of it, from just a handful of observations. The word "regression" itself comes from an unrelated but equally concrete source: Francis Galton's 1886 study of parent and child heights, where he noticed that very tall parents tended to have children shorter than themselves, and very short parents tended to have children taller than themselves — height "regressing" back toward the population average rather than drifting to ever-more extreme values, the pattern the SVG below reproduces with real-looking numbers.

For small-to-moderate datasets, linear regression has an exact closed-form answer — no iteration required at all. The normal equations solve directly for the coefficients that minimize squared error in one step, which is why fitting a simple linear regression on a dataset with a few thousand rows is close to instantaneous; gradient descent, the iterative approach covered under learning-rate, only becomes necessary once the number of features or rows gets large enough that directly solving the normal equations' matrix inversion becomes computationally expensive. Both approaches converge to the identical solution on data that genuinely has a linear relationship — they're two different ways of reaching the same minimum, not competing methods with different answers.

The coefficients that come out of a fitted linear regression are directly interpretable, in plain units, which is the method's biggest practical advantage over almost anything more complex. A slope of 0.65 in a height-prediction model means literally "each additional inch of parent height predicts 0.65 additional inches of child height, holding everything else constant" — no translation layer, no feature-importance approximation, just the coefficient itself as the answer, which is exactly the same kind of direct interpretability covered under logistic regression's odds ratios, just without the sigmoid transformation in between.

Linear regression's assumptions are testable, and violating them doesn't just lower accuracy — it invalidates the statistical guarantees the method is built on. The method assumes the relationship really is linear, that the scatter of residuals is roughly constant across the range of predictions (homoscedasticity) rather than fanning out, and that the errors aren't correlated with each other; a residual plot that curves instead of scattering randomly, or that widens as predictions get larger, is the standard diagnostic for a broken assumption, and the usual fixes are transforming a variable (such as taking a logarithm), adding a polynomial term, or reaching for a different model entirely rather than trusting a line that clearly doesn't fit the shape of the data.

Ridge and lasso regression are linear regression with the exact same regularization techniques covered in general elsewhere, applied to this exact model. Adding an L2 penalty to the squared-error objective produces ridge regression, which shrinks coefficients to control overfitting without eliminating any of them; adding an L1 penalty produces lasso, which can zero out coefficients entirely and perform automatic feature selection — both are the same regularization mechanism from the regularization entry, just paired with the simplest possible underlying model rather than a neural network.

Gauss's Ceres prediction and Galton's height data are two centuries apart and have nothing to do with each other substantively, but they demonstrate the same thing: a straight line, fit by minimizing squared error, captures real structure in data precisely often enough that it remains the first model worth trying on almost any numeric prediction problem, and the first one worth ruling out before reaching for something that explains its predictions far less clearly.