logistic regression

Logistic regression predicts the probability of a binary outcome by taking a weighted sum of the input features and squashing it through the sigmoid function into a number between 0 and 1 — and despite the name, it is a classification method, not a regression method; "regression" refers to the linear combination of weights underneath, not to the kind of output it produces. The sigmoid curve itself traces back to Pierre François Verhulst's 1838 work modeling population growth, and David Cox adapted it into the modern statistical framework for classification in his 1958 paper "The Regression Analysis of Binary Sequences." One of logistic regression's most consequential real deployments is the Framingham Risk Score, developed from the Framingham Heart Study and formalized by Wilson and colleagues in 1998, which estimates a patient's 10-year cardiovascular disease risk directly from a logistic regression fit on factors like age, cholesterol, blood pressure, and smoking status — still used in clinics today precisely because every coefficient in it has a transparent medical meaning.

Every coefficient in a logistic regression has a concrete, multiplicative meaning in terms of odds, which is exactly what makes it so hard to replace in medicine and credit scoring. A coefficient of 0.7 on a "currently smokes" feature means that smoking multiplies the odds of the outcome by roughly e to the power of 0.7, which comes out to about 2.0 — smoking is associated with roughly double the odds of the modeled disease, holding every other factor constant, a statement a doctor can actually explain to a patient. This odds-ratio interpretation is unique to logistic regression's sigmoid-of-linear-combination structure; a random forest or neural network can be just as accurate, or more, but neither hands you a single number with that same direct, multiplicative meaning for each input variable.

L1 and L2 regularization do the same job here that they do everywhere else, with one extra benefit specific to logistic regression's interpretability. L2 (ridge) shrinks every coefficient toward zero without eliminating any of them, keeping all features in the model with reduced influence; L1 (lasso) can shrink some coefficients all the way to exactly zero, which amounts to automatic feature selection — a logistic regression model with L1 regularization doesn't just predict well, it also tells you which features turned out not to matter at all, a form of free diagnostic the regularization entry covers in the general case.

Logistic regression assumes its decision boundary is a straight line (or flat hyperplane) in feature space, and that assumption is also its most common point of failure. Data where the two classes form concentric rings, or interleave in a spiral, simply cannot be separated by any straight line no matter how the weights are tuned — the same limitation covered for support vector machines without a kernel, and the same reason feature engineering (manually constructing interaction terms, polynomial features, or ratios) is so often paired with logistic regression specifically to give the model a straight line to draw in a space where one actually exists.

Logistic regression's continued use in medicine, credit scoring, and insurance isn't inertia — it's that regulators and doctors in those fields often need to justify a prediction to a human being, and a coefficient with a direct odds-ratio interpretation is a far easier thing to put in front of an auditor or a patient than a decision made by a thousand-tree ensemble or a neural network's weights, which is exactly the trade-off interpretability-sensitive fields keep making even when a more complex model would score a few points higher on accuracy.