gradient boosting

Gradient boosting builds an ensemble one tree at a time, and unlike random forest's independent trees, every new tree in a boosted ensemble is trained specifically to predict the error the current ensemble is still making. Jerome Friedman formalized the general method in his 2001 paper "Greedy Function Approximation: A Gradient Boosting Machine," and the idea is concrete enough to trace through real numbers: starting from a naive prediction of 300,000 dollars for every house in a dataset, a house actually worth 500,000 dollars has a residual error of 200,000; the first small tree is trained not on house prices but on that residual, learns a rule like "four bedrooms predicts plus 150,000," and the ensemble's prediction updates to 300,000 plus a shrunken fraction of that tree's output — with a learning rate of 0.1, that's 300,000 plus 15,000, landing at 315,000, which cuts the error to 185,000 for the next tree to chip away at.

Start with a naive prediction
e.g. the mean target value for every row
Compute residuals
actual minus current prediction, for every row
Fit a small tree to the residuals
this tree's job is predicting the ERROR, not the target
Update: add shrunken tree output
prediction += learning_rate × this tree's output — see learning-rate
Final prediction
sum of the initial guess plus every shrunken tree

Boosting and bagging attack the exact same problem — one weak model isn't good enough — with opposite architectures. Random forest trains hundreds of trees independently and in parallel, each one blind to what the others are doing, then averages their votes to cancel out uncorrelated mistakes; gradient boosting trains trees strictly in sequence, where tree 50 cannot exist until tree 49's residuals are known, because each tree's entire purpose is correcting what came immediately before it. That sequential dependency is why gradient boosting is harder to parallelize across trees than random forest, and also why, tuned well, it usually reaches lower error on tabular data than bagging does — it's actively hunting the remaining error rather than just averaging away noise.

Parallel and Independent vs. Sequential and Corrective random forest (bagging) vs. gradient boosting — same goal, opposite architecture Random Forest (bagging) Tree 1 Tree 2 Tree 3 All three trained at once, independently — none sees what the others are doing Majority vote / average Easy to parallelize — reduces VARIANCE Each tree can train on a separate CPU core at the same time, with no coordination needed. Gradient Boosting (sequential) Tree 1 Tree 2 Tree 3 Tree 2 needs Tree 1's residuals. Tree 3 needs Tree 1 + 2's residuals. Weighted sum, not a vote Must train in order — reduces BIAS Each tree directly hunts whatever error is still left after every prior tree's correction. Same ensemble idea, opposite trade: parallel simplicity vs. sequential accuracy.

Three production libraries turned Friedman's algorithm into the dominant tool on tabular machine learning, and each won on a different axis. XGBoost, introduced by Chen and Guestrin's 2016 paper, added regularization and efficient handling of sparse data, and the paper itself reports that XGBoost was used in 17 of the 29 winning solutions on Kaggle in 2015 alone; LightGBM, released by Microsoft researchers in 2017, grows trees leaf-wise rather than level-wise and uses histogram-based splitting to train dramatically faster on large datasets; CatBoost, from Yandex the same year, handles categorical features natively without requiring one-hot or target encoding beforehand, trading some speed for less preprocessing work and less risk of the encoding leakage described in feature-engineering. All three are implementations of the same core algorithm above — residuals, shrinkage, sequential trees — competing on engineering rather than on the underlying idea.

The learning rate is gradient boosting's most consequential single hyperparameter, for exactly the shrinkage reason the house-price example shows. A learning rate of 1.0 would add each tree's full correction immediately, converging fast but overfitting to noise in the residuals; a small learning rate like 0.01 forces many more trees to reach the same total correction, but each one only nudges the prediction slightly, which is consistently found to generalize better — the standard practical trade is a lower learning rate paired with more trees and early stopping against a validation set, rather than a high learning rate and few trees.

Gradient boosting's real insight is treating "what is this model still getting wrong" as the training target for the next piece of the model, rather than training every piece on the original labels — which is also exactly why it is unforgiving of a mistake early in the sequence: an error in tree 3's residual calculation doesn't stay contained to tree 3, it propagates into what every subsequent tree is told to fix, which is the price paid for a method that actively hunts remaining error instead of just averaging over it.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account