ml-003 · Random forest vs gradient boosting — Concepts

The big idea

Random forests and gradient-boosted trees are both ensembles of decision trees, but they combine trees in opposite ways. A random forest grows many deep trees independently on bootstrap samples and averages them — the averaging cancels out each tree’s variance, so the forest is hard to overfit by adding trees and works well straight out of the box. Gradient boosting grows shallow trees sequentially, each one fit to the residual errors of the ensemble so far, scaled by a learning rate — this attacks bias directly and usually wins on tabular benchmarks, but the learning rate and the number of rounds interact, so boosting can overfit and needs a validation set (or early stopping) to know when to stop. This practice makes both behaviors visible on one synthetic regression task.

Definitions

Bagging (bootstrap aggregating): fit the same model class on many bootstrap resamples of the training data and average the predictions. Reduces variance; does not reduce bias.
Random forest: bagging over deep decision trees, plus random feature subsetting at each split so the trees are decorrelated and the average helps more.
Boosting: build an additive model in stages; each new weak learner is fit to the current residuals (more precisely, the negative gradient of the loss) and added with a small step size.
Learning rate (eta / shrinkage): the multiplier applied to each new tree’s contribution. Smaller values need more rounds but usually generalize better.
n_estimators / nrounds: the number of trees in the ensemble. For RF it is a convergence knob (more is never worse, only slower); for boosting it is a capacity knob (more can overfit).
Early stopping: during boosting, monitor a validation metric and stop once it has not improved for a fixed number of rounds (“patience”), keeping the best iteration.
Feature importance: a per-feature score of how much the model relied on it — impurity-based for forests, split-gain-based for XGBoost. Both are heuristics, and they can disagree.

Why it matters

Tree ensembles are still the strongest default for tabular prediction problems, and the RF-vs-boosting choice comes up in almost every applied project: RF when you want a robust baseline with near-zero tuning, boosting when you want the last few points of accuracy and can afford to tune. The n_estimators/learning-rate interaction and early stopping are also the canonical example of why “more capacity” means different things in different ensemble families — a concept that transfers to neural network training (step size, epochs, early stopping) almost unchanged.

Pitfalls and misconceptions

“More trees overfit a random forest.” No — RF validation error flattens out as trees are added; extra trees only cost compute. Overfitting in RF comes from other knobs (tree depth, too-small leaves), not tree count.
“The default learning rate is fine.” Boosting with a large learning rate can bottom out early and then degrade; a small one can look terrible simply because you did not run enough rounds. Judge (learning rate, rounds) as a pair, never each alone.
Tuning the number of boosting rounds on the training loss — it decreases forever by construction. You need a held-out set.
Reading feature importances as causal effects, or expecting the two models’ importances to match exactly. Interactions get split between the participating features, and impurity vs gain measure different things.
Reporting the early-stopping validation RMSE as an unbiased estimate of generalization error — the validation set chose the number of rounds, so it is mildly optimistic; use a third split or CV for honest reporting.

Mental model

	Random forest	Gradient boosting
Trees are built	independently, in parallel	sequentially, on residuals
Individual trees	deep, low bias, high variance	shallow “weak learners”
Ensemble attacks	variance (by averaging)	bias (by stagewise fitting)
More trees	converges, never hurts accuracy	adds capacity, can overfit
Key knobs	mostly none (depth, mtry)	learning rate x rounds, depth
Failure mode	rarely dramatic	quiet overfitting past the optimum

A useful analogy: a random forest is a committee poll — many opinionated experts formed their views independently, and you average them. Boosting is an editing pass — each new editor fixes what the previous draft still got wrong, and the learning rate is how bold each edit is allowed to be.

Check your understanding

Q. Why does adding trees to a random forest not cause overfitting, while adding rounds to gradient boosting can?

Each RF tree is an independent, identically distributed draw given the data; averaging more draws only tightens the estimate of the ensemble’s expected prediction, so validation error flattens rather than rising. In boosting, every added tree changes the function being fit — the ensemble keeps climbing down the training loss — so past some point the extra rounds start fitting noise and validation error turns back up (or plateaus while training error keeps falling).

Q. You halve XGBoost’s learning rate and validation RMSE gets much worse. What is the most likely explanation and the fix?

You kept the number of rounds fixed. Half the step size with the same number of steps travels roughly half as far down the loss — the model is now underfit, not worse in general. Increase the rounds (roughly doubling them) or, better, set a generous round budget and let early stopping on a validation set choose the stopping point for the new learning rate.