Random Forest

A random forest is an ensemble learning method that combines many randomized decision trees. Each tree is built from a randomized view of the training data, and the ensemble aggregates the trees' predictions. The usual formulation uses bootstrap samples of observations and a random subset of candidate features at each split. Classification forests combine class decisions or class-probability estimates; regression forests average numeric predictions. ^[1][2]

Random forests are used for supervised classification and regression, especially on tabular data. Randomization reduces dependence among the trees, while aggregation reduces the instability of any one tree. The method can model nonlinear effects and interactions without specifying them in advance, but it is not automatically interpretable, calibrated, fair, causal, or immune to overfitting. Those properties must be evaluated for the data, implementation, and loss function in question. ^[1][6][8]

Definition and scope

Breiman defined a random forest as a collection of tree predictors whose tree-specific random vectors are independent and identically distributed. His 2001 construction combined bootstrap aggregating with randomized feature selection during tree growth and analyzed the ensemble through the strength of individual trees and the correlation between them. ^[1]

The term is also used more broadly for forests that randomize observations, features, split points, tree structure, or some combination of these. Consequently, two libraries can both expose a “random forest” while differing in sampling, split criteria, probability calculation, missing-value handling, class weighting, and defaults. A precise description should therefore identify the implementation and its settings, not only the algorithm family. ^[6][10][20]

Random forests are distinct from a single randomized tree. Their prediction is an aggregate over the fitted trees, and their statistical behavior depends on both the individual learners and the way their errors co-vary. ^[1][6]

History

Several earlier methods contributed to the modern algorithm:

• Tin Kam Ho's 1995 random decision forests combined classifiers built in randomly selected subspaces. Her 1998 random subspace paper developed the idea of constructing decision forests from feature subspaces. ^[3][4]
• Yali Amit and Donald Geman described randomized tree construction for shape recognition in 1997, including random selection during node construction. ^[5]
• Leo Breiman introduced bagging in 1996. Bagging fits a predictor on each of several bootstrap training sets and combines the resulting predictions; Breiman emphasized its benefits for unstable learning procedures. ^[2]
• Breiman's 2001 paper joined these lines of work into the formulation generally associated with random forests and described out-of-bag estimation, variable-importance measures, proximities, and margin-based analysis. ^[1]

Later research established theoretical results for simplified forests and, under additional assumptions, for versions closer to Breiman's algorithm. The assumptions matter: for example, Scornet, Biau, and Vert proved consistency of Breiman's original algorithm in an additive regression setting, not for every possible distribution, tree rule, and implementation. ^[6][7]

How the algorithm works

Sampling and tree growth

For a training set with n observations and p input features, a common random-forest procedure is:

1. Draw a bootstrap sample of n observations for a tree.
2. At a node, randomly select m of the p features as split candidates.
3. Choose a split from that candidate set according to the tree's criterion.
4. Continue growing the tree until its stopping rules are met.
5. Repeat the process for the requested number of trees.
6. Aggregate the trees' predictions for a new observation. ^[1][20]

Sampling with replacement means that an observation can appear more than once in a tree's training sample. The probability that a particular observation is absent after n draws is

$$\left(1-\frac{1}{n}\right)^n,$$

which approaches 1/e, or about 36.8%, as n increases. Observations absent from a tree's bootstrap sample are that tree's out-of-bag observations. ^[1][2]

Bootstrap sampling is common but is not logically required by every method called a random forest. Some implementations permit subsampling without replacement or use the full sample for each tree. Those choices alter both diversity and the availability or interpretation of out-of-bag estimates. ^[6][10][20]

Randomized split selection

At each node, the tree searches only a random subset of input features. If a few variables are strong predictors, unrestricted bagged trees may repeatedly make similar upper-level splits and remain highly correlated. Restricting the candidate features can make trees less alike, although an overly small candidate set can also weaken each tree. The useful setting therefore depends on the signal, number and type of predictors, sample size, and chosen implementation. ^[1][6][10]

For classification, common split criteria include Gini impurity and entropy or log loss. For regression, squared-error reduction is common. These are implementation choices rather than parts of a single universal definition. Modern libraries may also offer absolute-error, Poisson, missing-value, monotonicity, or other options with their own constraints. ^[20][21][22]

Aggregation

In Breiman's classification formulation, each tree casts a class vote and the forest selects the plurality class. Some software instead averages the class-probability estimates produced by individual trees and then chooses the class with the highest mean probability. Scikit-learn documents the latter behavior, so it should not be described as literally identical to hard voting in every case. ^[1][21]

For ordinary regression forests, the ensemble prediction is the average

$$\widehat{f}(x)=\frac{1}{B}\sum_{b=1}^{B}T_b(x),$$

where B is the number of trees and T_b(x) is tree b's prediction. With standard regression-tree leaves that predict an average of training responses, this construction generally interpolates within observed response values rather than extrapolating beyond them. Specialized forest estimators can target other quantities. ^[1][15][22]

Why aggregation helps

A deep decision tree can change substantially after modest changes to its training data. Bagging reduces this variance by averaging many fitted trees. If identically distributed tree predictions have variance σ² and equal pairwise correlation ρ, the variance of their average is

$$\rho\sigma^2+\frac{1-\rho}{B}\sigma^2.$$

This idealized expression illustrates why both the number of trees and the dependence between trees matter: increasing B reduces the second term, while randomizing split candidates is intended to reduce correlation. Real tree predictions do not necessarily satisfy the equal-variance and equal-correlation assumptions, so the expression is an intuition rather than a complete performance guarantee. ^[1][6]

Breiman's classification analysis used a margin measuring the difference between support for the true class and the strongest alternative. It related generalization error to tree strength and correlation. This offers a design principle—seek useful trees whose errors are not too correlated—but the resulting bound does not promise that a particular forest will be accurate. ^[1][6]

More trees and overfitting

Breiman proved that, as the number of trees grows, the classification forest's generalization error converges almost surely to a limit. This rules out indefinite divergence caused solely by Monte Carlo variation from adding trees. It does not establish that finite-forest error must improve monotonically, that the limiting forest is well specified, or that other forms of overfitting cannot occur. ^[1][8]

Probst and Boulesteix showed that expected 0–1 classification error can be non-monotonic in the number of trees, whereas smoother losses such as mean squared error, Brier score, and log loss have different behavior under their analysis. In practice, more trees usually stabilize estimates but increase training time, prediction time, and memory use. The number should be large enough that the relevant validation or out-of-bag metric has stabilized, subject to computational constraints. ^[8]

Consistency results

Theoretical analysis is difficult because practical forests combine data-dependent split selection, random feature subsets, resampling, and deep recursive trees. Reviews distinguish simplified models from Breiman-style implementations. Results proved for purely random forests or for an additive regression model should not be presented as universal consistency theorems for all random-forest classifiers and regressors. ^[6][7]

Out-of-bag estimation

For each training observation, an out-of-bag prediction can be formed using only trees whose bootstrap samples excluded that observation. Aggregating these predictions over observations provides an internal estimate of predictive error without fitting a separate forest for every validation fold. The same out-of-bag mechanism can support permutation-based importance calculations. ^[1][20]

Out-of-bag error is useful, but it is not automatically unbiased in every finite-sample problem. Janitza and Hornung documented overestimation in settings including small samples, many predictors, balanced classes, and weak effects; they also found that related resampling estimates can exhibit similar behavior. Model selection based repeatedly on the same out-of-bag estimates can introduce additional selection bias. An untouched test set or appropriately nested resampling remains important when an unbiased final performance estimate is required. ^[9]

Out-of-bag scoring also depends on bootstrap sampling. If bootstrap sampling is disabled, the standard out-of-bag construction is unavailable. With too few trees, some observations may receive too few out-of-bag predictions for a stable estimate. ^[20][21][22]

Hyperparameters

Random forests have a reputation for reasonable default performance, but defaults differ among libraries and versions, and tuning can materially affect accuracy and computation. A 2019 review found that tuning can improve performance and treated parameters such as the number of candidate features, minimum node size, sampling scheme, sample fraction, and number of trees separately. ^[10]

Scroll sideways for more →

As of the cited scikit-learn 1.9 documentation, RandomForestClassifier defaults to 100 trees, bootstrap sampling, and the square root of the feature count at each split. RandomForestRegressor also defaults to 100 trees and bootstrap sampling but uses all features (max_features=1.0) by default. These values describe that release, not a universal prescription. ^[21][22]

Hyperparameters should be selected inside the training data, for example with cross-validation, nested resampling, or a carefully separated validation set. The final reported metric should be computed on data not used to choose settings. For grouped, temporal, spatial, or person-level data, the split strategy must respect the relevant dependence structure; a random row split can leak information even when the estimator itself is correctly implemented. ^[26]

Feature importance and interpretation

Impurity-based importance

Mean decrease in impurity adds the weighted impurity reductions attributed to a feature across tree nodes, then averages over the forest. It is inexpensive because the required quantities arise during fitting. It is also sensitive to how many potential split points a predictor offers. Strobl and colleagues demonstrated selection and importance bias involving scale of measurement and number of categories, so a high impurity importance is not by itself evidence that a variable has a uniquely important real-world role. ^[11]

Permutation importance

Permutation importance measures the change in a chosen score after values of one feature are shuffled. It describes reliance of a fitted model on that feature for the evaluated data and metric. It does not measure a feature's intrinsic worth, and it is not a causal effect. Correlated predictors can share or substitute for information, making marginal permutation results difficult to interpret. Conditional permutation schemes were developed to preserve correlation structure and answer a different question from marginal permutation. ^[12][23]

Permutation importance should be evaluated on held-out data when the purpose is to explain generalization behavior. If the model has little predictive skill on that data, its feature rankings are correspondingly weak evidence. Repeated permutations help quantify Monte Carlo variability. ^[23]

SHAP and other explanations

TreeSHAP algorithms can efficiently compute Shapley-style additive explanations for tree ensembles under specified feature-dependence and output conventions. Such values allocate the model's prediction relative to a reference expectation; they do not establish that changing a feature would cause the allocated change in the outcome. Different background distributions, dependence assumptions, output scales, and model versions can produce different explanations. ^[13]

Partial-dependence and individual-conditional-expectation plots can summarize fitted response patterns, but correlated features can force evaluation in sparse or unrealistic regions. Proximity measures—based on how often observations reach the same leaves—were part of Breiman's original software and paper, yet they too are model-derived similarities rather than ground-truth distances. ^[1][20]

Classification and probability estimates

For multiclass classification, a forest outputs one score per class and selects a class according to its aggregation rule. Class weights, decision thresholds, and sampling strategies can be used when errors have unequal costs, but accuracy alone may conceal poor performance on a rare class. Precision, recall, class-specific error, area-under-curve measures, and cost-sensitive metrics answer different questions and should be chosen before evaluating a system.

The fraction or mean probability associated with a class is not guaranteed to be a calibrated probability. Calibration must be checked on data separated from model fitting and tuning. Post-hoc calibration introduces another fitted stage and therefore requires its own validation design. ^[25]

Regression behavior

A standard regression forest estimates a conditional mean by averaging tree predictions. It can represent nonlinear and non-additive patterns but usually produces piecewise-constant predictions. Because ordinary leaf predictions are averages of observed responses, extrapolation beyond the training response range is limited. This can be a serious constraint for forecasting under trend or distribution shift. ^[22]

Squared-error splits emphasize mean prediction, while absolute-error and Poisson criteria target different node objectives in implementations that provide them. Heteroscedasticity, skew, censoring, and tail-risk questions may motivate variants that estimate distributions, quantiles, or survival functions instead of only a mean. ^[15][16][22]

Missing values, categorical variables, and preprocessing

Handling of missing and categorical values is not uniform across random-forest software. The cited scikit-learn 1.9 classifiers and regressors document native learning of a left-or-right route for missing values at each candidate split. If a feature had no missing values during training, a missing value encountered at prediction time follows the child with more training samples. That is an implementation behavior and should not be attributed to every random forest. ^[21][22]

Some implementations require categorical variables to be encoded; others provide categorical split rules. Naive integer encoding can impose an artificial order, while one-hot encoding can expand dimensionality and affect how candidate features and importance scores behave. The preprocessing pipeline must be fitted only on the training portion in each resampling split to avoid leakage. ^[26]

Feature scaling is generally less central for axis-aligned decision-tree splits than it is for distance-based or gradient-based models. It may still matter elsewhere in a pipeline, such as imputation, feature construction, dimensionality reduction, or a model comparison.

Computational characteristics

Trees can usually be trained and evaluated in parallel because, conditional on the training data and random draws, each tree is fitted independently. Practical scaling is constrained by memory bandwidth, tree size, data representation, serialization, and parallel overhead. Prediction latency grows with the number and depth of trees unless the model is compressed or execution is otherwise optimized. ^[20]

A forest can occupy substantially more memory than one tree. Fully expanded trees may store many nodes, particularly with small leaves, many observations, or high-dimensional sparse inputs. Implementations differ in numeric precision, sparse-matrix support, threading, and determinism. Benchmark results should therefore include software version, hardware, preprocessing, parameter settings, and the metric being measured.

Related tree ensembles

Scroll sideways for more →

No row is uniformly best. Gradient boosting can outperform random forests on some tabular problems but often requires different tuning and regularization. Random forests can be easier to parallelize and can provide out-of-bag estimates when bootstrap sampling is used. Comparisons should use the same data splits and metrics and include tuned, reproducible pipelines.

Extensions

Quantile regression forests retain information about training responses in relevant leaves to estimate a conditional distribution and its quantiles, rather than only a conditional mean. Meinshausen presented consistency results under stated assumptions and demonstrated prediction-interval applications. ^[15]

Random survival forests adapt the ensemble to right-censored time-to-event data. Ishwaran and colleagues defined a conservation-of-events principle, an ensemble cumulative-hazard estimator, and an out-of-bag prediction-error procedure for survival settings. ^[16]

Generalized random forests use forest-derived adaptive neighborhoods to estimate quantities identified by local moment equations. The framework includes applications such as heterogeneous treatment effects and instrumental-variable settings, but causal interpretation requires the relevant identification assumptions; randomization inside the estimator does not create causal identification. ^[17]

Other extensions address multivariate outcomes, unsupervised similarity construction, online learning, spatial data, or distributional prediction. These methods can differ enough that performance or theory for one should not be transferred automatically to another.

Empirical evidence

Random forests have performed strongly in broad tabular benchmarks, but benchmark conclusions are bounded by the included datasets, preprocessing, metrics, and search budgets. In a 2014 comparison of 179 classifiers from 17 families on 121 datasets, Fernández-Delgado and colleagues reported that random-forest variants were among the strongest families; the difference between the leading random forest and the second-ranked radial-basis-function support-vector machine was not statistically significant under the study's comparison. ^[18]

A 2022 benchmark of medium-sized tabular datasets compared tree-based methods with neural-network alternatives after applying selection criteria to 45 datasets. It found a persistent advantage for tree-based models on the study's “typical tabular” setting and investigated sensitivity to uninformative features, data orientation, and irregular target functions. The result does not imply that trees dominate neural networks for images, language, very large tables, every metric, or every data-generating process. ^[19]

Dataset-specific validation is more informative than a global ranking. Leakage, duplicate entities, temporal drift, site effects, label construction, and an unrealistic split can dominate the apparent difference between algorithms.

Software

Scikit-learn provides RandomForestClassifier and RandomForestRegressor in its ensemble module. Its documentation specifies probability averaging for classification, current defaults, native missing-value routing, parallel execution parameters, and warnings about impurity-based feature importance. ^[20][21][22]

The R package randomForest, described by Liaw and Wiener, provides an interface based on Breiman and Cutler's software for classification and regression. Its historical defaults and feature set should not be assumed to match scikit-learn or other libraries. ^[24]

Other ecosystems implement related forests in distributed, GPU, database, and specialized statistical packages. Reproducibility requires recording the package and version because names shared across implementations do not guarantee identical training or prediction semantics.

Strengths and limitations

Common strengths include:

• support for nonlinear effects and feature interactions without pre-specifying their form;
• applicability to classification and regression;
• comparatively little sensitivity to feature scaling;
• parallel training and prediction across trees;
• out-of-bag evaluation when the sampling design supports it; and
• strong empirical baselines on many tabular problems. ^{[1][10][18][20]}

Important limitations include:

• lower global transparency than a small decision tree;
• memory and latency costs for large forests;
• biased impurity importance and dependence-sensitive permutation importance;
• limited extrapolation in ordinary regression forests;
• probability estimates that may require calibration;
• sensitivity to leakage, distribution shift, class imbalance, and evaluation design;
• implementation-dependent treatment of missing and categorical data; and
• no automatic causal or fairness interpretation. ^{[9][11][12][13][22][23]}

Random forest is therefore best treated as a well-established model family and a strong empirical baseline, not as a guarantee of accuracy or a substitute for problem-specific validation.

See also

• Bagging
• Decision tree
• Decision forest
• Ensemble learning
• Feature importance
• Gradient boosting
• Machine learning
• Overfitting
• Supervised learning

References

1. ^Leo Breiman, “Random Forests,” *Machine Learning* 45, 5–32 (2001). doi.org/...A:1010933404324
2. ^Leo Breiman, “Bagging Predictors,” *Machine Learning* 24, 123–140 (1996). doi.org/...BF00058655
3. ^Tin Kam Ho, “Random Decision Forests,” *Proceedings of the Third International Conference on Document Analysis and Recognition* (1995). doi.org/...ICDAR.1995.598994
4. ^Tin Kam Ho, “The Random Subspace Method for Constructing Decision Forests,” *IEEE Transactions on Pattern Analysis and Machine Intelligence* 20(8), 832–844 (1998). doi.org/...34.709601
5. ^Yali Amit and Donald Geman, “Shape Quantization and Recognition with Randomized Trees,” *Neural Computation* 9(7), 1545–1588 (1997). doi.org/...neco.1997.9.7.1545
6. ^Gérard Biau and Erwan Scornet, “A Random Forest Guided Tour,” *TEST* 25, 197–227 (2016). doi.org/...s11749-016-0481-7
7. ^Erwan Scornet, Gérard Biau, and Jean-Philippe Vert, “Consistency of Random Forests,” *The Annals of Statistics* 43(4), 1716–1741 (2015). doi.org/...15-AOS1321
8. ^Philipp Probst and Anne-Laure Boulesteix, “To Tune or Not to Tune the Number of Trees in Random Forest,” *Journal of Machine Learning Research* 18(181), 1–18 (2018). jmlr.org/...17-269
9. ^Silke Janitza and Roman Hornung, “On the Overestimation of Random Forest's Out-of-Bag Error,” *PLOS ONE* 13(8), e0201904 (2018). doi.org/...journal.pone.0201904
10. ^Philipp Probst, Marvin N. Wright, and Anne-Laure Boulesteix, “Hyperparameters and Tuning Strategies for Random Forest,” *WIREs Data Mining and Knowledge Discovery* 9(3), e1301 (2019). doi.org/...widm.1301
11. ^Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn, “Bias in Random Forest Variable Importance Measures,” *BMC Bioinformatics* 8, 25 (2007). doi.org/...1471-2105-8-25
12. ^Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, and Achim Zeileis, “Conditional Variable Importance for Random Forests,” *BMC Bioinformatics* 9, 307 (2008). doi.org/...1471-2105-9-307
13. ^Scott M. Lundberg et al., “From Local Explanations to Global Understanding with Explainable AI for Trees,” *Nature Machine Intelligence* 2, 56–67 (2020). doi.org/...s42256-019-0138-9
14. ^Pierre Geurts, Damien Ernst, and Louis Wehenkel, “Extremely Randomized Trees,” *Machine Learning* 63, 3–42 (2006). doi.org/...s10994-006-6226-1
15. ^Nicolai Meinshausen, “Quantile Regression Forests,” *Journal of Machine Learning Research* 7, 983–999 (2006). jmlr.org/...meinshausen06a
16. ^Hemant Ishwaran, Udaya B. Kogalur, Eugene H. Blackstone, and Michael S. Lauer, “Random Survival Forests,” *The Annals of Applied Statistics* 2(3), 841–860 (2008). doi.org/...08-AOAS169
17. ^Susan Athey, Julie Tibshirani, and Stefan Wager, “Generalized Random Forests,” *The Annals of Statistics* 47(2), 1148–1178 (2019). doi.org/...18-AOS1709
18. ^Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim, “Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?” *Journal of Machine Learning Research* 15, 3133–3181 (2014). jmlr.org/...delgado14a
19. ^Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux, “Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?” *Advances in Neural Information Processing Systems* 35 (2022). papers.nips.cc/...Abstract-Datasets_and_Benchmarks
20. ^Scikit-learn developers, “Forests of Randomized Trees,” *scikit-learn 1.9 User Guide*. scikit-learn.org/...ensemble
21. ^Scikit-learn developers, “RandomForestClassifier,” *scikit-learn 1.9 API Reference*. scikit-learn.org/...nsemble.RandomForestClassifier
22. ^Scikit-learn developers, “RandomForestRegressor,” *scikit-learn 1.9 API Reference*. scikit-learn.org/...ensemble.RandomForestRegressor
23. ^Scikit-learn developers, “Permutation Feature Importance,” *scikit-learn 1.9 User Guide*. scikit-learn.org/...permutation_importance
24. ^Andy Liaw and Matthew Wiener, “Classification and Regression by randomForest,” *R News* 2(3), 18–22 (2002). cran.r-project.org/...Rnews_2002-3.pdf
25. ^Scikit-learn developers, “Probability Calibration,” *scikit-learn 1.9 User Guide*. scikit-learn.org/...calibration
26. ^Scikit-learn developers, “Common Pitfalls and Recommended Practices,” *scikit-learn 1.9 User Guide*. scikit-learn.org/...common_pitfalls