# Gradient Boosting

> Source: https://aiwiki.ai/wiki/gradient_boosting
> Updated: 2026-07-31
> Fact-checked: 2026-07-31
> Categories: Machine Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Gradient Boosting." aiwiki.ai, 31 Jul 2026. https://aiwiki.ai/wiki/gradient_boosting
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Gradient boosting** is a supervised [machine learning](https://aiwiki.ai/wiki/machine_learning) method that constructs an additive prediction function in stages. At each stage it fits a new base learner to a direction that reduces a chosen [loss function](https://aiwiki.ai/wiki/loss_function), then adds that learner to the existing ensemble. The base learners are usually small [decision trees](https://aiwiki.ai/wiki/decision_tree), but the framework is defined in function space and is not limited to trees. It can be used for [regression](https://aiwiki.ai/wiki/regression), [classification](https://aiwiki.ai/wiki/classification), ranking, and other tasks for which a suitable optimization objective can be specified.[1][2]

Jerome H. Friedman's 2001 formulation connected stagewise additive modeling with steepest descent in function space and gave algorithms for squared-error, absolute-error, Huber, and multiclass logistic losses.[1] Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean independently described boosting as gradient descent in function space in work presented at the 1999 Neural Information Processing Systems conference.[2] Modern tree implementations such as [XGBoost](https://aiwiki.ai/wiki/xgboost), [LightGBM](https://aiwiki.ai/wiki/lightgbm), and [CatBoost](https://aiwiki.ai/wiki/catboost) add engineering and statistical techniques for scalable split finding, sparse inputs, categorical features, missing values, and regularization.[10][11][12]

Gradient boosting is especially prominent for structured or tabular prediction, but no model family is uniformly best. Two broad benchmark studies found that boosted-tree methods were strong on many medium-sized tabular datasets, while also finding datasets on which [neural networks](https://aiwiki.ai/wiki/neural_network) matched or exceeded them. Those findings are bounded by the datasets, preprocessing, algorithms, and tuning budgets in the studies; they do not establish a universal ranking of methods.[14][15]

## Intuitive explanation

Suppose a model makes one initial prediction for every example. The prediction will be too high for some examples and too low for others. A second, small model is trained to make a correction. A third model then targets what remains wrong after the first correction, and the process continues. The final prediction is the initial prediction plus all of the learned corrections.

For squared-error regression, those correction targets are ordinary residuals. For other losses they are negative derivatives of the loss with respect to the current predictions, often called pseudo-residuals. This is the source of the word "gradient" in gradient boosting: the ensemble is updated in a direction intended to decrease the empirical loss.[1][9]

This explanation has two important limits. First, a later tree does not merely memorize a list of mistakes. It learns a function of the input variables that approximates the current negative-gradient values. Second, adding more correction stages does not guarantee better performance on unseen data. Tree complexity, step size, stochastic sampling, stopping time, and the validation design all affect generalization.[8][31]

## Historical development

### From weak learning to adaptive boosting

Boosting developed from questions about whether a learning procedure that performs only slightly better than chance could be converted into a highly accurate procedure. Robert Schapire proved the equivalence of weak and strong learnability in the probably approximately correct learning model in 1990.[3] Yoav Freund and Schapire subsequently proposed AdaBoost, which repeatedly invokes a base learner while changing the distribution of weight over training examples and combines the resulting hypotheses.[4]

AdaBoost is related to gradient boosting but is not synonymous with it. [AdaBoost](https://aiwiki.ai/wiki/adaboost) specifies a particular adaptive reweighting procedure and is associated with exponential loss in statistical interpretations. Gradient boosting is a more general stagewise optimization framework in which the loss determines the pseudo-residuals.[8] The 1997 journal version of Freund and Schapire's paper received the 2003 Godel Prize.[5]

Leo Breiman studied adaptive resampling and combining under the name "arcing." His 1997 technical report "Arcing the Edge" analyzed an explanation based on increasing the minimum margin and compared it with alternative accounts of AdaBoost's behavior.[6] It is more precise to treat this work as part of the intellectual path connecting boosting, margins, and optimization than to attribute Friedman's complete gradient-boosting algorithm to Breiman.

### Functional gradient formulations

Mason and colleagues represented a classifier as a linear combination of functions and characterized boosting algorithms as gradient descent in function space. Their paper supplied a general view in which an inner product determines a functional gradient and a base learner is selected to align with the descent direction.[2]

Friedman's 2001 paper began from the problem of estimating a function that minimizes expected loss. It connected forward stagewise additive expansions with numerical optimization and replaced an unrestricted functional step with a fitted base learner. The paper then specialized the construction to several regression and classification losses and developed tree-specific updates, shrinkage, interaction analysis, and interpretation tools.[1]

These accounts overlap but emphasize different aspects. Mason and colleagues focused on a general functional-gradient characterization of boosting algorithms. Friedman developed a statistical estimation framework and detailed algorithms in which regression trees can serve as base learners. Statements that gradient boosting was invented on a single date obscure this parallel development. Friedman's underlying technical report circulated in 1999, and the peer-reviewed paper appeared in 2001.[1][2]

### Stochastic gradient boosting

Friedman extended the procedure in 2002 by fitting each successive learner on a random subsample drawn without replacement. His paper called the method stochastic gradient boosting and reported improved computational speed and, in its experiments, improved prediction accuracy relative to the deterministic version.[7] The result is empirical, not a guarantee that every subsampling fraction improves every dataset.

Later systems introduced additional sampling choices, including feature subsampling and sampling informed by gradient magnitude. Those mechanisms should not all be collapsed into Friedman's original definition. In particular, LightGBM's Gradient-based One-Side Sampling retains examples with large absolute gradients and samples from the remaining examples with a correction to the information-gain calculation.[11]

## Mathematical framework

Let a training set contain pairs $$(x_i,y_i)$$ for $$i=1,\ldots,n$$, and let $$L(y,F(x))$$ measure the loss incurred by prediction $$F(x)$$. Empirical risk minimization seeks a function with low average training loss:

$$
\hat{F}=\arg\min_F \sum_{i=1}^{n} L(y_i,F(x_i)).
$$

Gradient boosting restricts the solution to a stagewise additive expansion:

$$
F_M(x)=F_0(x)+\sum_{m=1}^{M}\nu \rho_m h_m(x),
$$

where $$F_0$$ is an initial constant or function, $$h_m$$ is the base learner selected at stage $$m$$, $$\rho_m$$ is a stage multiplier, and $$\nu$$ is a shrinkage factor. Not every implementation exposes a distinct $$\rho_m$$ and $$\nu$$, and tree-specific algorithms may optimize a different value in each terminal region.[1]

### Generic stagewise algorithm

A common form of the procedure is:

1. Choose an initial prediction that minimizes the empirical loss over constants:

   $$
   F_0(x)=\arg\min_{\rho}\sum_{i=1}^{n}L(y_i,\rho).
   $$

2. For boosting rounds $$m=1,\ldots,M$$, compute the negative gradient at every training prediction:

   $$
   r_{im}=-\left[
   \frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}
   \right]_{F=F_{m-1}}.
   $$

3. Fit a base learner $$h_m(x)$$ to the pairs $$(x_i,r_{im})$$.

4. Optionally perform a line search for a multiplier:

   $$
   \rho_m=\arg\min_{\rho}\sum_{i=1}^{n}
   L\left(y_i,F_{m-1}(x_i)+\rho h_m(x_i)\right).
   $$

5. Update the ensemble:

   $$
   F_m(x)=F_{m-1}(x)+\nu\rho_m h_m(x).
   $$

The fitted learner is generally only an approximation to the unrestricted negative-gradient vector. Consequently, this procedure is not identical to ordinary [gradient descent](https://aiwiki.ai/wiki/gradient_descent) on a fixed finite-dimensional parameter vector. It is a greedy optimization over the functions available in the base-learner class.[1][2]

### Squared-error example

For

$$
L(y,F)=\frac{1}{2}(y-F)^2,
$$

the negative derivative with respect to $$F$$ is

$$
-\frac{\partial L}{\partial F}=y-F.
$$

The pseudo-residual is therefore the ordinary residual. The initial constant is the sample mean, and each new regression learner is fitted to the current residuals. This special case provides a useful intuition, but the phrase "fit the residuals" is not a complete definition of gradient boosting because other losses produce different working responses.[1][9]

### Small regression example

Consider four observations with targets $$2,4,7,7$$ and use squared error. The constant that minimizes the initial loss is the mean, $$F_0=5$$. The first pseudo-residual vector is therefore

$$
(-3,-1,2,2).
$$

Suppose a one-split regression tree separates the first two observations from the last two. Its fitted leaf values for the residuals are $$-2$$ and $$2$$. With a learning rate of $$0.25$$ and no additional line-search multiplier, the predictions after one round are $$4.5$$ for the first group and $$5.5$$ for the second. The new residuals are $$(-2.5,-0.5,1.5,1.5)$$.

This toy calculation illustrates the stagewise update but not a complete training implementation. A real tree chooses a split from input features using an objective-dependent gain calculation. Modern systems may use gradients and Hessians, regularize leaf weights, sample rows or columns, and approximate split candidates. Also, the two observations within a leaf receive the same correction even though their residuals differ; the tree base learner constrains which directions the optimizer can take.[1][10]

### Trees and terminal-region updates

With a $$J$$-leaf regression tree, the base learner partitions the input space into disjoint regions $$R_{1m},\ldots,R_{Jm}$$:

$$
h_m(x)=\sum_{j=1}^{J} b_{jm}\mathbf{1}(x\in R_{jm}).
$$

Friedman's tree algorithm fits the partition to the pseudo-residuals and then finds a loss-minimizing update for each terminal region:

$$
\gamma_{jm}=\arg\min_{\gamma}
\sum_{x_i\in R_{jm}}L\left(y_i,F_{m-1}(x_i)+\gamma\right).
$$

The update is

$$
F_m(x)=F_{m-1}(x)+\nu
\sum_{j=1}^{J}\gamma_{jm}\mathbf{1}(x\in R_{jm}).
$$

This region-specific form is one reason that a generic single-step formula should not be read as the exact implementation of every gradient-boosted tree package. A separate AI Wiki article covers the tree-specific topic in more detail: [gradient boosted decision trees](https://aiwiki.ai/wiki/gradient_boosted_decision_trees_gbt).

## Loss functions and prediction targets

The objective defines both the statistical target and the pseudo-residuals. A loss may be smooth everywhere, differentiable almost everywhere with a chosen subgradient at a kink, or handled through a package-specific approximation. It is therefore inaccurate to say without qualification that gradient boosting supports "any differentiable loss" while also listing absolute loss, which is not differentiable where its residual is zero.

| Objective | Typical target | Negative-gradient behavior | Important qualification |
|---|---|---|---|
| Squared error | Conditional mean | Ordinary residual $$y-F(x)$$ | Large residuals receive quadratically increasing loss |
| Absolute error | Conditional median | Sign of the residual away from zero | A subgradient convention is required at zero |
| Huber loss | Robust location estimate | Residual near zero; clipped influence for large residuals | The transition threshold affects robustness and efficiency |
| Quantile loss | Conditional quantile | Asymmetric, piecewise-constant working response | The selected quantile must be specified |
| Binary logistic loss | Class probability through a link function | Difference related to observed label minus fitted probability | Prediction scale and label convention matter |
| Multinomial logistic loss | Multiclass probabilities | One working response per modeled class | Implementations differ in parameterization |
| Pairwise or listwise ranking objective | Ordering within query groups | Depends on document pairs, labels, and metric weighting | Query grouping and evaluation metric are part of the problem definition |

Friedman's original paper explicitly derived squared-error, least-absolute-deviation, Huber, and logistic procedures.[1] Later libraries expose additional built-in objectives. Some also allow user-defined objectives, but such interfaces impose mathematical and implementation constraints. XGBoost, for example, documents assumptions including smoothness and twice differentiability for custom objectives, additive per-row losses, unbounded score ranges, and convexity for its standard Hessian-based treatment.[36]

A loss should not be chosen solely because a library offers it. It encodes the error tradeoff. Squared error targets a conditional mean under the usual interpretation; absolute loss targets a median; quantile losses target selected conditional quantiles; and a classification loss operates on scores or probabilities through a link. Evaluation metrics can also differ from training objectives.

### Probability estimates and calibration

Optimizing logistic loss does not ensure that predicted probabilities are calibrated for a new population. Calibration is a relationship between predicted probabilities and observed frequencies, and it can be affected by sampling, class weighting, distribution shift, regularization, and tuning. Reliability diagrams and proper scoring rules can be used to assess it. If post-hoc calibration is applied, the calibrator must be fitted on data independent of the predictions used to train the base model; cross-validation is one way to obtain such predictions.[32]

### Probabilistic extensions

Standard point-prediction boosting estimates a conditional location, class score, probability, or quantile rather than a complete conditional distribution. Natural Gradient Boosting, or NGBoost, is a distinct probabilistic framework that fits parameters of a chosen predictive distribution using natural gradients and a proper scoring rule. It illustrates how boosting ideas can be extended to distributional prediction, but it should not be described as an automatic property of every gradient-boosting model.[35]

## Controlling model capacity

Gradient boosting can continue to reduce training loss after validation performance has stopped improving. Regularization therefore concerns the whole path of models, not merely the complexity of one tree.[8]

### Learning rate and number of rounds

Shrinkage multiplies each stage update by $$0<\nu\leq1$$. A smaller learning rate usually requires more boosting rounds to reach a similar training loss. Friedman's experiments found benefits from shrinkage in the settings studied, but there is no dataset-independent optimal range and no theorem that a smaller rate always gives better test performance.[1]

The number of rounds and learning rate must be considered together. A comparison that changes one while holding an unsuitable value of the other fixed can be misleading. More rounds also increase model size and prediction cost.

### Tree size

Depth, leaf count, minimum child weight or sample count, and minimum split gain constrain different aspects of a tree. A shallow depth limits the longest decision path. A leaf limit bounds terminal regions but can still produce an unbalanced tree. Minimum-observation and Hessian constraints restrict splits supported by little effective data. Parameter names that sound alike do not necessarily have identical semantics across libraries.[19][23][25]

Friedman related the number of terminal nodes to the order of interactions a tree can represent in his analysis, but this does not turn leaf count into a direct measurement of the interactions learned from finite data.[1] The useful size of a base tree depends on sample size, noise, feature representation, objective, and interactions.

### Row and feature sampling

Friedman's stochastic procedure samples rows without replacement at each round.[7] XGBoost, LightGBM, and CatBoost expose additional row- and feature-sampling controls, with semantics that vary by tree method and device.[19][23][25] Sampling can lower computation and change variance or dependence between trees, but it can also remove informative rare cases. The sampling design deserves particular scrutiny for imbalanced labels, grouped observations, and small datasets.

### Penalties and constrained structures

Modern implementations may penalize leaf weights, the creation of new leaves, or both. XGBoost's tree objective includes a tree-complexity term and second-order approximations; its parameters include L1 and L2 regularization of weights.[10][19] These penalties are features of that implementation, not defining properties of every gradient-boosting algorithm.

Some libraries also support monotonic constraints. A monotonic constraint directs the fitted score to be nondecreasing or nonincreasing with respect to a selected feature while other features are held fixed. XGBoost warns that, with histogram methods, constrained trees may become unnecessarily shallow and suggests increasing `max_bin` where appropriate.[37] A constraint can encode domain knowledge, but it does not make a causal claim and does not correct confounding or data leakage.

### Early stopping

Early stopping selects a boosting iteration by monitoring performance on validation data. A patience parameter allows training to continue for a specified number of non-improving rounds. Because the same validation trajectory is repeatedly consulted, the selected score is part of model selection rather than a final independent test estimate. An untouched test set or a suitable nested procedure is needed when an unbiased final comparison is important.[31]

Implementations differ over which dataset and metric control stopping, whether the best iteration is restored automatically, and how multi-metric evaluation is handled. These details should be checked in the documentation for the exact version in use.

### Dropout of trees

DART modifies multiple additive regression trees by randomly dropping existing trees while fitting a new one, then normalizing contributions. The authors proposed it in response to the possibility that early trees can dominate later corrections in a standard additive ensemble.[13] DART is a specific variant with its own sampling and normalization behavior; it is not the same operation as dropout in a [neural network](https://aiwiki.ai/wiki/neural_network).

## Efficient tree construction

The boosting rounds are sequential because round $$m$$ depends on predictions from earlier rounds. Work within a round, including histogram construction and split evaluation, can nevertheless be parallelized or distributed.

### Exact and approximate split finding

An exact greedy tree algorithm examines sorted feature values to identify candidate splits. Approximate algorithms restrict the candidate set, often through quantile sketches or histograms. XGBoost's paper described exact and approximate split algorithms, a weighted quantile sketch for weighted data, sparsity-aware split finding, cache-aware access, and out-of-core computation.[10]

XGBoost's current documented `auto` tree method selects `hist`, while `exact` remains a separate option. Its growth policy options include `depthwise`, which splits nodes closest to the root first, and `lossguide`, which selects the leaf with greatest loss change first.[19] These are current library semantics, not features of Friedman's original algorithm.

### Histogram algorithms

A histogram method maps continuous feature values into a limited number of bins. At a node it accumulates gradient statistics, and where required Hessian statistics, by bin. It then scans the bin boundaries rather than every distinct feature value. This reduces the number of candidate thresholds and can improve cache use and memory consumption.[10][11][22]

The claim that histogram training changes all split computation from $$O(nd)$$ to $$O(Bd)$$ is incomplete. Building a histogram still requires processing the observations assigned to a node unless a reusable or subtraction-based histogram is available. Binning reduces the threshold scan to the number of bins, and histogram subtraction can derive one child's histogram from the parent and the other child. Complexity depends on the algorithm, data representation, tree topology, reuse strategy, and parallel hardware.[11][22]

Scikit-learn introduced experimental histogram-based gradient-boosting estimators in version 0.21, stating in its changelog that they could be orders of magnitude faster than the older exact estimators for tens of thousands of samples or more.[33] Native categorical support was added in version 0.24, not version 1.0.[34] The current API uses at most 255 bins for each non-missing feature and reserves an additional bin for missing values.[28]

## Major implementations

The libraries below share a stagewise tree-boosting lineage but are not interchangeable. Defaults, objective definitions, categorical handling, missing-value behavior, determinism, and supported devices can change across versions. Reproducible work should record the package version and material parameters.

### XGBoost

The 2016 XGBoost paper presented a scalable tree-boosting system with a regularized objective, sparsity-aware learning, weighted quantile sketching, column-oriented data blocks, cache-aware access, and out-of-core algorithms.[10] Its competition evidence is historical and precisely scoped: the authors counted 17 of 29 winning solutions posted on the Kaggle blog during 2015 as using XGBoost, and reported that every top-10 team in the KDD Cup 2015 used it. Those counts do not show that XGBoost wins all contemporary tabular tasks.

Current XGBoost documentation includes CPU and GPU histogram training, row and column subsampling, categorical splits, monotonic constraints, and ranking objectives.[19][20][21] Native categorical support requires the caller to provide categorical data in a supported representation and enable it. The documentation distinguishes one-hot encoding from partition-based categorical splits and notes that saving categorical models requires JSON or UBJSON rather than an older binary model format.[20]

XGBoost learns a default branch for missing values in its sparsity-aware algorithm.[10] This means that missing entries can be routed without imputing a numeric value first; it does not mean that arbitrary patterns of missingness are harmless or that a missing-value mechanism is understood.

### LightGBM

The LightGBM paper introduced two named techniques. Gradient-based One-Side Sampling retains examples with large gradients while sampling examples with small gradients, with a correction when calculating information gain. Exclusive Feature Bundling combines sufficiently sparse, mostly mutually exclusive features to reduce the effective feature dimension.[11]

The authors reported speedups of up to more than 20 times over the compared conventional GBDT procedures while obtaining nearly the same accuracy in their experiments.[11] The phrase "up to" and the experimental comparison set are essential. It is not a general promise for every dataset, machine, or competing implementation.

LightGBM uses histogram-based algorithms and grows trees leaf-wise, choosing the leaf with the largest loss reduction. Its documentation warns that leaf-wise growth may overfit when data are limited and identifies `max_depth`, `num_leaves`, and minimum-data constraints as relevant controls.[22][23] It can find categorical splits directly from integer-coded categories using a partitioning method rather than requiring a full one-hot expansion.[22]

### CatBoost

CatBoost was designed around two related sources of prediction shift discussed by its authors: target-statistic construction for categorical variables and gradient estimates computed from models that have already observed the same examples. The paper proposed ordered target statistics and ordered boosting based on permutations of the training data.[12]

CatBoost's official documentation advises supplying categorical features in their original form rather than one-hot encoding them during preprocessing. The library constructs numeric features from categorical values and combinations according to its training mode and parameters.[24] This is not equivalent to saying that CatBoost needs no data-quality work: category spelling, unseen categories, leakage, identifier-like variables, and train-serving consistency remain relevant.

The default tree-growing policy is symmetric: nodes at the same depth use the same split. CatBoost also documents `Depthwise` and `Lossguide` policies, with restrictions on some analysis and export functions for nonsymmetric trees.[25] Its missing-value modes include `Forbidden`, `Min`, and `Max`; `Min` is the default for numerical features, and the algorithm guarantees consideration of a split separating missing values from other values.[26]

### Comparison boundaries

| Question | XGBoost | LightGBM | CatBoost |
|---|---|---|---|
| Distinctive primary-paper emphasis | Regularized scalable system, sparse-aware splitting, weighted sketch, systems optimizations | GOSS and EFB for efficient large-scale training | Ordered boosting and ordered categorical statistics |
| Commonly documented growth choices | Depthwise or lossguide | Leaf-wise best-first with optional depth constraint | Symmetric by default; depthwise and lossguide alternatives |
| Categorical path | One-hot or partition-based native splits when categorical support is enabled | Integer-coded categorical splits using category partitions | Ordered numeric statistics and feature combinations |
| Missing numerical values | Learned default direction in tree splits | Missing-value handling documented in the tree learner | Configurable missing-value modes |
| Important reproducibility detail | Tree method, device, categorical model format, sampling semantics | Leaf count, depth, binning, categorical codes, sampling mode | Boosting type, growth policy, permutations, categorical columns |

This table describes selected documented mechanisms, not a ranking. Statements such as "best for categorical data," "fastest," or "easiest to tune" require a defined dataset, metric, resource budget, and tuning protocol.

### Scikit-learn estimators

Scikit-learn maintains both the older `GradientBoostingClassifier` and `GradientBoostingRegressor` family and the histogram-based `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` family.[27] The histogram estimators support missing values, categorical features, early stopping, and monotonic constraints, subject to estimator and objective restrictions documented by the project.[28]

The project history matters when describing these features. Histogram estimators appeared experimentally in 0.21.[33] Native categorical support arrived in 0.24.[34] They later ceased to be experimental, but that later milestone should not be confused with the introduction of categorical support.

## Evaluation and data practice

### Validation design

Random train-test splitting is inappropriate when observations are ordered in time, grouped by subject, spatially dependent, or otherwise not exchangeable. Cross-validation tools include stratified, grouped, and time-aware splitters because the validation design must reflect how future predictions will be made.[31]

All preprocessing learned from data must be fitted inside each training fold. Examples include imputation values, category vocabularies, target encoding, feature selection, and scaling used by a comparison model. Scikit-learn's common-pitfalls guide identifies inconsistent preprocessing and data leakage as frequent causes of invalid evaluation and recommends pipelines to bind transformations to estimator fitting.[38]

Hyperparameter search consumes validation information. Reporting the same cross-validation score used to select a large search as though it were an untouched performance estimate can be optimistic. A separate test set or nested cross-validation is appropriate when the goal is an independent generalization estimate.

### Reproducibility and deployment

A reproducible record should include the training data version, row and feature definitions, split assignments, package and hardware versions, objective, evaluation metrics, categorical feature declarations, random seeds, stopping rule, selected iteration, and all nondefault parameters. A seed alone may not guarantee bitwise-identical results when parallel reductions, GPU algorithms, or implementation versions differ.

The serialized model must preserve more than a collection of split thresholds when library-specific metadata is required. For example, XGBoost warns that categorical model information is retained in JSON and UBJSON model formats, not its legacy binary format.[20] Deployment code must apply the same category typing, feature order, missing-value representation, and transformation logic used during training.

Tree predictions are piecewise constant within terminal regions for ordinary regression-tree leaves. Consequently, a tree ensemble does not extrapolate a linear trend beyond the observed feature range in the way a correctly specified linear model can. It can still produce a prediction for an out-of-range value by following split branches, but that prediction is assembled from learned leaf values rather than a learned continuation of the trend. Whether this behavior is acceptable depends on the task.

Monitoring should distinguish changes in input distributions, missingness, category frequencies, score distributions, calibration, and outcome performance. Outcome-based monitoring may be delayed when labels arrive late. Retraining is itself a new model-selection event and should repeat leakage controls and independent evaluation rather than assuming that a newer training window is better.

### Imbalance and weighting

Class weights, row weights, resampling, and threshold choice solve different problems. Weighting the training loss changes the fitted objective. Resampling changes the empirical training distribution. Moving a decision threshold changes decisions after scores are produced. None automatically produces calibrated probabilities for the original population.

Accuracy can conceal poor minority-class performance. Metrics should follow the decision problem and may include log loss, area under a precision-recall curve, class-specific recall, or cost-weighted utility. The chosen metric should be defined before comparing model variants where feasible.

### Missing values and categorical variables

Native missing-value routing avoids a separate imputation step, but missingness can encode a collection process that changes between training and deployment. A model may learn the fact of missingness as a signal. That can be useful, unstable, unfair, or leaky depending on why the value is absent.

High-cardinality categories and identifiers require special care. An unconstrained identifier can let a powerful tree ensemble memorize training entities. Ordered encodings reduce one form of target leakage but do not validate the feature's meaning or guarantee transfer to unseen entities.[12][24]

### Baselines and uncertainty

A tuned gradient-boosting model should be compared with simple baselines and reasonable alternatives under the same splits and metric. For some problems, a constant predictor, regularized linear model, [random forest](https://aiwiki.ai/wiki/random_forest), or domain rule may be competitive and easier to maintain. Scikit-learn's ensemble guide describes random forests as averages of randomized trees and gradient boosting as stagewise additions optimized against a loss; their error behavior cannot be reduced reliably to a universal "variance versus bias" table.[27]

Performance differences should be accompanied by uncertainty where possible. Repeated folds, bootstrap intervals on a fixed test set, or task-appropriate paired tests can show whether an observed difference is stable. If the full tuning process is repeated, its variability should be included rather than measuring only the final fixed model.

## Interpreting a fitted ensemble

Interpretation methods answer different questions and can be unstable under correlated features or distribution shift. None turns an associational predictor into a causal model.

### Split and gain summaries

Tree packages can count how often a feature is used or sum the gain attributed to its splits. These summaries are cheap and tied to the fitted tree structures. Features with many potential cut points can receive more opportunities to split, and correlated predictors can substitute for one another. Importance values from different packages may also use different definitions or normalizations.

### Permutation importance

Permutation importance measures the change in a selected evaluation score after a feature column is shuffled. It is model-agnostic, but it is conditional on the fitted model, dataset, scoring function, and permutation procedure. Scikit-learn advises first verifying that the model has predictive power on held-out data. Its documentation also notes that correlated features can each appear unimportant because the other feature preserves similar information when one is shuffled.[29]

Permutation importance on training data describes reliance for training performance; permutation importance on a held-out set describes reliance for that evaluation distribution. Neither should be labeled intrinsic feature importance without qualification.

### Partial dependence and ICE

Partial dependence averages model predictions over a distribution of other features while varying selected features. Individual Conditional Expectation curves show the corresponding trajectory for individual observations. These tools describe the fitted response surface, not a causal intervention.[30]

When features are correlated, the evaluation grid can combine values rarely or never observed together. Scikit-learn warns that partial-dependence interpretation assumes the target features are independent of the complement features. ICE can expose heterogeneity hidden by an average but does not remove the extrapolation problem.[30]

### SHAP and TreeSHAP

SHAP places additive feature attributions in a Shapley-value framework and identifies properties satisfied by a class of additive explanation methods.[17] TreeSHAP provides efficient algorithms for tree ensembles and enables aggregation from local attributions to global summaries.[18]

Attributions depend on how "missing" features and the background distribution are defined. Different approximations to conditional expectations can give different answers for dependent inputs. It is therefore inaccurate to claim simply that TreeSHAP "handles correlated features fairly." SHAP values explain a model relative to an attribution setup; they do not establish that a feature caused the prediction in the real world.[17][18]

## Learning to rank

Ranking data are organized into query or group units containing items with relevance labels. Metrics such as normalized discounted cumulative gain depend on the order of multiple items and involve a sorting operation, so they are not ordinary pointwise differentiable losses.

LambdaRank constructs pairwise "lambda" values that incorporate the metric change associated with swapping ranked items. LambdaMART fits Multiple Additive Regression Trees to those lambda targets. Christopher Burges's 2010 overview traced the path from RankNet through LambdaRank to LambdaMART and reported that an ensemble of LambdaMART rankers won Track 1 of the 2010 Yahoo Learning to Rank Challenge.[16] This is a specific historical competition result, not evidence that LambdaMART is always the best ranker.

XGBoost documents `rank:ndcg` as a LambdaMART objective. Its ranking guide describes query groups, pair construction, NDCG weighting, and an unbiased option intended to reduce position bias in click data.[21] Correct grouping is essential: rows from one query must not be treated as independent items from unrelated queries, and train-test splitting should avoid leaking query-specific information.

## Evidence on tabular data

Gradient-boosted trees are strong baselines for many tabular problems, but the magnitude and direction of their advantage depend on protocol.

Grinsztajn, Oyallon, and Varoquaux compared tree-based models with neural-network approaches on 45 tabular datasets containing about 10,000 observations after applying their inclusion and preprocessing rules. Within their benchmark, tree-based models remained strongest overall and showed advantages connected to irregular target functions and uninformative features.[14] The study did not cover every scale, modality, or modern architecture.

McElfresh and colleagues compared 19 algorithms across 176 classification datasets and analyzed dataset characteristics related to relative performance. They found that differences between the strongest boosted-tree and neural-network methods were often small, that light hyperparameter tuning was important, and that no family dominated every dataset.[15] Their result argues for protocol-matched comparison rather than an unconditional statement that either trees or neural networks own tabular learning.

Historical competition counts in the XGBoost paper provide a different kind of evidence: adoption in selected winning solutions reported on the Kaggle blog and KDD Cup results around 2015.[10] Those observations show practical impact at that time. They should not be projected into a current win rate without a new, reproducible census.

## Applications

Gradient boosting can be considered when examples can be represented by features and an objective captures the desired prediction. The method's availability for a task does not itself establish suitability or safety.

Common task forms include:

- predicting a numeric outcome such as demand, duration, or cost;
- estimating a binary or multiclass score;
- estimating conditional quantiles for asymmetric planning decisions;
- ranking items within a query group;
- combining engineered features from text, images, or events with other structured variables.

Whether gradient boosting is appropriate depends on data generation, latency and memory constraints, missingness, error costs, drift, and governance requirements. In high-stakes settings, performance must be assessed for relevant subgroups and operating points. Explanations such as feature importance do not substitute for validation, documentation, monitoring, or human review.

## Advantages and limitations

### Advantages

- The objective can be adapted to regression, classification, ranking, quantile estimation, and other well-specified targets.[1][16][36]
- Trees capture nonlinear thresholds and feature interactions without requiring polynomial terms to be written in advance.
- Mature implementations provide histogram algorithms, sparse-input handling, parallel work within tree construction, and CPU or GPU options.[10][11]
- Some implementations route missing numerical values and support categorical splits directly.[20][22][24][26]
- Shrinkage, sampling, tree constraints, penalties, early stopping, and monotonic constraints provide multiple ways to control the fitted function.[19][23][25][37]
- Strong performance has been documented in broad, but protocol-bounded, tabular benchmarks.[14][15]

### Limitations

- Boosting rounds are sequential, which limits parallelism across stages.
- Performance can be sensitive to the learning rate, number of rounds, tree size, sampling, and library-specific defaults.
- Large ensembles can increase training cost, model size, and prediction latency.
- Native missing-value or categorical support does not prevent leakage, unstable collection effects, or train-serving mismatches.
- Tree ensembles do not automatically exploit the spatial, sequential, or compositional structure that specialized models use for raw images, audio, and language.
- Split, permutation, partial-dependence, and SHAP explanations all have assumptions and can change under correlated features or a different reference distribution.[18][29][30]
- A fitted score can be discriminative yet poorly calibrated for the deployment population.[32]
- Flexible learners can exploit accidental identifiers, post-outcome variables, and preprocessing leakage unless the evaluation pipeline is designed carefully.[38]

## Terminology

| Term | Meaning |
|---|---|
| Gradient boosting machine (GBM) | Friedman's name for the general stagewise gradient-boosting construction |
| Gradient-boosted decision trees (GBDT) | Gradient boosting with decision-tree base learners |
| Multiple Additive Regression Trees (MART) | A name used for additive regression-tree boosting, including in LambdaMART |
| TreeBoost | Friedman's tree-specific gradient-boosting procedures |
| Stochastic gradient boosting | In Friedman's 2002 paper, row subsampling without replacement at each stage |
| Histogram-based gradient boosting | Tree construction using binned feature values and aggregated gradient statistics |
| DART | A tree-boosting variant that drops existing trees during training and normalizes contributions |

"Boosted trees," "GBDT," and "gradient boosting" are often used loosely as synonyms in software discussions. The general framework can use other base learners, while many system features described on this page are specific to tree implementations.

## See also

- [Boosting](https://aiwiki.ai/wiki/boosting)
- [Ensemble learning](https://aiwiki.ai/wiki/ensemble_learning)
- [Gradient descent](https://aiwiki.ai/wiki/gradient_descent)
- [Decision tree](https://aiwiki.ai/wiki/decision_tree)
- [Gradient boosted decision trees](https://aiwiki.ai/wiki/gradient_boosted_decision_trees_gbt)
- [Random forest](https://aiwiki.ai/wiki/random_forest)
- [Overfitting](https://aiwiki.ai/wiki/overfitting)
- [XGBoost](https://aiwiki.ai/wiki/xgboost)
- [LightGBM](https://aiwiki.ai/wiki/lightgbm)
- [CatBoost](https://aiwiki.ai/wiki/catboost)

## References

1. Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine." The Annals of Statistics, 29(5), 1189-1232. https://doi.org/10.1214/aos/1013203451
2. Mason, L., Baxter, J., Bartlett, P., and Frean, M. (2000). "Boosting Algorithms as Gradient Descent." Advances in Neural Information Processing Systems 12. https://papers.nips.cc/paper/1766-boosting-algorithms-as-gradient-descent.pdf
3. Schapire, R. E. (1990). "The Strength of Weak Learnability." Machine Learning, 5, 197-227. https://www.schapire.net/papers/strengthofweak.pdf
4. Freund, Y., and Schapire, R. E. (1997). "A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting." Journal of Computer and System Sciences, 55(1), 119-139. https://www.schapire.net/papers/FreundSc95.pdf
5. ACM Special Interest Group on Algorithms and Computation Theory. (2003). "2003 Godel Prize." https://sigact.org/prizes/g%C3%B6del/2003.html
6. Breiman, L. (1997). "Arcing the Edge." Technical Report 486, Department of Statistics, University of California, Berkeley. https://statistics.berkeley.edu/tech-reports/486
7. Friedman, J. H. (2002). "Stochastic Gradient Boosting." Computational Statistics and Data Analysis, 38(4), 367-378. https://doi.org/10.1016/S0167-9473(01)00065-2
8. Buhlmann, P., and Hothorn, T. (2007). "Boosting Algorithms: Regularization, Prediction and Model Fitting." Statistical Science, 22(4), 477-505. https://arxiv.org/pdf/0804.2752
9. Natekin, A., and Knoll, A. (2013). "Gradient Boosting Machines, a Tutorial." Frontiers in Neurorobotics, 7, 21. https://www.frontiersin.org/journals/neurorobotics/articles/10.3389/fnbot.2013.00021/full
10. Chen, T., and Guestrin, C. (2016). "XGBoost: A Scalable Tree Boosting System." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794. https://arxiv.org/pdf/1603.02754
11. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. (2017). "LightGBM: A Highly Efficient Gradient Boosting Decision Tree." Advances in Neural Information Processing Systems 30. https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf
12. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. (2018). "CatBoost: Unbiased Boosting with Categorical Features." Advances in Neural Information Processing Systems 31. https://papers.nips.cc/paper/7898-catboost-unbiased-boosting-with-categorical-features.pdf
13. Rashmi, K. V., and Gilad-Bachrach, R. (2015). "DART: Dropouts Meet Multiple Additive Regression Trees." Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, 489-497. https://proceedings.mlr.press/v38/korlakaivinayak15.pdf
14. Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). "Why Do Tree-Based Models Still Outperform Deep Learning on Typical Tabular Data?" Advances in Neural Information Processing Systems 35. https://papers.neurips.cc/paper_files/paper/2022/file/0378c7692da36807bdec87ab043cdadc-Paper-Datasets_and_Benchmarks.pdf
15. McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., and White, C. (2023). "When Do Neural Nets Outperform Boosted Trees on Tabular Data?" Advances in Neural Information Processing Systems 36. https://papers.neurips.cc/paper_files/paper/2023/file/f06d5ebd4ff40b40dd97e30cee632123-Paper-Datasets_and_Benchmarks.pdf
16. Burges, C. J. C. (2010). "From RankNet to LambdaRank to LambdaMART: An Overview." Microsoft Research Technical Report MSR-TR-2010-82. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/MSR-TR-2010-82.pdf
17. Lundberg, S. M., and Lee, S.-I. (2017). "A Unified Approach to Interpreting Model Predictions." Advances in Neural Information Processing Systems 30. https://papers.nips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf
18. Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I. (2020). "From Local Explanations to Global Understanding with Explainable AI for Trees." Nature Machine Intelligence, 2, 56-67. https://www.nature.com/articles/s42256-019-0138-9
19. XGBoost developers. "XGBoost Parameters." https://xgboost.readthedocs.io/en/stable/parameter.html
20. XGBoost developers. "Categorical Data." https://xgboost.readthedocs.io/en/stable/tutorials/categorical.html
21. XGBoost developers. "Learning to Rank." https://xgboost.readthedocs.io/en/stable/tutorials/learning_to_rank.html
22. LightGBM developers. "Features." https://lightgbm.readthedocs.io/en/stable/Features.html
23. LightGBM developers. "Parameters." https://lightgbm.readthedocs.io/en/stable/Parameters.html
24. CatBoost developers. "Categorical Features." https://catboost.ai/docs/en/features/categorical-features
25. CatBoost developers. "Common Training Parameters." https://catboost.ai/docs/en/references/training-parameters/common
26. CatBoost developers. "Missing Values Processing." https://catboost.ai/docs/en/concepts/algorithm-missing-values-processing
27. Scikit-learn developers. "Ensembles: Gradient Boosting and Random Forests." https://scikit-learn.org/stable/modules/ensemble.html
28. Scikit-learn developers. "HistGradientBoostingClassifier." https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html
29. Scikit-learn developers. "Permutation Feature Importance." https://scikit-learn.org/stable/modules/permutation_importance.html
30. Scikit-learn developers. "Partial Dependence and Individual Conditional Expectation Plots." https://scikit-learn.org/stable/modules/partial_dependence.html
31. Scikit-learn developers. "Cross-Validation: Evaluating Estimator Performance." https://scikit-learn.org/stable/modules/cross_validation.html
32. Scikit-learn developers. "Probability Calibration." https://scikit-learn.org/stable/modules/calibration.html
33. Scikit-learn developers. "Version 0.21 Changelog." https://scikit-learn.org/stable/whats_new/v0.21.html
34. Scikit-learn developers. "Release Highlights for scikit-learn 0.24." https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_0_24_0.html
35. Duan, T., Anand, A., Ding, D. Y., Thai, K. K., Basu, S., Ng, A., and Schuler, A. (2020). "NGBoost: Natural Gradient Boosting for Probabilistic Prediction." Proceedings of the 37th International Conference on Machine Learning, 2690-2700. https://proceedings.mlr.press/v119/duan20a/duan20a.pdf
36. XGBoost developers. "Custom Objective and Evaluation Metric." https://xgboost.readthedocs.io/en/stable/tutorials/custom_metric_obj.html
37. XGBoost developers. "Monotonic Constraints." https://xgboost.readthedocs.io/en/stable/tutorials/monotonic.html
38. Scikit-learn developers. "Common Pitfalls and Recommended Practices." https://scikit-learn.org/stable/common_pitfalls.html