Citation and evidence

XGBoost

28 min full readUpdated 24 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AlgorithmsMachine LearningOpen Source AI

Cite this article

See also: Gradient boosted (decision) trees, Gradient Boosting, LightGBM, CatBoost

XGBoost logo
XGBoost logo, from the official website xgboost.ai. Trademark of its owner.

XGBoost (short for eXtreme Gradient Boosting) is an open-source software library that implements an optimized, distributed version of gradient boosted decision trees, and it is one of the most widely used machine learning algorithms for tabular data. [1] [3] It was created by Tianqi Chen in 2014 as a research project at the University of Washington and popularized by the 2016 paper "XGBoost: A Scalable Tree Boosting System," co-authored with Carlos Guestrin and presented at the 22nd ACM SIGKDD conference; that paper has been cited more than 47,000 times (Semantic Scholar, 2026), making it one of the most cited works in applied machine learning. [1] [2] The library combines a regularized formulation of gradient boosting with a set of systems-level engineering tricks (sparsity-aware splitting, weighted quantile sketch, cache-aware histogram blocks, distributed training, and GPU acceleration) that together let it train on billions of examples while routinely topping the leaderboard on tabular problems. [1]

The paper itself states the goal directly: "The most important factor behind the success of XGBoost is its scalability in all scenarios. The system runs more than ten times faster than existing popular solutions on a single machine and scales to billions of examples in distributed or memory-limited settings." [1] For most of the period from 2014 through 2018, XGBoost was the single most cited algorithm in the winning solutions of Kaggle competitions, and even in 2026 it remains a default first choice for tabular regression and classification in finance, advertising, fraud detection, and risk modeling. The library is written in C++ and exposes idiomatic bindings for Python, R, Julia, Java, Scala, Ruby, Swift, and the JVM ecosystem (Spark, Flink, Hadoop), with a scikit-learn-compatible API and distributed training on Apache Spark, Dask, and Ray. [3] It is hosted by the Distributed Machine Learning Community (DMLC) on GitHub at dmlc/xgboost and has more than 28,000 GitHub stars as of 2026. [4]

eli5 (explain like i'm five)

Imagine you are trying to predict how much rain will fall tomorrow. You ask a hundred friends to make tiny guesses, and you add their guesses together to get a final answer. The trick is that each new friend is only allowed to look at the mistakes the previous friends made, and they have to write a small rule that fixes those mistakes a tiny bit. After a hundred friends, the total guess becomes very accurate.

XGBoost is a computer program that does this very fast and very well. Each "friend" is a small decision tree that splits the world into a few buckets and gives a number to each bucket. XGBoost adds a few extra clever ideas: it punishes friends who try to make rules that are too complicated, it skips empty data without complaining, and it uses every CPU core (and every GPU) on your computer at the same time so the whole thing finishes in seconds instead of hours.

"XGBoost Part 1 (of 4): Regression", video from StatQuest with Josh Starmer, published December 16, 2019. Watch on YouTube.

When was XGBoost created?

the dmlc origin (2014)

XGBoost began as a research project by Tianqi Chen, then a PhD student at the University of Washington under the supervision of Carlos Guestrin in the Paul G. Allen School of Computer Science. [6] Chen made the repository's initial commit on February 6, 2014, and tagged the first release, v0.1, on March 27, 2014. [4] The initial release was a small command-line C++ program configured through libsvm-format text files. According to a 2014 ATLAS collaboration article, Chen developed XGBoost because his own research code base was too slow. [21]

The project was hosted under the Distributed (Deep) Machine Learning Community (DMLC), an umbrella that Chen co-founded along with collaborators including Mu Li, Naiyan Wang, and Bing Xu. DMLC also hatched MXNet and TVM, both later donated to the Apache Software Foundation; TVM became a top-level Apache project, while MXNet was retired from the Apache Incubator without graduating.

the higgs boson breakthrough

XGBoost first attracted broad attention through the Higgs Boson Machine Learning Challenge that ran on Kaggle from May to September 2014, a competition that drew a then-record 1,785 teams. [13] [21] The challenge invited data scientists to help physicists at the Large Hadron Collider at CERN classify Higgs boson decay events from background noise. [13] Tianqi Chen released a public XGBoost benchmark script for the challenge that beat almost every other submission in the leaderboard for many weeks. Multiple top finishers attributed their solutions to XGBoost, and at the closing workshop Chen and Tong He (team Crowwork) received the Special "High Energy Physics meets Machine Learning" Award; the ATLAS collaboration wrote that their algorithm "was an excellent compromise between performance and simplicity." [21] By the time the competition closed, the project had jumped from a small research codebase to one of the most discussed tools in competitive machine learning.

The Higgs win catalyzed two follow-on developments. First, Bing Xu wrote the Python bindings (xgboost on PyPI) and Tong He wrote the R bindings (xgboost on CRAN), which together turned the project from a niche command-line tool into a library that could be plugged into any data scientist's workflow. Second, the team began to invest seriously in distributed training, leading to the early Apache Spark and YARN integrations.

the sigkdd 2016 paper

In March 2016, Chen and Guestrin posted the now-famous paper "XGBoost: A Scalable Tree Boosting System" to arXiv (1603.02754) and presented it at the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining in San Francisco. [1] [2] The paper formalized the algorithmic improvements behind the library: a regularized objective derived from a second-order Taylor expansion, a sparsity-aware split-finding algorithm that learns a default direction for missing values, a weighted quantile sketch for approximate split-point selection on huge datasets, and a cache-aware columnar block layout for parallel and out-of-core computation. [1] The authors summarized the contribution as "a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning." [1]

The paper has since been cited more than 47,000 times (Semantic Scholar, 2026), making it one of the most cited works in applied machine learning. [2] In the same year, Tong He and Chen shared the American Statistical Association's John M. Chambers Statistical Software Award for XGBoost (the award carries a cash prize of 2,000 dollars), and the team had earlier been recognized with the High Energy Physics meets Machine Learning Award for the role XGBoost played in the Higgs work. [13] [19]

kaggle dominance, then competition (2015-2020)

Between 2015 and 2018, XGBoost was the most popular algorithm on Kaggle. The original paper reports that among the 29 competition-winning solutions published on Kaggle's blog during 2015, 17 used XGBoost (eight of them as the sole model and the rest as a component of an ensemble), while the second most popular method, deep neural networks, was used in only 11 winning solutions. [1] The library became a kind of professional standard: tutorials, kernels, and Coursera courses all assumed it as the default tabular baseline.

This dominance prompted the creation of competing implementations. Microsoft Research released LightGBM in 2016 with leaf-wise growth and a more aggressive histogram algorithm that delivered substantial speedups on very wide datasets. [9] Yandex released CatBoost in 2017 with a focus on categorical-feature handling and ordered boosting. [10] Scikit-learn shipped its own histogram-based implementation, HistGradientBoosting, in version 0.21 in 2019, drawing heavily on the LightGBM design. By 2020, the gradient boosting market had three major open-source players plus a strong scikit-learn implementation, and XGBoost responded by adopting many of the same techniques (histogram trees, leaf-wise growth, native categorical support). [5]

major version milestones

The project followed a slow, deliberate release cadence in its first few years. With the 1.0 release in February 2020, the team adopted semantic versioning and committed to a [MAJOR].[FEATURE].[MAINTENANCE] scheme. [5] Subsequent major releases reworked GPU support, the Apache Spark integration, the JVM packages, the R API, and (in the 3.x line) external-memory training. [5]

How does the XGBoost algorithm work?

regularized objective

XGBoost optimizes a regularized objective at each boosting iteration t:

L(t) = sum_i l(y_i, y_hat_i^(t-1) + f_t(x_i)) + Omega(f_t)

where l is any twice-differentiable convex loss (squared error for regression, logistic loss for binary classification, softmax for multiclass, etc.), f_t is the new tree being added, and Omega(f) = gamma * T + 0.5 * lambda * ||w||^2 penalizes the number of leaves T and the L2 norm of the leaf weight vector w. [1] The library also supports an L1 penalty alpha * ||w||_1 on the leaf weights, giving both ridge and lasso flavors of regularization. [3]

Where classical gradient boosting (Friedman, 2001) approximated the loss with only the first-order gradient, XGBoost uses a second-order Taylor expansion. [8] After dropping constants, the per-iteration objective becomes:

L_tilde(t) ~= sum_i [ g_i * f_t(x_i) + 0.5 * h_i * f_t(x_i)^2 ] + Omega(f_t)

where g_i = d l / d y_hat_i^(t-1) and h_i = d^2 l / d y_hat_i^(t-1)^2 are the first and second derivatives of the loss at the previous prediction. For a fixed tree structure with leaf assignments q(x), the optimal leaf weight is w_j* = -G_j / (H_j + lambda) and the corresponding objective score is -0.5 * sum_j G_j^2 / (H_j + lambda) + gamma * T, where G_j and H_j are the sums of g_i and h_i over the instances in leaf j. The split-finding algorithm uses this score to greedily evaluate every candidate split, picking the one that maximizes:

Gain = 0.5 * [ G_L^2/(H_L+lambda) + G_R^2/(H_R+lambda) - (G_L+G_R)^2/(H_L+H_R+lambda) ] - gamma

The gamma term acts as a fixed cost for splitting and lets the algorithm prune any split whose gain does not exceed it. [1]

exact and approximate split finding

For small datasets the library uses an exact greedy search that enumerates every candidate split for every feature. For large datasets this becomes infeasible, so XGBoost falls back to an approximate algorithm based on a weighted quantile sketch. [1] The key insight is that the second-order weights h_i already act as natural importance weights for each example, so the histogram bin boundaries should be the quantiles of the empirical distribution of features weighted by h_i. The paper introduces a novel sketch data structure that supports merge and prune operations on weighted points, allowing the quantile boundaries to be computed in a single distributed pass over the data. [1]

In 2017 the team added a fully histogram-based tree method (tree_method="hist") that pre-bins the features once and then operates on integer bin indices, similar to LightGBM's default. This is now the recommended setting for both CPU and GPU training. [3]

sparsity-aware splitting

Real tabular data is full of missing values, one-hot encodings, and categorical sparsity. Rather than imputing, XGBoost defines a default direction at every split: when a feature value is missing (or zero in a sparse matrix), the instance is routed to the default branch. [1] The default direction is learned during training by trying both options for every candidate split and choosing whichever maximizes the gain. This means the model can take advantage of the meaningful pattern of missingness in datasets where missingness is informative (a common situation in survey data, electronic health records, and fraud detection). The original paper reported that the sparsity-aware algorithm ran more than 50 times faster on an Allstate sparse dataset than a naive implementation that did not exploit sparsity. [1]

parallel and cache-aware execution

XGBoost stores the training data as a set of compressed, in-memory column blocks sorted by feature value. [1] Each block can be scanned independently by a different CPU core to compute split statistics, which is the source of the library's intra-machine parallelism. The blocks are also designed to fit in CPU cache, with prefetching used to hide memory latency. [1] For datasets that do not fit in RAM, the blocks can be sharded onto disk; the library also supports an out-of-core mode that streams blocks from disk, and an external-memory mode (substantially overhauled in version 3.0 with the new ExtMemQuantileDMatrix) for datasets larger than host memory. [5]

gpu support

XGBoost has had a CUDA-based GPU implementation since version 0.7 (released in December 2017). [5] The current device="cuda" setting (which superseded the earlier tree_method="gpu_hist") implements the histogram algorithm directly on the GPU and supports both single-GPU and multi-GPU training (the latter via NCCL). [3] On large datasets it can be ten to twenty times faster than the CPU version. NVIDIA has invested directly in the project, and the GPU code is co-maintained by their RAPIDS team. [15]

monotonic and feature interaction constraints

Two constraint mechanisms make XGBoost especially useful for regulated domains. Monotonic constraints force the model to be monotonically increasing or decreasing in a chosen feature, which is often a regulatory requirement in credit scoring (a higher income can never lower the predicted creditworthiness). [3] The constraint is enforced during split-finding by rejecting any split that would violate the requested direction at the current node. Feature interaction constraints restrict which subsets of features are allowed to appear together in a single root-to-leaf path, which is useful for interpretable modeling and for reducing spurious interactions in noisy datasets. [3]

What are the most important XGBoost hyperparameters?

XGBoost has more than fifty parameters, but a small subset accounts for almost all the practical tuning. The table below lists the hyperparameters that data scientists most often touch. Defaults shown are for the Python XGBClassifier / XGBRegressor API as of version 3.2. [3]

parameterdefaulttypical rangerole
n_estimators100100 to 5000Number of boosting rounds (trees). Use early stopping to find the right value.
eta (learning_rate)0.30.01 to 0.3Shrinkage applied to each new tree. Lower values need more trees but generalize better.
max_depth63 to 10Maximum depth of each tree. Shallower trees reduce overfitting but may underfit.
min_child_weight11 to 10Minimum sum of Hessians required in a leaf. Acts as a complexity penalty.
gamma (min_split_loss)00 to 10Minimum loss reduction required to make a split. Larger values prune more aggressively.
subsample1.00.5 to 1.0Fraction of rows sampled for each tree. Adds randomness, similar to bagging.
colsample_bytree1.00.5 to 1.0Fraction of columns sampled for each tree.
colsample_bylevel1.00.5 to 1.0Fraction of columns sampled at each tree level.
colsample_bynode1.00.5 to 1.0Fraction of columns sampled at each split.
reg_lambda1.00 to 10L2 regularization on leaf weights.
reg_alpha0.00 to 10L1 regularization on leaf weights.
scale_pos_weight1.0dependsUp-weights the positive class for imbalanced classification.
tree_method"hist"hist, exact, approxAlgorithm used for split finding. Hist is now the default.
device"cpu"cpu, cuda, cuda:0Hardware backend. Set to cuda for GPU training.
objectivetask-dependentreg:squarederror, binary:logistic, multi:softprob, rank:pairwise, etc.Loss function used during training.
eval_metrictask-dependentrmse, logloss, auc, ndcg, etc.Metric reported on the validation set.
early_stopping_roundsNone10 to 100Stop training when validation metric does not improve for this many rounds.

Expanded article table

A common starting point for tabular classification is eta=0.05, max_depth=6, min_child_weight=1, subsample=0.8, colsample_bytree=0.8, with n_estimators=2000 and early_stopping_rounds=50. Bayesian optimization tools such as Optuna and Ray Tune are widely used to find better settings automatically.

distributed training

XGBoost has supported distributed training since its 0.4 release in May 2015. [24] The original implementation used the DMLC Rabit library for collective communication (allreduce, broadcast) over a small set of nodes connected via TCP or YARN. [3] Modern XGBoost integrates with three main distributed frameworks:

  • Apache Spark, via the xgboost4j-spark JVM package and the PySpark estimator interface added in version 1.7. [20] [24]
  • Dask, via the xgboost.dask Python module. Dask is the recommended choice for users who already have a Dask cluster and want to stay inside the Python ecosystem. The interface mirrors the standard scikit-learn API. [3]
  • Ray, via the xgboost_ray package originally developed at Anyscale and Uber. Ray adds elastic training (the job continues if a worker dies) and seamless integration with Ray Tune for hyperparameter optimization. [14]

All three integrations support multi-GPU training. Distributed XGBoost typically scales close to linearly up to a few dozen workers; beyond that, communication of the gradient histograms can become the bottleneck.

bindings and ecosystem

XGBoost ships native bindings for several languages, with the C++ core compiled as a shared library that is loaded by each language wrapper. [3]

  • Python (xgboost on PyPI): the dominant interface, including a scikit-learn-compatible API (XGBClassifier, XGBRegressor, XGBRanker) and the lower-level Booster API. [3]
  • R (xgboost on CRAN): a fully featured wrapper that integrates with the caret and tidymodels packages, substantially reworked into a more idiomatic interface in version 3.0. [3] [5]
  • JVM (Java, Scala, Kotlin): packaged as xgboost4j and xgboost4j-spark for Maven/Gradle. [20]
  • Julia (XGBoost.jl): a community-maintained wrapper.
  • Other: Ruby (xgboost-ruby), Swift (SwiftXGBoost), Go bindings, and a Rust crate (xgboost-rs) all exist with varying maturity.

Within the broader Python ecosystem, XGBoost integrates with scikit-learn pipelines, MLflow for experiment tracking, ONNX for model export, SHAP for interpretability (the shap.TreeExplainer is highly optimized for XGBoost), and almost every commercial ML platform including AWS SageMaker, Google Vertex AI, Azure Machine Learning, Databricks, and Snowflake. [16]

How does XGBoost differ from LightGBM and CatBoost?

The table below summarizes how XGBoost compares with the two main alternative implementations of gradient boosted trees and with scikit-learn's built-in HistGradientBoosting. All four are considered "production grade" and benchmarks generally rank them within a few percentage points of one another on most tabular tasks.

featureXGBoostLightGBMCatBoostscikit-learn HistGB
First release2014201620172019 (v0.21)
MaintainerDMLCMicrosoft ResearchYandexscikit-learn community
Default tree growthLevel-wise (depth-wise)Leaf-wiseSymmetric (oblivious)Level-wise
Default split algorithmHistogramHistogramHistogramHistogram
Categorical handlingNative (experimental from 1.3, all tree methods from 1.6)NativeNativeNative
Missing value handlingNative (sparsity-aware)NativeNativeNative
GPU trainingYes (CUDA, multi-GPU)Yes (CUDA, OpenCL)Yes (CUDA, multi-GPU)No
Distributed trainingSpark, Dask, RayDaskSpark, DaskNo
L1/L2 regularizationBothBothL2 onlyL2 only
Monotonic constraintsYesYesYesYes
Feature interaction constraintsYesYesNoYes
Best fitDefault tabular baseline; sparse, large dataWide datasets (many features), large row countsHeavy categorical features, low-tuning workflowsPipelines that must stay inside scikit-learn

Expanded article table

The rough heuristic among practitioners is: try XGBoost first as a strong baseline, switch to LightGBM if training time or memory becomes a bottleneck on very large data, and switch to CatBoost when the dataset is dominated by high-cardinality categorical features. In Kaggle competitions a stacked ensemble of all three is still common.

What is XGBoost used for?

kaggle and competitive machine learning

XGBoost was the dominant Kaggle algorithm from 2014 through 2018. In 2015, 17 of the 29 winning solutions published on Kaggle's blog used XGBoost, most of the non-solo ones combining it with neural nets in ensembles. [1] After 2018 the share of pure-XGBoost wins decreased as LightGBM and CatBoost matured and as deep learning models started to compete on image and text-heavy challenges, but XGBoost remains a near-universal component in tabular competitions.

financial risk and credit scoring

The library has been adopted across the financial industry for credit-risk modeling, loan default prediction, anti-money laundering, and trading signal research. The combination of strong out-of-the-box accuracy, monotonic constraints, and SHAP-based interpretability makes it a natural fit for regulated environments where the model must be auditable. [16]

click-through rate and ad ranking

Click-through rate (CTR) prediction is the canonical advertising-tech problem: given a user, a context, and a candidate ad, estimate the probability of a click. XGBoost was widely used in CTR models before deep learning architectures (Wide & Deep, DeepFM, DCN, DIN) took over the largest production systems. It is still a common baseline and feature-extraction stage. Hybrid pipelines of boosted trees plus logistic regression, introduced by Facebook in its 2014 "Practical Lessons from Predicting Clicks on Ads" paper (which used the leaf indices of boosted decision trees as one-hot features for a downstream linear model), are often rebuilt with XGBoost as the tree stage. [17]

churn, fraud, and demand forecasting

Customer churn prediction (telecom, banking, SaaS), credit-card fraud detection, supply-chain demand forecasting, energy load forecasting, and electronic health record (EHR) modeling are all dominated by gradient boosting in practice, and XGBoost is among the most common implementations.

scientific applications

Beyond the original Higgs Boson use case, XGBoost is used in particle physics, astronomy (galaxy classification, photometric redshift), bioinformatics (variant pathogenicity prediction, drug response modeling), epidemiology, and climate science. [13] The combination of fast training on large tabular tables and built-in feature importance makes it attractive whenever a researcher needs both predictive accuracy and a rough sense of which inputs matter.

Does deep learning beat XGBoost on tabular data?

A recurring debate in the machine learning community is whether deep neural networks can out-perform gradient boosted trees on tabular data. For classic deep tabular architectures, the answer has largely been no; since 2025, however, tabular foundation models such as TabPFN have been reported to beat tuned gradient-boosted trees on datasets of up to 10,000 samples. [23] The 2021 Shwartz-Ziv and Armon paper "Tabular Data: Deep Learning Is Not All You Need" benchmarked XGBoost against TabNet, NODE, and several other tabular deep models on 11 datasets and concluded that XGBoost outperformed every deep model on average and required dramatically less hyperparameter tuning. [11] The 2022 Grinsztajn et al. NeurIPS paper "Why do tree-based models still outperform deep learning on typical tabular data?" reached the same conclusion on a curated benchmark of 45 datasets, finding that tree-based models such as XGBoost "remain state-of-the-art on medium-sized data (~10K samples)." [12]

Several structural reasons explain the persistent gap. Decision trees are robust to uninformative features (they simply do not split on them), while neural networks must learn to ignore noise. Trees are invariant to monotonic transformations of individual features, so they do not require careful normalization. Trees handle missing values and heavy-tailed distributions natively. Finally, neural networks for tabular data tend to require large amounts of data to compete, while most tabular problems are small or medium-sized. [12]

That said, deep models close the gap or surpass XGBoost in two specific situations. First, when the dataset is very large (tens of millions of rows or more) and there are meaningful interactions between many features, transformer-based architectures such as TabTransformer, FT-Transformer, and SAINT can produce slightly better results, often at much higher computational cost. Second, in multimodal pipelines where tabular features must be combined with text, image, or sequence data, deep models offer a natural way to fuse modalities. In multimodal pipelines a common pattern is to use XGBoost on the tabular part and concatenate or stack its predictions with the deep model's outputs.

limitations

Despite its strengths, XGBoost has well-known weaknesses that practitioners need to be aware of.

  • Limited interpretability of large ensembles: While SHAP makes per-prediction attribution feasible, an ensemble of a thousand depth-6 trees is not directly inspectable. In regulated domains a simpler model (logistic regression, EBM, decision tree) may still be preferred for top-level reasoning. [16]
  • Hyperparameter sensitivity: Although less sensitive than deep networks, XGBoost has many hyperparameters and a poorly tuned model can lag a well-tuned one by several percentage points. Automated tuning (Optuna, HyperOpt, Vizier) is essentially mandatory for production use.
  • Not ideal for unstructured data: Images, raw text, and audio do not have the well-defined feature columns that tree models expect. For these modalities convolutional networks, transformers, and pre-trained foundation models perform far better.
  • Memory and compute footprint at scale: While XGBoost can train on billion-row datasets with the histogram method and external memory, very large sparse problems (such as ad-click prediction with billions of features) still favor specialized linear models or factorization machines.
  • Sequential dependence: Each tree depends on the residuals of the previous trees, so boosting itself is inherently sequential. The parallelism is at the per-split level within each tree, not across boosting rounds. This caps the wall-clock speedups that distributed training can achieve, particularly on small to mid-size datasets. [8]
  • Risk of overfitting on noisy or small datasets: Without strong regularization or early stopping, XGBoost can memorize a small training set. Cross-validation and early stopping on a held-out set are essential.
  • No native time-series handling: There is no built-in concept of time. Practitioners must hand-engineer lag features, rolling statistics, and seasonal indicators, or pair XGBoost with a specialized library such as Nixtla's MLForecast or Skforecast.

version history

The table below lists the major XGBoost releases and the headline features they introduced. Patch releases and pre-1.0 versions are omitted for brevity. Dates follow the official release notes (NEWS.md for versions before 2.1); highlights are drawn from the same notes. [5] [18] [24]

versionrelease datehighlights
0.4May 2015Distributed version on YARN; S3 and HDFS input; scikit-learn wrapper in Python; experimental external memory.
0.6July 2016Core refactor to C++11; JVM packages (XGBoost4J) with distributed training on Spark and Flink; DART booster; callback API.
0.7December 2017First CUDA GPU implementation (gpu_hist).
0.8August 2018Redesigned XGBoost4J-Spark API; NCCL2 for multi-GPU training; pre-built Python wheels and Maven Central JARs.
0.9May 2019Python 2 support dropped; scikit-learn-style random forest API (XGBRFClassifier, XGBRFRegressor); multiclass metrics on GPU.
1.0February 2020Semantic versioning and Apache-style governance; native Dask interface; first-class cuDF and CuPy input; Ruby binding.
1.1May 2020Deterministic GPU training for regression and classification; thread-safe in-place prediction; accelerated failure time survival objective.
1.2August 2020GPU training in XGBoost4J-Spark; CUDA 11 support; pseudo-Huber loss.
1.3December 2020Experimental native categorical support (GPU); GPU-accelerated TreeSHAP; new Python callback API; model slicing.
1.4April 2021Pre-built R package with GPU support; unified, thread-safe prediction functions; feature-complete Dask interface.
1.5October 2021Expanded experimental categorical support (prediction, SHAP, plotting; training on gpu_hist); new iterator-based external memory interface.
1.6April 2022Categorical training on hist, approx and gpu_hist with partition-based splits; experimental multi-output models; rewritten approx; UBJSON model format.
1.7October 2022Initial PySpark estimator; QuantileDMatrix extended to CPU; experimental federated learning.
2.0September 2023Initial vector-leaf multi-target trees; new device parameter replacing gpu_hist; hist as default tree method; quantile regression; column-split support for vertical federated learning.
2.1June 2024Rewritten networking/collective module (RABIT revamp, NCCL from PyPI); multi-output and federated learning progress; R package skipped pending a new interface.
3.0February 2025Major external-memory rework with ExtMemQuantileDMatrix; redesigned R package; restructured Spark/JVM packages; reduced GPU memory use.
3.1September 2025Categorical re-coder that keeps encodings consistent at inference; adaptive CUDA external-memory cache; CUDA 13 wheels (xgboost-cu13).
3.2February 2026Vector-leaf multi-target tree progress (reduced-gradient "sketch boost"); enhanced GPU external-memory training; removal of the deprecated CLI.
3.3June 2026Expectile regression; categorical feature support enabled by default; faster, more stable TreeSHAP; gblinear deprecated.
3.4August 2026Vector-leaf implementation for hist feature-complete (experimental); smooth approximations for quantile and MAE objectives; column-split support removed.

Expanded article table

installation and quick start

The Python package is the most common entry point and is installed with pip install xgboost or conda install -c conda-forge xgboost. [3] A minimal binary classification example using the scikit-learn wrapper looks like this:

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, random_state=42
)

model = xgb.XGBClassifier(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.05,
    objective="binary:logistic",
    eval_metric="logloss",
    early_stopping_rounds=20,
    tree_method="hist",
    device="cuda",  # remove this line for CPU training
)
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
print("Accuracy:", model.score(X_test, y_test))

The lower-level Booster API offers more control and is preferred for production deployments and custom training loops:

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

params = {
    "objective": "binary:logistic",
    "eval_metric": "auc",
    "max_depth": 6,
    "eta": 0.05,
    "tree_method": "hist",
    "device": "cuda",
}

booster = xgb.train(
    params,
    dtrain,
    num_boost_round=2000,
    evals=[(dtrain, "train"), (dtest, "valid")],
    early_stopping_rounds=50,
    verbose_eval=100,
)

interpretability and model debugging

XGBoost exposes several tools for understanding what a trained model is doing. [3]

  • model.feature_importances_ returns one importance array whose type is set by importance_type, and Booster.get_score(importance_type=...) returns per-feature scores; the types are weight (number of splits a feature is used in), gain (average loss reduction from those splits), cover (average Hessian coverage of those splits), and their totals total_gain and total_cover. [3]
  • xgb.plot_importance and xgb.plot_tree visualize feature importance and individual trees. [3]
  • The SHAP library provides exact, fast Shapley values for tree ensembles via shap.TreeExplainer. The implementation supports XGBoost alongside other tree ensembles and runs in polynomial time rather than enumerating feature subsets. [16]
  • The pred_interactions=True option in Booster.predict returns the SHAP interaction matrix, a richer attribution that splits each feature's contribution across pairwise interactions. [16]

relationship to the broader gradient boosting family

XGBoost belongs to the family of gradient boosting machines first formalized by Jerome Friedman in 2001. [8] Its closest cousins are:

  • The original gbm R package (Greg Ridgeway, 2003), which was the first widely used implementation but was slow and single-threaded.
  • Scikit-learn's GradientBoostingClassifier and GradientBoostingRegressor (introduced in version 0.11, 2012) [22], which are exact-greedy and now superseded by HistGradientBoosting.
  • LightGBM (Microsoft, 2016), a histogram-based implementation with leaf-wise growth optimized for very wide datasets. [9]
  • CatBoost (Yandex, 2017), an ordered-boosting implementation with industry-leading categorical handling. [10]
  • AdaBoost (Freund and Schapire, 1995), which preceded gradient boosting, and random forests (Breiman, 2001) are closely related ensemble methods that complement it on tabular tasks.

Is XGBoost open source?

XGBoost is released under the Apache License 2.0, a permissive open-source license. [4] Development is coordinated through the dmlc/xgboost GitHub repository, with committers whose listed affiliations include universities (University of Washington, University of Waikato), companies (NVIDIA, Amazon, Uber, Intel, Criteo), and the open-source community. [4] The project follows a lightweight RFC process for major design changes and uses GitHub Actions for continuous integration across Linux, macOS, and Windows on x86-64 and ARM64 architectures.

Continue reading

More from See also

More to read

Topics:AlgorithmsMachine LearningOpen Source AI

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Chen, Tianqi, and Carlos Guestrin. "XGBoost: A Scalable Tree Boosting System." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016): 785-794. dl.acm.org/...2939672.2939785
  2. ^1 ^2 ^3Chen, Tianqi, and Carlos Guestrin. "XGBoost: A Scalable Tree Boosting System." arXiv preprint arXiv:1603.02754 (2016). arxiv.org/...1603.02754
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17XGBoost official documentation. xgboost.readthedocs.io
  4. ^1 ^2 ^3 ^4dmlc/xgboost GitHub repository. github.com/...xgboost
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7XGBoost release notes and version history. xgboost.readthedocs.io/...changes
  6. ^Tianqi Chen. Personal home page, Carnegie Mellon University. tqchen.com
  7. Tianqi Chen. Faculty profile, CMU School of Computer Science. csd.cmu.edu/...tianqi-chen
  8. ^1 ^2 ^3Friedman, Jerome H. "Greedy Function Approximation: A Gradient Boosting Machine." Annals of Statistics 29.5 (2001): 1189-1232.
  9. ^1 ^2Ke, Guolin, et al. "LightGBM: A Highly Efficient Gradient Boosting Decision Tree." Advances in Neural Information Processing Systems 30 (2017).
  10. ^1 ^2Prokhorenkova, Liudmila, et al. "CatBoost: unbiased boosting with categorical features." Advances in Neural Information Processing Systems 31 (2018).
  11. ^Shwartz-Ziv, Ravid, and Amitai Armon. "Tabular Data: Deep Learning is Not All You Need." arXiv preprint arXiv:2106.03253 (2021). arxiv.org/...2106.03253
  12. ^1 ^2Grinsztajn, Leo, Edouard Oyallon, and Gael Varoquaux. "Why do tree-based models still outperform deep learning on typical tabular data?" Advances in Neural Information Processing Systems 35 (2022). arxiv.org/...2207.08815
  13. ^1 ^2 ^3 ^4Higgs Boson Machine Learning Challenge. Kaggle, 2014. kaggle.com/...higgs-boson
  14. ^Anyscale. "Introducing Distributed XGBoost Training with Ray." anyscale.com/...tributed-xgboost-training-with-ray
  15. ^NVIDIA RAPIDS. "GPU-Accelerated XGBoost." developer.nvidia.com/rapids
  16. ^1 ^2 ^3 ^4 ^5SHAP documentation. "TreeExplainer." shap.readthedocs.io
  17. ^He, Xinran, et al. "Practical Lessons from Predicting Clicks on Ads at Facebook." Proceedings of the Eighth International Workshop on Data Mining for Online Advertising (2014).
  18. ^XGBoost 3.0.0, 3.1.0, and 3.2.0 release notes. xgboost.readthedocs.io/...changes
  19. ^American Statistical Association John M. Chambers Statistical Software Award. community.amstat.org/...john-m-chambers
  20. ^1 ^2XGBoost JVM packages and Apache Spark integration. xgboost.readthedocs.io/...jvm
  21. ^1 ^2 ^3ATLAS Collaboration. "Machine Learning Wins the Higgs Challenge." CERN, 2014. atlas.cern/...machine-learning-wins-higgs-challenge
  22. ^scikit-learn release history, "Version 0.11" (May 7, 2012). scikit-learn.org/...older_versions
  23. ^Hollmann, Noah, et al. "Accurate predictions on small data with a tabular foundation model." Nature 637 (2025): 319-326. nature.com/...s41586-024-08328-6
  24. ^1 ^2 ^3XGBoost NEWS.md (release notes before 2.1). github.com/...NEWS.md

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

8 revisions · v8 · 5,634 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check of the full article against its cited sources (130 claims, 34 sources fetched); 19 material + 24 minor fixed

Cite this page: AI Wiki. "XGBoost." aiwiki.ai, updated 28 Oct 2026, fact-checked 2 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/xgboost

Edit article