Splitter

See also: Machine learning terms

A splitter is a term used in two distinct senses in machine learning. The first and most common sense is a utility that partitions a dataset into subsets such as training, validation, and test sets, or into the folds of a cross-validation scheme. The second sense, used in the decision tree literature and in Google's machine learning glossary, refers to the routine inside a tree-learning algorithm that chooses the best condition at each internal node.^[7] Both meanings share the underlying idea of dividing data so that downstream learning is honest, reproducible, and accurate, but the algorithms involved are quite different. This article covers the data splitter first, then the decision-tree splitter.

Splitter as a data partitioner

A data splitter is a method or class that divides a dataset into subsets, typically a training set, a validation set, and a test set, or into k folds for cross-validation. Splitting matters because a model evaluated on the same data it was trained on will look better than it really is, often dramatically so when the model has high capacity. A held-out test set, untouched until the very end of model development, is the closest a practitioner can get to an honest estimate of generalization error.

Splitters do three jobs at once. They support honest model evaluation by reserving data the learning algorithm has never seen. They enable hyperparameter tuning, by giving an inner validation set or cross-validation loop separate from the final test set. And they prevent a class of bugs known as data leakage, in which information from the test set sneaks into training, sometimes through a careless preprocessing step, sometimes through ignoring grouping or temporal structure in the data.

Common splitting strategies

There is no single best splitting strategy; the right choice depends on the size of the dataset, the structure of the labels, the presence of grouping, and whether the data are temporal. The most common strategies form a small family of named procedures.

Scroll sideways for more →

The choice between these is rarely arbitrary. Stratification is essential when the positive class is rare, because a plain random split can produce folds with no positive examples at all, breaking metrics like recall and area under the precision-recall curve. Group splitters matter whenever the same entity appears in many rows, for example a patient with several scans or a user with several sessions; a random split would put part of the same patient's data in train and part in test, producing optimistic accuracy that does not survive deployment.

Time-series splitters enforce a different rule: the test set must come strictly after the training set in time. Random k-fold on time-stamped data is a classic data leakage trap, because the training fold can contain points from after the test fold and the model effectively peeks at the future. The scikit-learn TimeSeriesSplit and walk-forward variants like rolling-origin evaluation produce a sequence of expanding or sliding windows that mimic a real deployment in which a model is retrained periodically and used to predict the next interval.^[2]

In finance the temporal leakage problem is sharper still, because a label is often defined over an interval (for example the return realised between a trade entry and exit) rather than at a single instant, so a training observation can overlap in time with a test observation even when their timestamps differ. Marcos López de Prado's purged k-fold cross-validation addresses this by purging training samples whose label-formation window overlaps the test fold, and by embargoing a buffer of observations immediately after each test fold so that features cannot draw on data adjacent to the test period. The generalisation, combinatorial purged cross-validation, builds many train-test combinations under the same purge-and-embargo rules to produce a distribution of out-of-sample results rather than a single backtest path.^[19]

scikit-learn splitter classes

The scikit-learn library exposes splitters as iterators in sklearn.model_selection.^[1] Each class implements a split(X, y, groups) method that yields pairs of integer index arrays for the training and validation portions of each fold. A model selection routine such as cross_val_score or GridSearchCV accepts any of these objects as its cv argument, which makes swapping splitters trivial. When cv is passed as a plain integer, scikit-learn picks the splitter automatically: a stratified k-fold for classifiers and a plain k-fold for regressors, with a default of five folds.^[2]

Scroll sideways for more →

The official scikit-learn user guide for cross-validation walks through these classes with diagrams that show, for each splitter, which samples land in the training and validation sets across folds.^[3] The diagrams make the differences between, say, KFold and GroupKFold easy to see at a glance and are worth consulting before choosing a splitter for a new project.

TimeSeriesSplit deserves a closer look because its behaviour is governed by more than n_splits (default five). In the i-th split it returns the first folds as training and the next chunk as test, so successive training sets are supersets of earlier ones, giving an expanding window. Three further parameters tune it: max_train_size caps the training window so that very old data can be dropped (turning the expanding window into a sliding one), test_size fixes the number of test samples per fold, and gap (added in scikit-learn 0.24) drops a fixed number of samples between each training set and its test set, a lightweight way to imitate the embargo used in purged cross-validation.^[18]

The splitter object only produces indices; the surrounding helper decides what to do with them. cross_val_score returns one score per fold, cross_validate returns a dictionary that can include multiple metrics, fit and score times, and the fitted estimators, and cross_val_predict returns an out-of-fold prediction for every sample. The documentation warns that cross_val_predict is meant for visualisation or stacking, not for estimating generalisation error, because its aggregation differs from averaging per-fold scores.^[2]

Train/validation/test ratios

No official rule fixes the proportions of a hold-out split. Practitioners use a few defaults that have stood up in practice. An 80/20 split between train and test is common when no separate validation set is needed because hyperparameter tuning is done with k-fold cross-validation on the training portion. A 70/15/15 or 60/20/20 three-way split adds an explicit validation set and is a typical recipe for medium-sized datasets and deep learning. For very large datasets, the training fraction is often pushed higher (say 95/2.5/2.5) because even a small percentage corresponds to a statistically reliable test set. For very small datasets, k-fold cross-validation or leave-one-out is preferred to a single hold-out, because a single small test set is too noisy to trust.^[2]

Stratification and imbalanced data

Stratification matters most when the data are imbalanced. Suppose 1% of customers churn in a given month. A plain random split with 100 test points has an expected count of one churner, with a non-trivial probability of zero churners and therefore zero recall. Stratified splitters fix this by sorting examples by class and then sampling from each class separately, so each fold contains the same 1% positive rate as the full dataset. The same idea applies to multi-class classification; StratifiedKFold extends naturally to k classes by stratifying along the full label distribution.

Stratification is incompatible with strict temporal order, because shuffling and resorting by class breaks the chronology. For time-series classification, practitioners typically use a time-series split first and verify after the fact that the training and test windows have similar class distributions. If the class distribution drifts over time, stratification cannot save the model from concept drift; that is a separate problem.

Splitters and data leakage

Many high-profile errors in published machine-learning results trace back to a poorly chosen splitter. The most common patterns are: random splitting time-series data so the model peeks at future values, ignoring patient or user grouping so the same person appears in train and test, and applying preprocessing such as scaling or feature selection to the full dataset before splitting, so the test set influences the training pipeline.^[16]

A correctly chosen splitter is the first defence against these bugs. Group-aware splitters prevent same-entity leakage; time-series splitters prevent future-information leakage; and pipeline objects such as scikit-learn's Pipeline ensure that all preprocessing is fit on the training fold only.^[2] The combination of the right splitter and a pipeline is the standard recipe for honest cross-validation in modern Python machine learning.

Nested cross-validation and selection bias

A subtle trap appears when the same split is used both to tune hyperparameters and to report the final score. Tuning is itself a form of fitting: a search over many candidate configurations will eventually find one that happens to score well on the validation folds, so the validation score becomes an optimistic estimate of true performance. Gavin Cawley and Nicola Talbot documented this selection bias in performance evaluation and argued that the variance of the model-selection criterion, not just its bias, drives the over-fitting, which is why low-variance estimates such as repeated cross-validation help.^[17]

The standard remedy is nested cross-validation. An outer splitter holds out a test fold used only for scoring, while an inner splitter, run on each outer training fold, performs the hyperparameter search. Because the outer test folds never participate in tuning, the averaged outer score is an almost unbiased estimate of how the whole pipeline, including its model-selection step, would generalise. In scikit-learn this is expressed by wrapping a GridSearchCV (which carries the inner splitter) inside a cross_val_score call that carries the outer splitter.^[2]

Splitter inside a decision tree

The second meaning of splitter comes from the decision tree literature. Google's Decision Forests glossary defines a splitter as the routine and algorithm responsible for finding the best condition at each node while training a decision tree.^[7] In other words, the splitter is the inner-loop search that, given a node and the data routed to it, picks a feature and a threshold (or a feature and a category set, or a hyperplane) that splits the node's data into two child nodes in a way that improves a chosen criterion. In the glossary's vocabulary a condition (also called a test or simply a split) is any internal node that performs a test, so the splitter is precisely the procedure that manufactures conditions during training.^[7]

A decision tree learning algorithm is built around two nested loops. The outer loop grows the tree, deciding which leaves to expand and when to stop. The inner loop, the splitter, scans the candidate splits at each leaf and picks one. Almost every difference between popular tree algorithms (CART, ID3, C4.5, Extra Trees, LightGBM, XGBoost) lives in this inner loop: which candidate splits are considered, what criterion is used to score them, and how ties and edge cases are handled.

Split criteria

The scoring function inside the splitter is called the split criterion. For classification trees, the criterion measures the impurity of a node and the algorithm chooses the split that reduces the weighted impurity of the children by the largest amount. For regression trees, the criterion measures variance or some loss-derived quantity around the node's mean prediction. Writing p_mk for the proportion of class k among the n_m samples at node m, scikit-learn defines Gini impurity as the sum over classes of p_mk(1 minus p_mk), which is algebraically 1 minus the sum of the squared class proportions, and entropy as the negative sum of p_mk times log p_mk. For regression it uses the within-node mean and reports mean squared error as the average squared deviation from that mean, half the mean Poisson deviance for count targets, and the mean absolute deviation from the node median for the absolute-error criterion.^[6]

Scroll sideways for more →

Gini impurity ranges from 0 (a node where every sample has the same label) to 1 minus 1/k for k equally distributed classes (0.5 in the binary case). Shannon entropy ranges from 0 to log k. The two criteria almost always pick the same split in practice, and the choice between them is more about computational cost (Gini avoids logarithms and is slightly faster) than statistical accuracy. The scikit-learn documentation notes that using entropy as the node-splitting criterion is equivalent to minimising the log loss, which is why it exposes the two under the same numerical value.^[6] Friedman MSE is a special-purpose criterion designed for additive regression trees in gradient boosting; it maximises the squared mean difference between children, weighted by their sizes, instead of using the standard MSE reduction.^[12] The set of criteria available is documented per estimator in the DecisionTreeClassifier and DecisionTreeRegressor references.^[4][5]

Split-finding algorithms

Given a criterion, several algorithms compete to find a good split efficiently. The differences between them dominate the runtime characteristics of modern tree libraries.

Scroll sideways for more →

The exact greedy algorithm tries every value as a candidate threshold for every feature at every node. It is the textbook description of CART and scales as O(features times samples) per split, which is fine for small data and prohibitive for large data. Histogram-based splitters trade a small amount of accuracy for speed by binning continuous features into, by default, 255 buckets (max_bin=255 in LightGBM) and scanning only the bucket boundaries. The trick is that once the per-bin gradient and Hessian sums are computed, scoring all candidate splits for one feature is O(bins) rather than O(samples). LightGBM also exploits histogram subtraction, obtaining one child's histogram by subtracting its sibling's histogram from the parent's, so only the smaller child needs to be built directly.^[15] These optimisations are the reason histogram-based gradient boosting is the default choice on tabular data with millions of rows.

XGBoost's approximate greedy algorithm uses a different idea, the weighted quantile sketch introduced in the original XGBoost paper by Chen and Guestrin.^[13] Instead of binning into fixed-width buckets, it places candidate thresholds at quantiles of the feature distribution weighted by the second-order gradient (the Hessian) so that each bucket carries roughly equal loss curvature. This produces accurate splits even when the feature distribution is skewed and is provably mergeable across distributed shards. The same paper introduces the sparsity-aware split finding that the table above lists, in which each split learns a default direction for missing or zero entries by trying both branches and keeping whichever yields the higher gain; the authors report this sparse path runs roughly fifty times faster than a dense scan.^[13]

A property worth keeping in mind is that all of these splitters are heuristics. Learning a globally optimal decision tree is NP-complete under several notions of optimality, so practical algorithms make a locally optimal choice at each node and never reconsider it, which is exactly what the greedy and approximate splitters above do. Growing many trees in an ensemble is the usual way to soften the resulting suboptimality.^[6]

Random splitters and Extra Trees

The random splitter is a deliberate weakening of the exact greedy algorithm. Instead of scanning every threshold, it draws a single random threshold for each feature in the candidate set and picks the best of those random thresholds. The motivation is variance reduction in an ensemble: when many random trees are aggregated, the noise introduced by random thresholds averages out, while the trees become less correlated and the ensemble's variance falls.

Extremely Randomized Trees, introduced by Pierre Geurts, Damien Ernst, and Louis Wehenkel in their 2006 Machine Learning paper of the same name, take this idea further.^[11] At each node, Extra Trees draw a random subset of features (like random forest) and then a single random threshold per feature, and pick the best of those random splits using the standard impurity criterion. The result is faster training (no full sort per feature) and often comparable accuracy to a random forest, especially when the dataset is noisy and the decorrelation benefits dominate. In scikit-learn, ExtraTreesClassifier and ExtraTreesRegressor implement this algorithm, and the underlying ExtraTreeClassifier corresponds to DecisionTreeClassifier(splitter="random").^[4]

scikit-learn's splitter parameter

Scikit-learn's DecisionTreeClassifier and DecisionTreeRegressor expose the choice of splitting algorithm through a splitter parameter with two settings:^[4][5]

Scroll sideways for more →

A standalone DecisionTreeClassifier(splitter="random") is rarely competitive with the default, but it shines inside an ensemble such as BaggingClassifier or ExtraTreesClassifier, where decorrelation between trees boosts the aggregate accuracy. The criterion parameter (Gini, entropy, log loss, MSE, Friedman MSE, MAE, Poisson) is independent of the splitter parameter and chooses the scoring function the splitter uses.^[4]

The choice also changes the cost of training. The scikit-learn documentation gives a worst-case total training cost of O(n_features times n_samples squared times log n_samples) for the "best" splitter and O(n_features times n_samples squared) for "random", the difference being the per-node sort that the exact search needs and the random search skips. Real implementations cache the sorted order of indices so that features are not re-sorted at every node, which brings the practical cost down to roughly O(n_features times n_samples times log n_samples).^[6] Two further parameters interact closely with the splitter: max_features restricts how many features the splitter may consider at each node (the mechanism behind random-forest-style feature subsampling), and min_impurity_decrease rejects any split whose weighted impurity reduction falls below a threshold, both defaulting to no restriction.^[4]

For a deeper treatment of how thresholds are chosen for numerical features, see threshold (decision trees), which discusses tie-breaking, missing-value handling, and how histogram-based libraries propose candidate thresholds.

Splitters in CART, ID3, and C4.5

The historical decision-tree algorithms differ mainly in their splitters. The CART algorithm, introduced by Breiman, Friedman, Olshen, and Stone in 1984, uses exact greedy splitting with Gini impurity for classification and MSE for regression, and produces strictly binary trees.^[8] ID3, introduced by Quinlan in 1986, uses information gain with entropy and produces multiway splits on categorical features.^[9] C4.5, Quinlan's 1993 extension of ID3, uses gain ratio (information gain divided by split entropy) to penalise high-cardinality features, supports continuous attributes via threshold splits, and includes pessimistic error pruning.^[10] A fourth classical family, CHAID (chi-square automatic interaction detection), was published by Gordon Kass in 1980 and takes a statistical rather than impurity-based approach: its splitter merges and splits predictor categories by the adjusted significance of a chi-square test (with a Bonferroni correction), it grows multiway rather than binary trees, and a continuous-target variant was originally known as XAID.^[20] The differences are entirely inside the splitter and the tree construction loop, not in the prediction logic, which is identical: traverse the tree, return the leaf prediction.

Modern gradient-boosted tree libraries inherit from this lineage but have re-engineered the splitter for scale. LightGBM adds gradient-based one-side sampling (GOSS) on top of histogram splitting: it keeps the instances with the largest gradients and randomly subsamples the rest, then re-weights the sampled small-gradient instances by the factor (1 minus a) divided by b so that the information-gain estimate stays unbiased, where a is the top-gradient fraction and b the sampling rate (defaults top_rate=0.2 and other_rate=0.1, so each tree trains on roughly thirty percent of the rows).^[14] LightGBM's second trick, exclusive feature bundling, packs features that are rarely nonzero at the same time, such as one-hot columns, into a single bundle to cut the effective feature count, an idea the paper shows is NP-hard to optimise but cheap to approximate greedily; together these let LightGBM match conventional gradient boosting while training up to twenty times faster.^[14] XGBoost adds the weighted quantile sketch and a sparsity-aware default direction for missing values;^[13] gradient boosting frameworks such as CatBoost re-engineer the splitter differently again, growing oblivious (symmetric) trees that apply the same feature and threshold at every node of a given depth, and pairing them with ordered boosting and ordered target statistics that compute each row's statistics from earlier rows in a random permutation to avoid the prediction-shift form of target leakage.^[21]

Two meanings, one underlying idea

The two senses of splitter, the data partitioner and the tree-node split-finder, share a common premise: machine learning needs principled rules for dividing data so that what comes out of the model is honest. The data splitter divides examples between training and evaluation; the tree splitter divides examples between branches of a tree. Both have to fight against shortcuts that look attractive in the short term and break the model in the long term, whether that is a leaky split that flatters the test metric or a greedy threshold that overfits a noisy feature. The right choice in either case depends on the structure of the data, the size of the dataset, and the downstream use of the model.

References

1. Pedregosa, F. et al. (2011). Scikit-learn: Machine Learning in Python. *Journal of Machine Learning Research*, 12, 2825-2830. https://jmlr.org/papers/v12/pedregosa11a.html ↩
2. scikit-learn developers. Cross-validation: evaluating estimator performance (user guide). https://scikit-learn.org/stable/modules/cross_validation.html ↩
3. scikit-learn developers. Visualizing cross-validation behavior in scikit-learn. https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html ↩
4. scikit-learn developers. DecisionTreeClassifier reference documentation. https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html ↩
5. scikit-learn developers. DecisionTreeRegressor reference documentation. https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html ↩
6. scikit-learn developers. Decision Trees (user guide). https://scikit-learn.org/stable/modules/tree.html ↩
7. Google for Developers. Machine Learning Glossary: Decision Forests, entry for *splitter*. https://developers.google.com/machine-learning/glossary/df ↩
8. Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). *Classification and Regression Trees*. Wadsworth. ↩
9. Quinlan, J. R. (1986). Induction of decision trees. *Machine Learning*, 1(1), 81-106. ↩
10. Quinlan, J. R. (1993). *C4.5: Programs for Machine Learning*. Morgan Kaufmann. ↩
11. Geurts, P., Ernst, D., & Wehenkel, L. (2006). Extremely randomized trees. *Machine Learning*, 63(1), 3-42. https://link.springer.com/article/10.1007/s10994-006-6226-1 ↩
12. Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. *Annals of Statistics*, 29(5), 1189-1232. ↩
13. Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. *KDD '16*. https://arxiv.org/abs/1603.02754 ↩
14. Ke, G. et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. *NeurIPS 2017*. https://proceedings.neurips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf ↩
15. LightGBM developers. Features documentation, including histogram-based learning. https://lightgbm.readthedocs.io/en/latest/Features.html ↩
16. AWS Prescriptive Guidance. Splits and data leakage. https://docs.aws.amazon.com/prescriptive-guidance/latest/ml-operations-planning/splits-leakage.html ↩
17. Cawley, G. C., & Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. *Journal of Machine Learning Research*, 11, 2079-2107. https://www.jmlr.org/papers/v11/cawley10a.html ↩
18. scikit-learn developers. TimeSeriesSplit reference documentation. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html ↩
19. López de Prado, M. (2018). *Advances in Financial Machine Learning*. John Wiley & Sons. (Purged and combinatorial purged cross-validation.) https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086 ↩
20. Kass, G. V. (1980). An exploratory technique for investigating large quantities of categorical data. *Journal of the Royal Statistical Society, Series C (Applied Statistics)*, 29(2), 119-127. https://en.wikipedia.org/wiki/Chi-square_automatic_interaction_detection ↩
21. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. *NeurIPS 2018*. https://arxiv.org/abs/1706.09516 ↩

Explain Like I'm 5 (ELI5)

Imagine you want to teach your little robot to recognize different animals. You have lots of pictures of animals to help your robot learn. To make sure your robot really knows its stuff, you need to test it using some of the pictures, but pictures the robot has never seen before.

A splitter in machine learning is like a helper who organizes the pictures into groups for teaching and testing the robot. The helper makes sure that the robot sees a variety of animals in each group, so it learns how to recognize all of them properly. Once the robot has seen many different groups of pictures, you can be more confident that it can recognize animals it hasn't seen before.

There is also a second kind of splitter inside a special model called a decision tree. A decision tree is like a game of twenty questions: at every step it asks one yes-or-no question about the picture, like "does it have whiskers?" The splitter is the part of the program that decides which question to ask at each step, by trying lots of possible questions and picking the one that does the best job of separating cats from dogs from rabbits. Both kinds of splitters share the same goal: chopping data into smaller pieces in a smart way, so the robot can learn from the pieces and we can check it has really learned.