CtrlK
Feature selection is the process of selecting a subset of relevant input variables (features) from a larger pool of candidates for use in machine learning model construction. The goal is to identify the smallest set of features that retains, or even improves, predictive performance while reducing computational cost, mitigating overfitting, and producing models that humans can more easily interpret. Feature selection sits at the core of the broader practice of feature engineering and is closely related to dimensionality reduction, although the two are not the same. Feature selection keeps a subset of the original variables intact, whereas dimensionality reduction techniques such as principal component analysis project the data into a new coordinate system whose axes are linear or nonlinear combinations of the originals.
The topic was popularised in machine learning by Isabelle Guyon and André Elisseeff in their 2003 Journal of Machine Learning Research survey, "An Introduction to Variable and Feature Selection," which framed selection as a way to improve prediction accuracy, lower inference cost, and gain insight into the data-generating process. The Guyon and Elisseeff paper still serves as the canonical reference and has been cited more than fifteen thousand times. Earlier foundational work by Ron Kohavi and George John in 1997, "Wrappers for Feature Subset Selection," formalised the wrapper paradigm in which selection is treated as a search problem guided by an actual learner's cross-validated score.
In the modern era of large neural networks, explicit feature selection is sometimes assumed to be obsolete because deep models perform end-to-end representation learning. That perception is only partially correct. For unstructured data such as images, text, audio, and video, learned representations have indeed displaced hand-crafted feature pipelines. For structured tabular data, however, where gradient-boosted decision trees such as XGBoost, LightGBM, and CatBoost still dominate Kaggle leaderboards and production pipelines, careful feature selection remains a high-leverage activity. Recent benchmarks across diverse tabular datasets continue to show that boosted trees with well-engineered, well-selected features match or beat large transformer-based tabular models on the majority of problems.
When the number of input variables grows, models become harder to estimate, slower to train and serve, and more prone to memorising noise. The classical motivations for selecting a small subset of features are summarised below.
Richard Bellman coined the phrase curse of dimensionality in 1957 to describe the geometric and statistical pathologies that arise when the number of dimensions grows. As the dimensionality d of the input space increases, the volume of the space grows exponentially, and any fixed sample of points becomes increasingly sparse. Distances between points become more uniform, which weakens distance-based methods such as k-nearest neighbours and clustering. The number of training examples required to densely sample a d-dimensional space grows exponentially in d, a fact that limits any nonparametric learner.
A closely related result is the Hughes phenomenon, which states that for a fixed training-set size the expected accuracy of a classifier first rises with the number of features and then, beyond some peak dimensionality, falls again. The peak occurs because each additional feature carries useful signal up to a point but then begins to inject more noise and estimation variance than information. Feature selection is one of the simplest tools for keeping a model on the favourable side of that peak.
The bias-variance decomposition makes the cost of irrelevant features concrete. With too few features the model is biased and underfits; with too many it has high variance and overfits the training set. Each parameter in a high-dimensional model must be estimated from finite data, so the estimate's variance grows with the number of parameters. By eliminating features that contribute no signal, feature selection reduces the effective parameter count, lowers variance, and often improves out-of-sample accuracy. This rationale aligns with Occam's razor: among models that explain the data equally well, the simpler one tends to generalise better.
Fewer features means smaller models, faster training, lower inference latency, and reduced storage. In production systems where features must be computed at request time from upstream services, every dropped feature also reduces the number of network calls and feature-store lookups. Sparse models are also easier to debug and to explain to non-technical stakeholders, which is increasingly important in regulated domains such as finance, healthcare, and credit scoring where model decisions must be auditable.
Feature selection and feature extraction are often confused. Feature selection chooses a subset of the existing variables and leaves them otherwise untouched, so the chosen variables retain their original semantics and units. Feature extraction transforms the input space into a new one. PCA, autoencoders, kernel projections, and word embeddings are extraction methods because each output dimension is a function of many inputs. Selection preserves interpretability at the cost of being limited to whatever signal already exists in the raw inputs, while extraction can synthesise new informative coordinates at the cost of opaqueness. In practice the two are often combined: extract a wide set of candidate features, then select a sparse, predictive subset.
The community generally groups feature selection algorithms into three families: filter methods, wrapper methods, and embedded methods. A fourth, hybrid methods, blends elements of the others. The taxonomy was crystallised by the Kohavi and John paper and refined by Guyon and Elisseeff.
| Property | Filter | Wrapper | Embedded |
|---|---|---|---|
| Selection signal | Statistical score independent of any learner | Cross-validated score of an actual learner | Score produced as a side effect of model training |
| Computational cost | Lowest, scales to millions of features | Highest, requires fitting many models | Moderate, one model fit per regularisation path |
| Captures feature interactions | Mostly no, univariate by default | Yes, evaluates subsets jointly | Partially, depends on model class |
| Risk of overfitting the selection step | Low | High when the number of candidate subsets is large | Low to moderate, controlled by regularization |
| Coupling to downstream model | Loose, the same selection works for many models | Tight, results are tuned to one learner | Tight, baked into the learner |
| Typical examples | Pearson correlation, chi-square, ANOVA F-test, mutual information, variance threshold | Forward selection, backward elimination, recursive feature elimination, exhaustive search | L1 (Lasso) regression, Elastic Net, tree-based feature importance, SHAP-guided pruning |
| Best when | Quick first-pass screening on very wide data | Small to medium feature counts and a fixed downstream model | The same model is used for both selection and final prediction |
Filter methods rank features using a statistic computed directly from the data, with no reference to a downstream learner. Because they ignore feature interactions, they are fast and embarrassingly parallel, which makes them the workhorse for the first pass over very wide datasets such as gene-expression matrices, sparse text-document term matrices, or click-stream logs with thousands of one-hot encoded categorical levels.
Common filters include:
f_classif for classification and f_regression for regression.A pragmatic filter pipeline first drops zero-variance and duplicate columns, then ranks the survivors by a target-aware score such as mutual information, and finally keeps a top fraction such as the top 10 percent for downstream evaluation.
Wrapper methods treat feature selection as a search problem in which each candidate subset is scored by training a learner and measuring its cross-validated performance. They are conceptually appealing because the score they optimise is the metric the user actually cares about, and they detect interactions because subsets are evaluated jointly. The trade-off is computational: with p features there are 2 to the power p possible subsets, so exhaustive search is feasible only for very small p. Practical wrappers therefore use heuristic search.
Widely used wrapper algorithms include:
Wrappers can overfit the selection process itself when the number of candidate subsets evaluated approaches or exceeds the number of training examples. Nested cross-validation, in which feature selection runs inside an inner loop and an outer loop estimates generalisation, is the standard guard against optimistic bias.
Embedded methods perform selection as part of the model-fitting procedure itself. They are usually cheaper than wrappers because the selection signal is produced for free during a single model fit, and they capture feature interactions through whatever mechanism the underlying learner uses.
The most widely used embedded approaches are:
shap-select formalise this with statistical significance tests.Hybrid pipelines combine families to balance speed and accuracy. A typical recipe is to use a cheap filter to shrink the feature pool from tens of thousands to hundreds, then run a wrapper such as RFE on the survivors, and finally fit an embedded model such as Lasso on the chosen subset for the production model. The Boruta algorithm, described later, is sometimes called a hybrid because it wraps a random forest in a statistical test.
The table below summarises algorithms practitioners reach for most often, along with the family they belong to and a representative software entry point.
| Algorithm | Family | Strengths | Limitations | Representative implementation |
|---|---|---|---|---|
| Variance threshold | Filter | Removes constants, very fast | Ignores the target | sklearn.feature_selection.VarianceThreshold |
| Pearson correlation ranking | Filter | Simple, interpretable | Linear only | pandas.DataFrame.corr |
| Chi-square test | Filter | Sound for non-negative categorical features | Requires binning of numeric features | sklearn.feature_selection.chi2 |
| ANOVA F-test | Filter | Cheap, principled for numeric vs categorical | Assumes equal variance, linear separability | sklearn.feature_selection.f_classif |
| Mutual information | Filter | Captures nonlinear dependence | Higher sample complexity | sklearn.feature_selection.mutual_info_classif |
| Forward sequential selection | Wrapper | Detects interactions, returns small subsets | O(p^2) model fits | sklearn.feature_selection.SequentialFeatureSelector |
| Backward elimination | Wrapper | Strong when most features are useful | Cannot start when p is huge | SequentialFeatureSelector(direction='backward') |
| Recursive Feature Elimination | Wrapper | One fit per round, scales to thousands | Needs coef_ or feature_importances_ | sklearn.feature_selection.RFE |
| RFE with cross-validation | Wrapper | Picks the subset size automatically | More expensive than RFE | sklearn.feature_selection.RFECV |
| Exhaustive search | Wrapper | Globally optimal for the chosen metric | Combinatorial cost | mlxtend.feature_selection.ExhaustiveFeatureSelector |
| Lasso (L1) | Embedded | Sparse, interpretable, statistically grounded | Linear, can drop correlated peers | sklearn.linear_model.Lasso |
| Elastic Net | Embedded | Handles correlated groups | Two hyperparameters to tune | sklearn.linear_model.ElasticNet |
| Tree feature importance | Embedded | Cheap, captures interactions | MDI biased toward high-cardinality features | xgboost.Booster.get_score, lightgbm.Booster.feature_importance |
| Permutation importance | Embedded | Model-agnostic, low bias | Expensive | sklearn.inspection.permutation_importance |
| SHAP-guided pruning | Embedded | Theoretically grounded, model-agnostic | SHAP can be slow on non-tree models | shap.TreeExplainer, shap-select |
| Boruta | Hybrid | All-relevant, principled cut-off | Slow on wide data | boruta_py (scikit-learn-contrib) |
Boruta, introduced by Miron Kursa and Witold Rudnicki in 2010, takes the all-relevant view of selection: it tries to find every feature that carries any predictive signal, rather than the smallest sufficient subset. It works by duplicating each feature, randomly shuffling the values in the copy to destroy any relationship with the target, and concatenating the shuffled shadow features to the original matrix. A random forest is then fit, and each real feature's importance is compared with the maximum importance achieved by any shadow feature. A real feature that beats the best shadow significantly more often than expected by chance is confirmed as relevant; one that consistently loses is rejected. The procedure repeats until every feature has been classified or a maximum number of iterations is reached. Boruta is popular on Kaggle and is well suited to medium-width tabular datasets where any subtle predictive signal matters.
SHAP (SHapley Additive exPlanations) values, introduced by Scott Lundberg and Su-In Lee in 2017, allocate each prediction's deviation from the model's mean prediction to the input features in a manner that satisfies the Shapley fairness axioms from cooperative game theory. The mean absolute SHAP value of a feature across a representative sample of predictions is a model-agnostic, theoretically principled importance score. Pruning workflows compute SHAP values, drop features whose mean absolute SHAP value falls below a threshold, retrain, and verify that performance is preserved. The newer shap-select library combines SHAP scoring with logistic-regression significance tests to choose a subset automatically.
Most practical Python feature-selection work uses scikit-learn. The library exposes selectors as transformers with a uniform fit, transform, and get_support interface so they slot into Pipeline objects and GridSearchCV cleanly.
| Class or function | Module | Family | Typical use |
|---|---|---|---|
VarianceThreshold(threshold) | sklearn.feature_selection | Filter | Drops low-variance features such as one-hot columns of rare categories |
SelectKBest(score_func, k) | sklearn.feature_selection | Filter | Keeps the top k features by a univariate score such as chi2, f_classif, mutual_info_classif |
SelectPercentile(score_func, percentile) | sklearn.feature_selection | Filter | Same as SelectKBest but expressed as a percentage |
SelectFpr, SelectFdr, SelectFwe | sklearn.feature_selection | Filter | Univariate selection at a target false-positive, false-discovery, or family-wise error rate |
GenericUnivariateSelect | sklearn.feature_selection | Filter | Configurable wrapper over the above |
chi2, f_classif, f_regression, mutual_info_classif, mutual_info_regression, r_regression | sklearn.feature_selection | Filter | Score functions consumed by the univariate selectors |
RFE(estimator, n_features_to_select, step) | sklearn.feature_selection | Wrapper | Recursive feature elimination with a chosen subset size |
RFECV(estimator, cv, scoring) | sklearn.feature_selection | Wrapper | RFE that tunes the subset size by cross-validation |
SequentialFeatureSelector(estimator, n_features_to_select, direction, scoring, cv) | sklearn.feature_selection | Wrapper | Greedy forward or backward selection that does not require coef_ or feature_importances_ |
SelectFromModel(estimator, threshold) | sklearn.feature_selection | Embedded | Keeps features whose absolute coefficient or importance is above a threshold; works with Lasso, Elastic Net, SVMs with L1 loss, and tree ensembles |
permutation_importance(estimator, X, y) | sklearn.inspection | Embedded | Computes permutation importance for any fitted estimator |
A worked example that combines these classes might filter low-variance columns, rank the survivors with mutual information, and then run RFECV with a logistic-regression base learner inside a Pipeline. Wrapping the whole pipeline in GridSearchCV lets the cross-validation procedure choose both the filter cut-off and the final subset size jointly, avoiding the data leakage that arises when selection is performed on the full training set before splitting.
Several specialised libraries extend the core ecosystem:
A defensible feature-selection workflow follows a few simple rules.
Feature selection sits at the intersection of statistics, information theory, and combinatorial optimisation. Several theoretical results frame what selection can and cannot achieve.
Deep neural networks blur the line between feature engineering, selection, and extraction. Hidden layers learn distributed representations that subsume much of the work that previously belonged to hand-crafted features, and convolutional, recurrent, and attention-based architectures discover useful local statistics on their own. For unstructured modalities such as image classification, machine translation, and speech recognition, end-to-end representation learning is now standard and explicit feature selection is rarely useful.
For structured data the picture is more nuanced. Tabular benchmarks such as the work by Shwartz-Ziv and Armon (2021), and follow-up surveys, repeatedly find that gradient-boosted trees match or exceed deep tabular models such as TabNet and FT-Transformer on the majority of public datasets, especially when training samples are limited. In those settings, careful feature selection is still one of the highest-return investments a practitioner can make.
Within deep learning itself, several techniques recover some of the benefits of selection without exiting the differentiable framework. L1 penalties on input-layer weights drive entire input columns to zero; group Lasso applied across input neurons does the same at the column level. Concrete dropout and the L0 regularisation of Louizos and colleagues add learnable gates that produce truly sparse input usage. Attention weights over input tokens can be inspected post hoc as a soft selection mechanism, and integrated gradients or DeepLIFT scores serve a similar role to SHAP for tree ensembles.
Feature selection chooses a small, useful subset of input variables to feed to a model, balancing predictive accuracy against complexity, latency, and interpretability. The field's three classical families - filters, wrappers, and embedded methods - each trade computational cost for selection quality differently, and modern hybrid pipelines combine them to scale gracefully from a handful of features to millions. While deep learning has reduced the need for manual feature engineering on unstructured data, feature selection remains central to tabular machine learning, where boosted trees still dominate and every dropped feature pays dividends in cost, latency, and clarity. The discipline rests on a long lineage of statistical and information-theoretic results, summarised most influentially by Guyon and Elisseeff in 2003, and continues to evolve through tools such as Boruta, SHAP-guided pruning, and shap-select.