CtrlK
Feature engineering is the process of using domain knowledge to create, transform, and select features from raw data so that machine learning models can learn more effectively. It sits at the core of applied machine learning and is often the single largest factor separating a mediocre model from a highly accurate one. As Andrew Ng famously stated: "Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering."
While raw data can sometimes be fed directly into an algorithm, most real-world datasets contain noise, irrelevant columns, incompatible scales, or implicit patterns that a model cannot detect on its own. Feature engineering bridges the gap between raw data and the mathematical representations that algorithms require.
The choice of features has a more direct impact on model performance than the choice of algorithm in many practical settings. A simple logistic regression trained on well-crafted features can outperform a complex neural network trained on poorly prepared inputs. Several factors explain this outsized influence:
Kaggle competitions have repeatedly demonstrated this principle. Top competitors regularly report that feature engineering, not hyperparameter tuning, accounts for the largest share of their final score improvement.
A feature (also called an attribute, variable, or predictor) is a measurable property of the phenomenon being modeled. In a tabular dataset, features correspond to columns and individual observations correspond to rows. For a house-price prediction task, typical features include the number of bedrooms, total living area in square feet, year built, and neighborhood.
Not all features carry useful information. Some are redundant (highly correlated with another feature), some are noisy (contain mostly random variation), and some are irrelevant (have no relationship with the target). A central goal of feature engineering is to maximize the ratio of informative features to uninformative ones.
Different data types require different engineering strategies. The table below summarizes the most common transformation families.
| Data type | Common transformations | When to use |
|---|---|---|
| Numerical | Scaling, normalization, log transform, Box-Cox, binning, polynomial features | Continuous measurements with skewed distributions or differing scales |
| Categorical | One-hot encoding, label encoding, target encoding, frequency encoding | Nominal or ordinal variables with a limited or large set of categories |
| Text | Bag of words, TF-IDF, word embeddings, n-grams | Unstructured text fields such as reviews, descriptions, or titles |
| Date and time | Component extraction, cyclical sine/cosine encoding, time deltas | Timestamps, event logs, scheduling data |
| Geospatial | Haversine distance, clustering, proximity to landmarks | Latitude/longitude coordinates, postal codes |
Numerical features are the most straightforward to work with, but they still benefit from careful preprocessing.
Scaling and standardization. Many algorithms, including linear models, support vector machines, and neural networks, are sensitive to the absolute scale of input features. Min-max scaling rescales values to a fixed range (typically 0 to 1), while z-score standardization centers values around zero with unit variance. Tree-based models such as random forests and gradient boosting are invariant to monotonic transformations and generally do not require scaling.
Log and power transforms. Skewed distributions compress most values into a narrow range while a long tail stretches out in one direction. Applying a logarithmic transformation (or the more general Box-Cox or Yeo-Johnson transforms) pulls the tail inward, producing a distribution closer to normal. This often improves the performance of models that assume normally distributed inputs.
Binning (discretization). Continuous values can be grouped into discrete bins. For example, age might be split into brackets such as 18 to 25, 26 to 35, and so on. Binning can smooth out noise and reduce the influence of outliers. Scikit-learn provides KBinsDiscretizer for this purpose, supporting uniform, quantile, and k-means strategies.
Polynomial and interaction features. Creating squared terms, cubed terms, or products of two features allows linear models to capture nonlinear relationships. Scikit-learn's PolynomialFeatures transformer systematically generates all combinations up to a specified degree. However, the number of generated features grows rapidly, so this technique works best when combined with feature selection.
Categorical variables represent discrete groups or labels. Different encoding strategies suit different situations.
One-hot encoding. Each category becomes a separate binary column. This is the safest default for nominal (unordered) categories because it introduces no artificial ordinal relationship. The downside is that high-cardinality features (such as zip codes with thousands of unique values) produce very wide, sparse matrices.
Label encoding. Each category is assigned a unique integer. This is appropriate for ordinal variables where the categories have a natural order (for example, "low", "medium", "high"). For nominal variables, label encoding can mislead the model into treating the integers as having meaningful magnitude.
Target encoding (mean encoding). Each category is replaced with the mean of the target variable for observations in that category. Target encoding is effective for high-cardinality features but introduces a risk of data leakage. Regularization techniques such as smoothing or leave-one-out encoding help mitigate this risk.
Frequency encoding. Each category is replaced with its frequency (count or proportion) in the training set. This approach is useful when the prevalence of a category correlates with the target. It produces a single numeric column regardless of cardinality.
| Encoding method | Handles high cardinality | Preserves ordinality | Risk of data leakage | Typical use case |
|---|---|---|---|---|
| One-hot | No (creates many columns) | No | None | Nominal features with few categories |
| Label | Yes | Yes | None | Ordinal features |
| Target | Yes | No | Moderate (needs regularization) | High-cardinality nominal features |
| Frequency | Yes | No | Low | High-cardinality features correlated with target frequency |
Unstructured text must be converted into numeric vectors before a model can process it.
Bag of words (BoW). The simplest text representation counts the occurrences of each word in a document, ignoring word order and grammar. The result is a sparse vector whose length equals the vocabulary size.
TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF extends bag of words by weighting each term according to how common it is across the entire corpus. Words that appear in many documents (such as "the" or "is") receive low weights, while words that are distinctive to a few documents receive high weights. TF-IDF remains widely used in search engines, text classification, and keyword extraction.
N-grams. Instead of treating each word independently, n-grams consider sequences of n consecutive words. Bigrams (n=2) can capture compound terms like "New York" or "machine learning" that a unigram model would split into separate tokens.
Word embeddings. Dense, low-dimensional vector representations learned from large text corpora. Word2Vec, GloVe, and FastText are among the most widely used pre-trained embedding models. Unlike bag of words, embeddings capture semantic similarity: words with similar meanings are mapped to nearby points in the vector space. More recent transformer-based models such as BERT produce contextualized embeddings where the same word receives different vectors depending on surrounding context.
Timestamps contain implicit patterns that raw numeric representations obscure.
Component extraction. Extracting the hour, day of week, month, quarter, or year from a timestamp creates individual features that capture periodic patterns. For example, retail sales tend to spike on weekends and during holiday months.
Cyclical encoding. Hours, days, and months are cyclical: hour 23 is close to hour 0, and December is close to January. Encoding these features as sine and cosine pairs preserves this circular relationship. For a feature with period P, the transformation is: sin(2 * pi * x / P) and cos(2 * pi * x / P). This ensures that the Euclidean distance between the encoded representations of 23:00 and 00:00 is small, matching the intuitive closeness of those times.
Time deltas. Computing the elapsed time between events (for example, days since last purchase, seconds between page views) often reveals behavioral patterns that static timestamps miss.
Location data in the form of latitude and longitude coordinates requires specialized treatment.
Distance calculations. The Haversine formula computes the great-circle distance between two points on the Earth's surface. Features like "distance to city center" or "distance to nearest hospital" are common in real estate, logistics, and ride-hailing models.
Spatial clustering. Applying k-means or DBSCAN to coordinates groups nearby points into clusters, creating a categorical "region" feature that can capture local effects.
Proximity counts. Counting the number of points of interest within a given radius (for example, restaurants within 500 meters) provides a density signal useful for property valuation and urban analytics.
Beyond transforming existing columns, practitioners often create entirely new features by combining or aggregating raw data.
Interaction features capture joint effects that individual features cannot express. Multiplying "house size" by "number of floors" gives "total floor area," a feature that may be more predictive than either input alone. Ratio features (feature A divided by feature B) and difference features (feature A minus feature B) are also common. In marketing analytics, dividing total revenue by number of transactions gives average order value, a feature with strong predictive power for customer lifetime value.
The most powerful features often come from deep understanding of the problem domain rather than generic transformations. Examples include:
Collaborating with subject matter experts is one of the most reliable ways to discover high-value domain features that automated tools would miss.
When working with relational data (for example, a customer table joined to a transactions table), aggregation features summarize child records for each parent entity. Common aggregations include count, sum, mean, min, max, and standard deviation. A customer's average transaction amount over the past 90 days is a typical aggregation feature in credit scoring.
After generating a large pool of candidate features, selecting the most informative subset prevents overfitting and speeds up training. Feature selection methods fall into three broad families.
Filter methods evaluate each feature independently of the model, using statistical measures to rank features by relevance.
Filter methods are computationally cheap and scale well, but they ignore feature interactions.
Wrapper methods train a model on different feature subsets and evaluate each subset using cross-validated performance.
RFE and RFECV classes automate this process.Wrapper methods tend to find better subsets than filter methods but are far more expensive because they require training the model many times.
Embedded methods perform feature selection as an integral part of the model training process.
| Method family | Pros | Cons | Example techniques |
|---|---|---|---|
| Filter | Fast, scalable, model-agnostic | Ignores feature interactions | Correlation, mutual information, chi-squared |
| Wrapper | Finds strong subsets, accounts for interactions | Computationally expensive | RFE, forward selection, backward elimination |
| Embedded | Efficient, built into training | Tied to a specific model type | Lasso, tree-based importance, elastic net |
The amount and type of feature engineering needed varies by algorithm family.
Linear models (linear regression, logistic regression, SVMs with linear kernels) learn weighted sums of inputs. They benefit heavily from scaling, normalization, polynomial features, and interaction terms because they cannot learn nonlinear relationships on their own.
Tree-based models (decision trees, random forests, XGBoost, LightGBM) split the feature space using threshold rules. They are invariant to monotonic transformations, handle categorical variables natively (in some implementations), and automatically discover interaction effects through successive splits. Feature scaling is unnecessary, though creating ratio or difference features can still help by reducing the number of splits required.
Deep learning models (convolutional and recurrent neural networks, transformers) can learn hierarchical representations from raw data, reducing the need for manual feature engineering. However, they still benefit from proper input normalization, embedding layers for categorical inputs, and data augmentation. On tabular data, deep learning models rarely outperform well-engineered gradient boosting pipelines without substantial effort.
| Model family | Scaling needed | Handles categoricals natively | Benefits from interaction terms | Typical feature engineering effort |
|---|---|---|---|---|
| Linear models | Yes | No | High | Heavy |
| Tree-based models | No | Partially | Moderate | Moderate |
| Neural networks | Yes (normalization) | Via embeddings | Low (learns them) | Light to moderate |
Manual feature engineering is labor-intensive. Several open-source libraries aim to automate parts of the process.
Featuretools is the most widely adopted automated feature engineering library. It uses an algorithm called Deep Feature Synthesis (DFS), which stacks simple transformation and aggregation primitives across relational tables to generate complex features automatically. For example, given a customers table and a transactions table, DFS can compute features like "mean transaction amount per customer in the last 30 days" without manual coding.
autofeat takes a different approach, generating nonlinear transformations of individual features (logarithms, squares, square roots) and then selecting the most useful ones with L1-regularized regression. It was designed with scientific datasets in mind and works well for heterogeneous data containing measurements with different physical units.
tsfresh specializes in time-series data, automatically extracting hundreds of statistical features (autocorrelation, entropy, spectral peaks) from sequential observations.
While these tools accelerate exploration, they do not replace domain knowledge. The features they generate should be reviewed by a practitioner to ensure they are meaningful, free from leakage, and computationally feasible at inference time.
As organizations scale from individual models to production ML systems, managing engineered features becomes a significant operational challenge. A feature store is a centralized repository that stores, versions, and serves precomputed features for both model training and real-time inference.
Key capabilities of a feature store include:
Prominent feature store platforms include Feast (open-source, highly configurable), Tecton (managed platform built by the creators of Uber's Michelangelo), and Hopsworks (an all-in-one AI lakehouse with strong governance features). Cloud providers also offer integrated feature stores within their ML platforms, such as Amazon SageMaker Feature Store and Google Vertex AI Feature Store.
Feature engineering has been a central part of machine learning practice since the field's earliest days. In the 1990s and 2000s, the success of a machine learning project depended almost entirely on the skill of the practitioner in crafting features by hand. Computer vision researchers spent years designing features like SIFT (Scale-Invariant Feature Transform, introduced by David Lowe in 1999) and HOG (Histogram of Oriented Gradients, introduced by Navneet Dalal and Bill Triggs in 2005) that could robustly describe image content.
The rise of deep learning in the 2010s shifted this paradigm. Geoffrey Hinton and collaborators demonstrated in 2006 that deep neural networks could learn useful representations directly from raw data through multiple layers of nonlinear transformations, a capability now called representation learning or feature extraction. The landmark success of AlexNet in the 2012 ImageNet competition showed that convolutional neural networks could learn image features far more powerful than any hand-designed descriptor.
Despite these advances, feature engineering remains essential for tabular and structured data, which still accounts for the majority of real-world ML applications. Deep learning's ability to learn features automatically does not extend well to small, heterogeneous tabular datasets where gradient boosting with hand-engineered features continues to dominate benchmarks.
Imagine you are trying to guess which dog will win a race. You could look at every single thing about each dog: its color, its name, how many spots it has, what it ate for breakfast. But most of that information does not help you predict speed. Feature engineering is like picking out just the useful clues: the dog's leg length, its weight, and how fast it ran last time. You might even combine clues, like figuring out the ratio of leg length to body weight. The better your clues, the better your guess, even if your guessing method is simple.
Pipeline and ColumnTransformer classes chain preprocessing steps together, ensuring that transformations are applied consistently during training and inference.