Feature Extraction

See also: Machine learning terms, Feature engineering, Dimension reduction

Feature extraction is the process of transforming raw data into a smaller set of derived, informative numerical variables called features that capture the essential structure of the input while discarding noise and redundancy. In machine learning and pattern recognition, these extracted features serve as inputs to classifiers, regressors, and clustering algorithms, reducing computational cost, mitigating the curse of dimensionality, and often improving model accuracy. Feature extraction differs from feature selection in one key respect: extraction creates new features by combining or transforming the originals (for example, with Principal Component Analysis or a convolutional neural network), whereas selection keeps a subset of the original variables unchanged.^[10] The importance of how data is represented is foundational to the field: as Bengio, Courville, and Vincent put it, "the success of machine learning algorithms generally depends on data representation."^[9]

Introduction

Feature extraction is a process in machine learning and pattern recognition that transforms raw data into a set of meaningful, informative representations called features. Rather than feeding unprocessed data directly into a model, feature extraction produces derived variables that capture the essential structure of the input while discarding noise and redundancy. The resulting features serve as inputs for classifiers, regressors, and other learning algorithms, enabling them to operate more effectively on complex, high-dimensional data.

The practice sits at the core of the broader feature engineering pipeline and is closely related to dimensionality reduction. By converting raw signals (pixels, audio waveforms, text strings) into compact numerical representations, feature extraction reduces computational cost, mitigates the curse of dimensionality, and often improves model accuracy.

How does feature extraction differ from feature selection?

Feature extraction and feature selection are both dimensionality reduction strategies, but they work in fundamentally different ways.^[10]

Scroll sideways for more →

In practice, practitioners often combine both approaches. For instance, a computer vision pipeline might use a convolutional neural network for feature extraction and then apply feature selection to the resulting embedding before training a final classifier.

What is handcrafted (manual) feature extraction?

Before the rise of deep learning, domain experts designed feature descriptors by hand to capture relevant patterns in specific data types. Handcrafted features remain useful in settings where data is limited, latency matters, or interpretability is required.

Computer Vision Descriptors

Scroll sideways for more →

SIFT and HOG both rely on gradient orientation histograms, but they target different problems. SIFT identifies sparse keypoints and produces a descriptor for each one, making it effective in cluttered scenes; David Lowe described SIFT features as "invariant to image scaling, translation, and rotation, and partially invariant to illumination changes and affine or 3D projection."^[1] HOG computes dense descriptors over a detection window, making it better suited for detecting objects with consistent shape, such as human silhouettes.^[2] The expanded SIFT paper, published in the International Journal of Computer Vision in 2004, has been cited more than 70,000 times, making it one of the most influential works in computer vision.^[1]

Audio and Speech Descriptors

Scroll sideways for more →

The MFCC computation pipeline follows several steps: the audio signal is pre-emphasized, divided into overlapping frames, windowed (usually with a Hamming window), converted to the frequency domain via FFT, passed through a mel-spaced filter bank, log-compressed, and finally transformed with a discrete cosine transform (DCT) to produce the cepstral coefficients.^[8] Typically 12 to 20 MFCCs are retained per frame, along with delta and delta-delta coefficients that capture temporal dynamics.

What is learned feature extraction?

Modern deep learning models learn feature representations directly from raw data during training, eliminating the need for manual descriptor design. This approach, sometimes called representation learning, has become the dominant paradigm in computer vision, natural language processing, and speech processing.^[9]

Convolutional Neural Networks (CNNs)

A convolutional neural network learns a hierarchy of features through stacked convolutional layers. Early layers detect low-level patterns such as edges and textures. Middle layers combine these into parts and motifs. Deeper layers assemble high-level semantic concepts such as faces, objects, or scenes. By the time an input image reaches the final convolutional layer, it has been transformed from a high-resolution grid of pixels into a compact feature map that encodes the most discriminative information.

Architectures like VGG, ResNet, and EfficientNet have demonstrated that deeper networks can learn increasingly abstract and powerful features.^[9] The intermediate representations (feature maps) produced by these networks are widely reused across tasks through transfer learning.

Autoencoders

An autoencoder is a neural network trained to compress its input into a low-dimensional bottleneck representation and then reconstruct the original input from that compressed code.^[7] The encoder half of the network acts as a feature extractor: it learns a mapping from high-dimensional input to a compact latent space that retains the most important information.

Once trained, the decoder is discarded, and the encoder is used as a standalone feature extractor. The bottleneck vectors can then serve as input features for downstream classifiers or clustering algorithms. Variants include:

• Denoising autoencoders, which learn robust features by reconstructing clean data from corrupted inputs.
• Variational autoencoders (VAEs), which impose a probabilistic structure on the latent space and are useful for generative modeling.
• Convolutional autoencoders, which use convolutional layers instead of fully connected layers, making them especially effective for image data.

Transformer-Based Feature Extraction

Transformer models have become the dominant architecture for learned feature extraction across multiple modalities. In NLP, models like BERT and GPT produce contextual embeddings that capture rich semantic information. In vision, Vision Transformers (ViT) split images into patches and process them with self-attention to produce powerful visual features. Multimodal transformers like CLIP learn joint representations of images and text.

How is feature extraction used in natural language processing?

Text data requires conversion from strings of characters into numerical representations before any machine learning model can process it. NLP feature extraction techniques span a wide range of complexity.

Scroll sideways for more →

TF-IDF (Term Frequency-Inverse Document Frequency) remains a practical baseline. It assigns each word a score that increases with its frequency in a document but decreases with its frequency across the entire corpus, effectively highlighting words that are distinctive to a particular document. Despite its simplicity, TF-IDF can match or even outperform transformer-based methods on certain classification tasks while running orders of magnitude faster.

Word embeddings such as Word2Vec and GloVe map each word to a dense vector in a continuous space where semantic relationships are encoded as geometric relationships.^[4] The classic example is that the vector arithmetic "king" minus "man" plus "woman" yields a vector close to "queen," a result Mikolov and colleagues demonstrated to show that word vectors capture syntactic and semantic regularities through simple vector offsets.^[4][11] However, these embeddings are static: the word "bank" receives the same vector whether it refers to a financial institution or a river bank.

BERT and contextual embeddings solve the polysemy problem by producing a different vector for each occurrence of a word, conditioned on the surrounding text. BERT uses a bidirectional transformer architecture that considers both the left and right context simultaneously, yielding embeddings that capture nuanced semantic meaning.^[5] On release in 2019, BERT pushed the GLUE benchmark score to 80.5, a 7.7 point absolute improvement over the prior state of the art, and set new records on eleven natural language processing tasks.^[5] These contextual vectors are widely used as features for downstream tasks including sentiment analysis, named entity recognition, and question answering.

How is feature extraction used in computer vision?

In computer vision, feature extraction converts raw pixel arrays into representations that encode visual content at various levels of abstraction.

Classical Pipeline

The traditional computer vision pipeline follows a two-stage approach: (1) extract handcrafted features such as SIFT, HOG, or local binary patterns, and (2) feed those features into a separate classifier such as a support vector machine (SVM) or random forest. This approach dominated until 2012, when AlexNet won the ImageNet Large Scale Visual Recognition Challenge with a top-5 error rate of 15.3 percent, far ahead of the runner-up's 26.2 percent and marking the moment learned features overtook handcrafted pipelines on standard benchmarks.^[12]

Deep Learning Pipeline

Modern pipelines use end-to-end deep learning. A convolutional neural network simultaneously learns the feature extraction and classification stages. The convolutional layers act as learned feature extractors, and the final fully connected layers perform classification. Architectures like ResNet, Inception, and EfficientNet achieve state-of-the-art results on image classification, object detection, and segmentation tasks.

CNN feature maps at different layers capture different information:

Scroll sideways for more →

How does transfer learning use feature extraction?

Transfer learning is one of the most impactful applications of feature extraction in modern deep learning. Instead of training a model from scratch on a new task, practitioners take a network pretrained on a large dataset (such as ImageNet for vision or a large text corpus for NLP) and repurpose its learned representations.

There are two primary strategies:

1. Feature extraction (frozen backbone). The pretrained model's weights are frozen, and its output (or an intermediate layer's output) is used as a fixed feature vector for the new task. A new classifier head is trained on top. This approach is fast and works well when the new dataset is small or similar to the pretraining data.^[6]

2. Fine-tuning. Some or all layers of the pretrained model are unfrozen and trained with a low learning rate on the new task's data. This allows the features to adapt to the specific characteristics of the new domain while still benefiting from the pretrained initialization.^[6]

Transfer learning dramatically reduces the data and compute requirements for new tasks. A model pretrained on ImageNet's 1.2 million images can be adapted to a specialized medical imaging task with only a few hundred labeled examples, achieving performance that would be impossible if training from scratch.

How does PCA perform feature extraction?

Principal Component Analysis (PCA) is one of the most widely used linear feature extraction techniques. It transforms a dataset into a new coordinate system defined by the directions of maximum variance in the data, called principal components.^[3]

How PCA Works

1. Standardize the data so that each feature has zero mean and unit variance.
2. Compute the covariance matrix of the standardized features.
3. Calculate eigenvectors and eigenvalues of the covariance matrix. Each eigenvector defines a principal component direction, and its corresponding eigenvalue indicates how much variance that component explains.
4. Rank the components by eigenvalue in descending order.
5. Select the top k components that collectively explain a desired fraction of the total variance (commonly 95% or 99%).
6. Project the original data onto the selected components to obtain the reduced representation.

The resulting principal components are uncorrelated by construction, which removes redundancy among features.^[3] PCA is computationally efficient for moderate-dimensional data and serves as a strong baseline before applying more complex methods.

Limitations of PCA

PCA assumes linear relationships among features. When the underlying structure is nonlinear, kernel PCA or nonlinear methods such as t-SNE and UMAP may be more appropriate. Additionally, because principal components are linear combinations of all original features, interpreting what each component represents can be difficult.

How is feature extraction used in audio and speech?

Audio signals are continuous waveforms that must be converted into structured numerical representations for machine learning. Feature extraction bridges the gap between raw audio and models for speech recognition, music analysis, sound classification, and other audio tasks.

The standard processing pipeline begins by segmenting the audio into short overlapping frames (typically 20 to 40 milliseconds). Each frame is then transformed into one or more feature representations:

• Spectrograms show the distribution of frequency energy over time. A mel spectrogram applies the mel scale to better match human frequency perception and is widely used as input to deep learning models for audio.
• MFCCs provide a compact summary of the spectral envelope for each frame and are the most established feature type in speech recognition.^[8]
• Chroma and tonnetz features capture harmonic and tonal content, making them useful in music information retrieval.
• Zero-crossing rate, spectral centroid, and bandwidth are simple time-domain and frequency-domain statistics that can complement more complex features.

In modern systems, raw spectrograms or mel spectrograms are often fed directly into convolutional neural networks or transformer models, which learn task-specific features automatically. Models such as Wav2Vec 2.0 and Whisper learn powerful audio representations from large-scale self-supervised pretraining and can be fine-tuned for downstream tasks like transcription or speaker verification. Wav2Vec 2.0 demonstrated the strength of this approach: pretraining on 53,000 hours of unlabeled speech and then fine-tuning on just ten minutes of labeled audio achieved a word error rate of 4.8 on the LibriSpeech clean test set.^[13]

How is feature extraction related to dimensionality reduction?

Feature extraction is closely related to dimensionality reduction, and the two terms are sometimes used interchangeably. Both aim to produce a lower-dimensional representation of the data. The distinction, where one exists, is primarily one of emphasis: dimensionality reduction focuses on reducing the number of variables, while feature extraction emphasizes creating informative representations that improve downstream task performance.

Common dimensionality reduction techniques that double as feature extraction methods include:

Scroll sideways for more →

Explain Like I'm 5 (ELI5)

Imagine you have a huge box of LEGO bricks in all different shapes, sizes, and colors. You want to build a specific spaceship, but you do not need every single brick. Feature extraction is like sorting through the box and pulling out only the pieces that matter for your spaceship: the wing shapes, the cockpit pieces, and the right colors. You might even snap a few small bricks together to make one special piece that is easier to work with. In the end, you have a smaller, more useful pile of parts that helps you build your spaceship faster and better. Machine learning models do the same thing with data: they take a huge pile of numbers and turn it into a smaller, smarter set of numbers that makes learning easier.

References

1. Lowe, D. G. (2004). "Distinctive Image Features from Scale-Invariant Keypoints." International Journal of Computer Vision, 60(2), 91-110. doi:10.1023/B:VISI.0000029664.99615.94 ↩
2. Dalal, N., & Triggs, B. (2005). "Histograms of Oriented Gradients for Human Detection." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). ↩
3. Jolliffe, I. T. (2002). *Principal Component Analysis*, 2nd ed. Springer Series in Statistics. ↩
4. Mikolov, T., et al. (2013). "Efficient Estimation of Word Representations in Vector Space." Proceedings of the International Conference on Learning Representations (ICLR). arXiv:1301.3781 ↩
5. Devlin, J., et al. (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of NAACL-HLT. arXiv:1810.04805 ↩
6. Yosinski, J., et al. (2014). "How transferable are features in deep neural networks?" Advances in Neural Information Processing Systems (NeurIPS). arXiv:1411.1792 ↩
7. Bank, D., Koenigstein, N., & Giryes, R. (2023). "Autoencoders and their applications in machine learning: a survey." Artificial Intelligence Review, 56, 8191-8232. ↩
8. Davis, S. B., & Mermelstein, P. (1980). "Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences." IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4), 357-366. ↩
9. Bengio, Y., Courville, A., & Vincent, P. (2013). "Representation Learning: A Review and New Perspectives." IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798-1828. arXiv:1206.5538 ↩
10. Guyon, I., & Elisseeff, A. (2003). "An Introduction to Variable and Feature Selection." Journal of Machine Learning Research, 3, 1157-1182. ↩
11. Mikolov, T., Yih, W., & Zweig, G. (2013). "Linguistic Regularities in Continuous Space Word Representations." Proceedings of NAACL-HLT, 746-751. ↩
12. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet Classification with Deep Convolutional Neural Networks." Advances in Neural Information Processing Systems (NeurIPS), 25. ↩
13. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." Advances in Neural Information Processing Systems (NeurIPS), 33. arXiv:2006.11477 ↩