Pipeline

See also: Machine learning, MLOps, Model deployment

This article is about machine learning pipelines (the end-to-end ML workflow). For splitting a model across devices by layer during distributed training, see pipeline parallelism.

A pipeline in machine learning is a sequence of data processing steps chained together into an automated, end-to-end workflow, where each step takes the previous step's output as its input. A pipeline typically chains the stages of an ML project (data ingestion, preprocessing, feature engineering, model training, evaluation, deployment, and monitoring) so that a single command can run the whole sequence reproducibly. Pipelines are fundamental to modern ML practice because they enforce reproducibility, reduce manual errors, prevent data leakage, and make complex workflows easier to manage, test, and deploy ^[1][2].

The concept of a pipeline draws from software engineering and manufacturing, where assembly lines process raw materials through a series of stations. In ML, the "raw material" is data, and the "stations" include everything from data cleaning to model training to production serving. By formalizing these steps into a pipeline, teams ensure that every run follows the same process and produces consistent, auditable results ^[2][3]. The term applies at two levels: a code-level object that chains transformers and an estimator (such as the scikit-learn Pipeline class ^[1]), and an infrastructure-level orchestrated workflow that runs each stage as a step in a managed system such as Kubeflow Pipelines, TensorFlow Extended (TFX), or Apache Airflow ^[2][4][6][7].

What are the core stages of a machine learning pipeline?

A typical ML pipeline consists of several distinct stages. While the exact steps vary depending on the project, most pipelines include the following components ^[2][3].

Data Ingestion

The pipeline begins by collecting raw data from one or more sources. These sources can include databases, APIs, data lakes, streaming platforms like Apache Kafka, or flat files such as CSV and Parquet. The ingestion step handles connection logic, schema validation, and initial quality checks to confirm that the incoming data meets expected formats and constraints.

Data Preprocessing

Preprocessing is often the most time-consuming stage. It involves cleaning and transforming raw data into a structured format suitable for analysis. Common preprocessing tasks include:

• Handling missing values through imputation or removal
• Removing duplicate records
• Encoding categorical variables (one-hot encoding, label encoding, or target encoding)
• Scaling and normalizing numerical features using techniques like min-max scaling or standardization
• Parsing dates, text fields, and other unstructured data
• Detecting and handling outliers

Preprocessing quality directly affects downstream model performance, making this stage critical to the overall pipeline.

Feature Engineering

Feature engineering involves creating new input variables or modifying existing ones to improve model predictive power. Examples include generating polynomial features, computing rolling averages, extracting text embeddings, creating interaction terms between variables, and applying domain-specific transformations. Feature engineering often requires deep understanding of the problem domain and is considered one of the most impactful steps in the pipeline.

Model Training

During training, a machine learning algorithm fits a model to the prepared data. This step involves selecting an appropriate algorithm (linear regression, random forests, neural networks, gradient boosting, and so on), configuring hyperparameters, and running the optimization process. Many pipelines integrate hyperparameter tuning methods such as grid search, random search, or Bayesian optimization at this stage.

Model Evaluation

After training, the model is evaluated on a held-out validation or test set using metrics appropriate to the task. For classification problems, common metrics include accuracy, precision, recall, F1 score, and area under the ROC curve. For regression, metrics like mean squared error, mean absolute error, and R-squared are typical. The evaluation stage may also include fairness audits, bias detection, and error analysis to ensure the model behaves as expected across different subgroups.

Model Deployment

Once a model passes evaluation, it is deployed to a production environment where it can serve predictions. Deployment can take several forms: a REST API endpoint, a batch prediction job, an embedded model within a mobile app, or integration into an existing software system. The deployment stage also involves packaging the model with its dependencies, setting up version control, and configuring scaling and load balancing.

Monitoring and Maintenance

After deployment, continuous monitoring tracks model performance, data drift, and system health. Models can degrade over time as the distribution of incoming data shifts away from the training distribution, a phenomenon known as concept drift. Monitoring systems detect these changes and can trigger automated retraining pipelines to keep the model current ^[2].

How does a training pipeline differ from an inference pipeline?

Machine learning systems typically require two separate but related pipelines.

Scroll sideways for more →

A well-designed system keeps these two pipelines decoupled but connected through a shared model registry and feature store, ensuring consistency between the features used during training and those used during inference. Google's MLOps reference architecture describes this division explicitly, noting that the goal of an automated training pipeline is to "perform continuous training of the model by automating the ML pipeline," which in turn "lets you achieve continuous delivery of model prediction service" ^[2].

What is the scikit-learn Pipeline?

The scikit-learn library provides a Pipeline class that is one of the most widely used implementations for building ML pipelines in Python. The official documentation defines it as a tool that lets you "sequentially apply a list of transformers to preprocess the data and, if desired, conclude the sequence with a final predictor for predictive modeling" ^[1]. It chains multiple processing steps (transformers) with a final estimator (classifier or regressor) into a single object that can be fitted and used for predictions.

The scikit-learn documentation lists three reasons the Pipeline is useful: "Convenience and encapsulation. You only have to call fit and predict once on your data to fit a whole sequence of estimators"; "Joint parameter selection. You can grid search over parameters of all estimators in the pipeline at once"; and "Safety. Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors" ^[1].

How It Works

A scikit-learn Pipeline is constructed from a list of (name, estimator) tuples. All steps except the last must implement a transform() method (making them transformers). The last step can be any estimator. When fit() is called on the pipeline, each transformer in sequence calls fit_transform() on the data, and the final estimator calls fit(). When predict() is called, each transformer calls transform(), and the final estimator calls predict() ^[1].

from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.svm import SVC

# Explicit construction
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('reduce_dim', PCA(n_components=10)),
    ('clf', SVC())
])

# Shorthand with auto-generated names
pipe = make_pipeline(StandardScaler(), PCA(n_components=10), SVC())

pipe.fit(X_train, y_train)
predictions = pipe.predict(X_test)

How does a pipeline prevent data leakage?

One of the most important benefits of using a scikit-learn Pipeline is its ability to prevent data leakage. Data leakage occurs when information from the test set (or future data) inadvertently influences the training process, leading to overly optimistic performance estimates that do not generalize to real-world data ^[10].

Without a pipeline, a common mistake is to fit a scaler or other transformer on the entire dataset before splitting into training and test sets. This allows statistics from the test set (such as the mean and standard deviation) to "leak" into the training process ^[10].

# BAD: data leakage - scaler learns from test data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)   # fits on ALL data including test
X_train, X_test = train_test_split(X_scaled)

# GOOD: pipeline ensures scaler only fits on training data
pipe = make_pipeline(StandardScaler(), SVC())
pipe.fit(X_train, y_train)           # scaler fits ONLY on X_train
score = pipe.score(X_test, y_test)   # scaler transforms X_test only

When used with cross-validation functions like cross_val_score, the pipeline ensures that transformers are fit exclusively on each training fold, and the validation fold is only transformed (never fitted), preserving a clean separation between training and evaluation data ^[1][10].

ColumnTransformer and FeatureUnion

Scikit-learn extends the basic Pipeline with two additional composition tools:

• ColumnTransformer applies different transformations to different columns of the input data. For example, numerical columns might be scaled while categorical columns are one-hot encoded.
• FeatureUnion runs multiple transformers in parallel and concatenates their outputs side by side, enabling the combination of different feature extraction strategies.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

preprocessor = ColumnTransformer([
    ('num', StandardScaler(), ['age', 'income']),
    ('cat', OneHotEncoder(), ['city', 'gender'])
])

pipe = Pipeline([
    ('preprocessor', preprocessor),
    ('clf', LogisticRegression())
])

What is pipeline orchestration?

At the infrastructure level, ML pipelines are often represented as Directed Acyclic Graphs (DAGs). In a DAG, each node represents a processing step and edges define the dependencies between steps. The "acyclic" constraint means there are no circular dependencies, so the pipeline has a clear execution order. DAG-based orchestration allows the system to determine which steps can run in parallel and which must wait for upstream steps to complete ^[7].

Pipeline orchestrators are tools that manage the scheduling, execution, monitoring, and error handling of these DAG-based workflows. They handle concerns like retrying failed steps, managing compute resources, passing data between steps, and sending alerts when something goes wrong ^[4][7].

Which frameworks and tools are used to build ML pipelines?

Several frameworks have emerged to support building, orchestrating, and managing ML pipelines at scale.

TensorFlow Extended (TFX)

TFX is Google's open-source, end-to-end platform for deploying production ML pipelines ^[6]. It is designed specifically for TensorFlow models and provides a set of standardized components:

Scroll sideways for more →

TFX pipelines can be orchestrated using Apache Airflow, Apache Beam, or Kubeflow Pipelines ^[6]. The platform excels in scenarios requiring enterprise-grade reliability and can handle real-time serving at scale.

Kubeflow Pipelines

Kubeflow Pipelines is a Kubernetes-native platform for building and deploying portable ML workflows ^[4]. Each pipeline step runs in its own container, providing strong isolation and reproducibility. The system uses Argo Workflows under the hood to orchestrate Kubernetes Pods that carry out each step.

Key features include:

• A Python SDK for defining pipelines as code
• A web UI for visualizing pipeline runs, comparing experiments, and inspecting outputs
• Built-in support for distributed training, hyperparameter tuning (via Katib), and model serving (via KServe)
• Integration with feature stores (Feast) and model registries

Kubeflow is well suited for organizations that already run Kubernetes and need fine-grained control over resource allocation and scaling ^[4].

MLflow

MLflow is a framework-agnostic, open-source platform that manages the entire ML lifecycle. Unlike TFX and Kubeflow, which focus heavily on pipeline orchestration, MLflow provides a broader set of capabilities ^[5]:

• Tracking: Log parameters, metrics, and artifacts for every experiment run
• Projects: Package ML code in a reproducible format
• Models: Serve models across multiple deployment targets
• Model Registry: Version, stage, and manage models through their lifecycle

MLflow supports Python, R, and Java, and integrates with virtually any ML framework. Many teams combine MLflow with a dedicated orchestrator (like Airflow or Kubeflow) to get the best of both worlds: MLflow handles experiment tracking and model management, while the orchestrator handles scheduling and execution ^[5].

Apache Airflow

Apache Airflow is a general-purpose workflow orchestration platform widely adopted for ML pipelines ^[7]. Pipelines are defined as Python DAGs, and Airflow provides a scheduler, a web UI for monitoring, and a plugin system for integrating with external services.

A typical ML pipeline in Airflow might define tasks for data extraction, preprocessing, model training, evaluation, and deployment, with dependencies ensuring they run in the correct order. Airflow's distributed architecture consists of a scheduler that orchestrates workflow execution, executors responsible for running task instances on workers, and a metadata database that stores state and history ^[7].

Airflow is tool-agnostic and can orchestrate actions across any service with an API, making it a flexible backbone for MLOps workflows. However, Airflow was originally designed for data engineering workflows, so it lacks some ML-specific features (like built-in experiment tracking) that purpose-built ML platforms provide.

Comparison of Pipeline Frameworks

Scroll sideways for more →

What is a feature store?

A feature store is a centralized repository that manages feature engineering outputs for ML models ^[8]. It serves as a single source of truth for feature definitions, ensuring that the same features are used consistently during both training and inference. Without a feature store, teams often end up with duplicated feature computation code and inconsistencies between training and serving environments, a problem known as training-serving skew ^[8].

A feature store typically has two storage layers:

• Offline store: Holds historical feature data for batch training jobs, usually backed by a data warehouse or data lake
• Online store: Serves precomputed features at low latency for real-time inference, often backed by a key-value store like Redis or DynamoDB

Popular feature store implementations include Feast (open source), Tecton, Hopsworks, and built-in feature stores from cloud providers such as Amazon SageMaker Feature Store, Google Vertex AI Feature Store, and Databricks Feature Store ^[8].

What is a model registry?

A model registry is a centralized system for storing, versioning, and managing trained ML models throughout their lifecycle. It acts as the bridge between training pipelines and deployment pipelines, providing a structured way to track which models are in production, which are in staging, and which are experimental ^[5].

Key capabilities of a model registry include:

• Version control: Automatically tracks each model version along with its training parameters, metrics, and data lineage
• Stage management: Models can be assigned stages (e.g., "staging," "production," "archived") to manage promotion workflows
• Traceability: Each registered model version links back to the specific training run, dataset, and code that produced it, enabling full reproducibility
• Governance: Role-based access controls and approval workflows ensure that only validated models reach production

MLflow Model Registry, Amazon SageMaker Model Registry, Google Vertex AI Model Registry, and Azure ML Model Registry are widely used implementations ^[5].

How does CI/CD work for machine learning?

Continuous Integration and Continuous Deployment (CI/CD) practices, originally developed for traditional software, have been adapted for ML pipelines. However, ML CI/CD differs in important ways from conventional software CI/CD because in ML, the final product depends on both code and data. A change in either one can alter model behavior ^[2][9].

Google has defined three levels of MLOps maturity that describe how organizations adopt pipeline automation ^[2]:

Scroll sideways for more →

According to Google Cloud, "MLOps is an ML engineering culture and practice that aims at unifying ML system development (Dev) and ML system operation (Ops)," and the purpose of advancing through these levels is to "automate and monitor all steps of ML system construction, including integration, testing, releasing, deployment and infrastructure management" ^[2].

Key CI/CD Components for ML

• Data validation: Automated checks for schema changes, missing values, and data drift
• Model validation: Performance benchmarking against baseline models before promotion
• Deployment strategies: Blue-green deployment (two identical environments with traffic switching), canary releases (routing a small percentage of traffic to the new model), and shadow deployment (running the new model alongside the old one without affecting users)
• Automated retraining: Pipelines that trigger when performance drops below a threshold or new data arrives

What should you monitor in a deployed pipeline?

Monitoring a deployed ML pipeline involves tracking several dimensions beyond traditional software monitoring ^[2]:

• Data quality: Monitoring input data distributions for drift, missing values, and schema violations
• Model performance: Tracking prediction accuracy, latency, and throughput over time
• Feature drift: Detecting when the statistical properties of input features change relative to the training distribution
• Concept drift: Identifying when the relationship between input features and the target variable shifts
• System health: Monitoring CPU, memory, GPU utilization, and pipeline execution times

Tools such as Evidently AI, WhyLabs, Arize AI, and Amazon SageMaker Model Monitor provide specialized capabilities for ML monitoring.

What are the best practices for ML pipelines?

1. Modularize pipeline steps. Each step should have a single responsibility, clearly defined inputs and outputs, and be independently testable.
2. Version everything. Track versions of data, code, models, and pipeline configurations. Tools like DVC (Data Version Control) complement Git for data and model versioning.
3. Automate testing. Include data validation tests, unit tests for transformation logic, integration tests for the full pipeline, and model performance tests.
4. Use containers. Package each pipeline step in a Docker container to ensure consistent execution across development, staging, and production environments.
5. Implement idempotency. Pipeline steps should produce the same output when run multiple times with the same input, making reruns and debugging safer.
6. Log metadata extensively. Record parameters, metrics, data lineage, and execution times for every pipeline run to support debugging and compliance.
7. Separate concerns. Keep training pipelines, inference pipelines, and monitoring systems as independent components connected through well-defined interfaces.
8. Plan for failure. Design pipelines with retry logic, graceful error handling, and alerting so that transient failures do not require manual intervention.

Explain Like I'm 5 (ELI5)

Imagine you are making cookies. A machine learning pipeline is like the recipe and all the steps you follow to make the cookies. First, you gather and prepare the ingredients, which is like collecting and cleaning data. Then you mix them together in just the right way, which is like feature engineering. Next, you put the cookies in the oven and bake them, which is like training the model. You taste one cookie to check if they are good, which is like evaluation. Finally, you put all the cookies on a plate for your friends, which is like deployment. If you write down every step and follow it the same way each time, your cookies will always come out right. That is what a pipeline does for machine learning: it writes down all the steps so the computer follows them in the same order every time.

References

1. Scikit-learn documentation. "Pipelines and composite estimators." https://scikit-learn.org/stable/modules/compose.html ↩
2. Google Cloud Architecture Center. "MLOps: Continuous delivery and automation pipelines in machine learning." https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning ↩
3. Valohai. "What is a Machine Learning Pipeline?" https://valohai.com/machine-learning-pipeline/ ↩
4. Kubeflow documentation. "Pipelines Overview." https://www.kubeflow.org/docs/components/pipelines/overview/ ↩
5. MLflow documentation. "MLflow Model Registry." https://mlflow.org/docs/latest/ml/model-registry/ ↩
6. TensorFlow. "TFX: TensorFlow Extended." https://www.tensorflow.org/tfx ↩
7. Apache Airflow documentation. https://airflow.apache.org/ ↩
8. Snowflake. "Feature Store for Machine Learning." https://www.snowflake.com/en/fundamentals/feature-store/ ↩
9. DataCamp. "A Beginner's Guide to CI/CD for Machine Learning." https://www.datacamp.com/tutorial/ci-cd-for-machine-learning ↩
10. Scikit-learn documentation. "Common pitfalls and recommended practices." https://scikit-learn.org/stable/common_pitfalls.html ↩