# TensorFlow Extended

> Source: https://aiwiki.ai/wiki/tfx
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Developer Tools, Google, MLOps, Machine Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "TensorFlow Extended." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/tfx
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**TensorFlow Extended** (**TFX**) is an open-source platform from [Google](https://aiwiki.ai/wiki/google) for building and running production [machine learning](https://aiwiki.ai/wiki/machine_learning) pipelines on top of [TensorFlow](https://aiwiki.ai/wiki/tensorflow). A TFX pipeline chains standard components for data ingestion, data validation, [feature engineering](https://aiwiki.ai/wiki/feature_engineering), training, model evaluation and deployment, and records the artifacts and executions of each run in a metadata store, so that a model can be traced back to the data and pipeline run that produced it.[3][4][5] Google described the platform publicly at the KDD 2017 conference, launched it inside the company the same year, and released the public version in early 2019.[1][2][7] The code is published under the Apache 2.0 license on GitHub.[6]

Within Google, TFX succeeded an older platform called Sibyl and by 2020 had become, in the words of its developers, "the most widely used, general purpose E2E ML platform at Alphabet".[2] Outside Google it has been used directly and through Google Cloud services such as Cloud AI Platform Pipelines and Vertex AI Pipelines.[13][14] As of November 2026 the latest release is TFX 1.21.0 (June 2026), and Google has said that work on TFX and related TensorFlow libraries now concentrates on security fixes, critical bug fixes and dependency updates.[9][11]

## History

### Sibyl, the predecessor

Google's earlier end-to-end platform, Sibyl, was founded in 2007. It trained "wide" models (linear, logistic and Poisson regression, later factorization machines) and bundled tools for data ingestion, data analysis and validation, training, model analysis and training-serving skew detection into one integrated product.[2] According to a 2020 history written by members of the TFX team, Sibyl stayed in production for about 14 years before it was decommissioned, and most of its workloads moved to TFX.[2] Sibyl ran only on CPUs in Google's own data centers, on the Borg cluster manager, and its pipeline topology was largely fixed.[2]

The same account says that TensorFlow, released publicly in 2015, offered modeling flexibility but no complete production system, while Sibyl had end-to-end tooling but limited flexibility. Google built TFX to combine the two and launched it internally in 2017.[2] TFX reused many of Sibyl's system designs, and where Sibyl had relied on MapReduce and Flume for distributed data processing, TFX adopted their portable successor, Apache Beam.[2]

### The KDD 2017 paper

The platform was first described in "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform", a paper by Denis Baylor and 21 co-authors at Google, presented in the Applied Data Science track of KDD 2017 in Halifax, Nova Scotia.[1] The paper argued that teams often wired together learners, data checks, model checks and serving systems with ad hoc glue code, which duplicated effort and produced fragile systems. It reported that integrating these pieces into one platform with a shared configuration reduced the time to production "from the order of months to weeks".[1]

The paper's case study was the ranking model behind app recommendations on the Google Play store, which at the time had more than one billion active users and more than one million apps.[1] The ranking model was retrained continuously on training sets of hundreds of billions of examples and served through [TensorFlow Serving](https://aiwiki.ai/wiki/tensorflow_serving) at thousands of queries per second with a latency budget of tens of milliseconds.[1] By comparing statistics of serving logs with training data from the same day, the data validation component found features that were always present in training but always missing at serving time; an online A/B experiment showed that removing this skew raised the app install rate on the store's main landing page by 2%.[1] Play also used warm-starting, initializing a subset of parameters such as [embeddings](https://aiwiki.ai/wiki/embeddings) from a previous model, so that it could push fresh models frequently without training from scratch.[1]

A follow-up paper at the 2019 USENIX Conference on Operational Machine Learning (OpML '19) described how TFX runs continuous pipelines in production at Google and the lessons learned from that deployment.[12]

### Open-source release and TFX 1.0

The GitHub repository was created in February 2019, and the first stable package on the Python Package Index, version 0.12.0, was published on March 6, 2019, after release candidates that began on March 2.[6][7] The 2020 history paper says the public offering "was released in early 2019" and was adopted within a year in on-premises environments and on Google Cloud.[2]

TFX 1.0.0, released on July 22, 2021, was the first post-beta release.[3][8] It introduced the `tfx.v1` public API namespace with stability guarantees for pipelines across upgrades, and added support in the TFX command-line tool for Vertex Pipelines through an `--engine=vertex` flag.[3][8]

### Later releases and 2026 status

Releases slowed after 2024. Version 1.16.0 appeared in December 2024; it dropped support for the TensorFlow Estimator API and deprecated the runner for version 1 of the Kubeflow Pipelines SDK in favour of the version 2 runner.[17] Version 1.17.0 followed in March 2026, targeting TensorFlow 2.17.[10] After patch releases up to 1.17.3 in April 2026, the next release was 1.21.0 on June 18, 2026.[10] Its notes list support for TensorFlow 2.21.0 and Protobuf 6.x on Python 3.10 through 3.13, drop four outdated or incompatible dependencies (TensorFlow Decision Forests, TensorFlow Ranking, TensorFlow Text and TensorFlow.js) to avoid installation conflicts, skip legacy tests for the retired Google Cloud AI Platform integration, and add a pure-Python fallback for ML Metadata environments that lack the ZetaSQL engine.[9]

In a March 6, 2026 post on TensorFlow 2.21, Google said the TensorFlow team would focus on security and critical bug fixes, minor releases for dependency updates such as new Python versions, and accepting community bug fixes. The post listed TFX, TensorFlow Serving, TensorFlow Data Validation, TensorFlow Transform and TensorFlow Model Analysis among the libraries those commitments cover, and pointed developers starting new generative AI work toward [Keras](https://aiwiki.ai/wiki/keras) 3, [JAX](https://aiwiki.ai/wiki/jax) and [PyTorch](https://aiwiki.ai/wiki/pytorch).[11]

| Date | Event |
|---|---|
| 2007 | Sibyl, TFX's predecessor, founded at Google[2] |
| 2017 | TFX launched inside Google[2] |
| August 2017 | TFX paper presented at KDD 2017[1] |
| February 2019 | tensorflow/tfx repository created on GitHub[6] |
| March 2019 | First stable PyPI release, version 0.12.0[7] |
| March 2020 | Cloud AI Platform Pipelines announced with a TFX SDK in preview[13] |
| July 2021 | TFX 1.0.0, first post-beta release[8] |
| December 2024 | TFX 1.16.0[10] |
| March 2026 | TFX 1.17.0; TensorFlow 2.21 post narrows ecosystem work to fixes and dependency updates[10][11] |
| June 2026 | TFX 1.21.0, supporting TensorFlow 2.21[9] |

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/Mxk4qmO_1B4" title="What exactly is this TFX thing? (TensorFlow Extended)"></iframe>
<figcaption>"What exactly is this TFX thing? (TensorFlow Extended)", official video from TensorFlow, published July 11, 2019. <a href="https://www.youtube.com/watch?v=Mxk4qmO_1B4">Watch on YouTube</a>.</figcaption>
</figure>

## Architecture

### Pipelines, components and artifacts

A TFX pipeline is a portable description of a machine learning workflow made of component instances and input parameters.[4] Each component has three parts: a component specification that declares its input artifacts, output artifacts and parameters; an executor that does the actual work; and a component interface that packages the two for use in a pipeline.[4] The outputs of components are called artifacts, and every artifact must have a type registered in the ML Metadata store.[4] TFX works out the execution order by building a directed acyclic graph from the artifact dependencies, so components whose inputs do not depend on each other, such as ExampleValidator and Transform, can run in parallel.[4] Users can write custom components alongside the standard ones.[4]

### Standard components

The TFX user guide lists the following standard components.[3]

| Component | Role |
|---|---|
| ExampleGen | Ingests the input data, optionally splits it, and formats it as `tf.Example` records |
| StatisticsGen | Computes statistics over the dataset |
| SchemaGen | Infers a data schema from the statistics |
| ExampleValidator | Looks for anomalies, missing values and wrong types using the statistics and schema |
| Transform | Performs feature engineering and exports the transformation graph |
| Tuner | Tunes model hyperparameters, using KerasTuner |
| Trainer | Trains the model |
| Evaluator | Computes metrics, compares a candidate model with a baseline, and can block deployment when thresholds are not met |
| InfraValidator | Makes a canary deployment of the model in a sandbox to check that it can actually be served |
| Pusher | Deploys the model to serving infrastructure |
| BulkInferrer | Runs batch inference on unlabeled data |

The guide stresses the role of Transform in avoiding [training-serving skew](https://aiwiki.ai/wiki/training-serving_skew): its output is a [SavedModel](https://aiwiki.ai/wiki/savedmodel) containing the preprocessing graph, which the Trainer imports so that the same transformations run during training and at inference time.[3] The KDD paper made the same point, noting that integrating data analysis, training and serving lets the platform guarantee that identical transformations are applied at training and serving, which "prevents one form of training-serving skew".[1]

### Libraries

Many standard components are built on libraries that provide their base functionality and can also be used on their own.[3] The 2020 history paper describes TFX as layered: managed cloud services at the top, TFX pipelines below them, the TensorFlow Serving binary, and then a set of libraries.[2]

| Library | Purpose |
|---|---|
| TensorFlow Data Validation (TFDV) | Computes summary statistics, infers schemas and detects anomalies such as missing features, out-of-range values or wrong types; includes viewers for statistics and anomalies[3] |
| TensorFlow Transform (TFT) | Preprocessing that needs a full pass over the data, such as normalizing by mean and standard deviation, building vocabularies or bucketizing values[3] |
| TensorFlow Model Analysis (TFMA) | Distributed model evaluation, with metrics computed over slices of the data[3] |
| TensorFlow Metadata (TFMD) | Standard formats for schemas and dataset statistics[3] |
| ML Metadata (MLMD) | Records and retrieves metadata and lineage for pipeline artifacts and executions[3] |
| TFX Basic Shared Libraries (TFX-BSL) | Shared library layer listed in the TFX architecture[2] |
| TensorFlow Serving | Serves trained models over REST and gRPC[3] |

Evaluation slices let users check whether a model that performs well on average does badly for a smaller group, for example a particular customer segment.[3] The history paper adds that the Evaluator supports custom metrics, confidence intervals and Fairness Indicators through TFMA.[2]

### ML Metadata and lineage

ML Metadata is a library for recording and querying metadata about machine learning workflows. It is part of TFX but designed to be usable independently.[5] Its data model stores artifacts and their types, executions of components, events linking executions to the artifacts they consumed and produced, and contexts that group artifacts and executions, such as a pipeline run or an experiment.[5] With these records a user can answer questions such as which dataset a model was trained on, which hyperparameters were used, which pipeline run created a model, or when a failed model was pushed.[5] MLMD can also tell whether a component already ran on the same inputs, so that earlier outputs can be reused.[5] The storage backend is pluggable: MLMD ships reference implementations for SQLite and MySQL, and its documentation also shows how to connect to PostgreSQL.[5]

The history paper compares artifact lineage to a stack trace in ordinary software debugging, and says MLMD, together with the underlying storage layer, forms the foundation of TFX's compliance mechanisms.[2]

### Data processing and orchestration

TFX uses Apache Beam for data-parallel processing, and the default installation includes Beam's DirectRunner for local execution; distributed runs can use back ends such as Apache Flink, Apache Spark and Google Cloud Dataflow.[3] For its compute-heavy libraries TFX adopted Apache Arrow as an in-memory columnar format.[2]

Pipelines are portable across orchestrators. The documentation names Apache Airflow, Apache Beam and [Kubeflow](https://aiwiki.ai/wiki/kubeflow) Pipelines, and uses the term DagRunner for the code that connects a pipeline to a given orchestrator.[3][4] A command-line tool can create, update, run, list and delete pipelines on these orchestrators.[3]

### Deployment targets

Trained models are exported as SavedModels and can be deployed to three classes of target: TensorFlow Serving for server-side inference, [TensorFlow Lite](https://aiwiki.ai/wiki/tensorflow_lite) for mobile and IoT devices, and [TensorFlow.js](https://aiwiki.ai/wiki/tensorflow_js) for browsers and Node.js.[3] In a typical pipeline, InfraValidator first launches a canary TensorFlow Serving instance with the new model, and Pusher deploys it only if that check passes.[3]

## Continuous training

TFX was designed for pipelines that keep running as new data arrives, not only for one-off training jobs.[1][2] The KDD paper describes continuation strategies built from two settings: data visitation, which can be static or can follow a rolling range of directories, and warm-starting, which initializes some model parameters from an earlier state.[1] The history paper says TFX was built from the ground up for continuous learning: pipelines detect new data and schedule updated models, recover from intermittent failures, and support "backfilling", where a pipeline is rerun on existing data with updated components or on corrected data. It noted in 2020 that the public release did not yet offer continuous pipelines.[2]

## Google Cloud integration

In March 2020 Google Cloud announced Cloud AI Platform Pipelines, a beta service that offered two authoring options: the Kubeflow Pipelines SDK and the TFX SDK, which was then in preview. Google described the TFX SDK as the higher-level option, "with prescriptive, but customizable components", recommended for end-to-end pipelines built on TensorFlow.[13] Vertex AI Pipelines, the serverless pipeline service of Google's later Vertex AI platform, accepts pipelines written with TFX (version 0.30.0 or later) or the Kubeflow Pipelines SDK, and its documentation recommends TFX for TensorFlow workflows that process terabytes of structured or text data and the Kubeflow Pipelines SDK for other cases.[14] On April 22, 2026 Google made [Vertex AI](https://aiwiki.ai/wiki/google_vertex_ai) part of Gemini Enterprise Agent Platform and renamed the service Agent Platform Pipelines; the same TFX support is documented under the new name.[14][16]

## Adoption

The 2020 history paper reported that inside Alphabet TFX was used by thousands of users and powered hundreds of products, including Cloud AI services on Google Cloud. It said that on a typical day thousands of TFX pipelines were running, processing exabytes of data and producing tens of thousands of models, which served hundreds of millions of inferences per second.[2]

Spotify described its move to TFX and Kubeflow Pipelines in a December 2019 engineering post. TensorFlow Data Validation was the first TFX component it used, to find skew, bad values and excess nulls, and its platform wrapped TFX components inside centrally managed Kubeflow Pipelines clusters. Spotify launched an alpha version of the platform in August 2019 and reported about 100 users and 18,000 runs by December; early analysis indicated that some teams were producing "7x more experiments".[15]

## Trade-offs

The TFX team's own 2020 account lists the costs of several design choices.[2] Taking dependencies on external systems such as Kubeflow Pipelines or Airflow required effort that grew faster than linearly with the number of dependencies, and those costs sometimes reached users as version conflicts or incompatibilities between environments. Supporting many combinations of orchestrators, runners and clouds produced more deployment configurations than the team could test. Offering several public layers (libraries, pipelines and services) confused some users about which layer to use, and the platform's flexibility made it harder to build, maintain and support.[2] Dependency management remained a theme in 2026, when the 1.21.0 release dropped several TensorFlow libraries from TFX's dependency list to stop pip from backtracking through version conflicts during installation.[9]

## See also

- [MLOps](https://aiwiki.ai/wiki/mlops)
- [TensorFlow](https://aiwiki.ai/wiki/tensorflow)
- [TensorFlow Serving](https://aiwiki.ai/wiki/tensorflow_serving)
- [Kubeflow](https://aiwiki.ai/wiki/kubeflow)
- [MLflow](https://aiwiki.ai/wiki/mlflow)
- [Feature spec](https://aiwiki.ai/wiki/feature_spec)
- [Feature store](https://aiwiki.ai/wiki/feature_store)
- [Concept drift](https://aiwiki.ai/wiki/concept_drift)

## References

1. Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., Koo, C. Y., Lew, L., Mewald, C., Modi, A. N., Polyzotis, N., Ramesh, S., Roy, S., Whang, S. E., Wicke, M., Wilkiewicz, J., Zhang, X., Zinkevich, M. "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform". Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '17), pp. 1387-1395, August 2017. https://doi.org/10.1145/3097983.3098021
2. Katsiapis, K., Karmarkar, A., Altay, A., Zaks, A., Polyzotis, N., Ramesh, A., Mathes, B., Vasudevan, G., Giannoumis, I., Wilkiewicz, J., Simsa, J., Hong, J., Trott, M., Lutz, N., Dournov, P. A., Crowe, R., Sirajuddin, S., Warkentin, T. B., Li, Z. "Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX)". arXiv:2010.02013, September 2020. https://arxiv.org/abs/2010.02013
3. TensorFlow. "The TFX User Guide". TensorFlow documentation. https://www.tensorflow.org/tfx/guide
4. TensorFlow. "Understanding TFX Pipelines". TensorFlow documentation. https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines
5. TensorFlow. "ML Metadata". TensorFlow documentation. https://www.tensorflow.org/tfx/guide/mlmd
6. TensorFlow. "tensorflow/tfx: TFX is an end-to-end platform for deploying production ML pipelines". GitHub. https://github.com/tensorflow/tfx
7. Python Package Index. "tfx: release history". PyPI. https://pypi.org/project/tfx/#history
8. TensorFlow. "TFX 1.0.0 Release". GitHub, July 22, 2021. https://github.com/tensorflow/tfx/releases/tag/v1.0.0
9. TensorFlow. "TFX 1.21.0 Release". GitHub, June 18, 2026. https://github.com/tensorflow/tfx/releases/tag/v1.21.0
10. TensorFlow. "Releases: tensorflow/tfx". GitHub. https://github.com/tensorflow/tfx/releases
11. Kuppala, P., Witcher, R. "What's new in TensorFlow 2.21". Google Developers Blog, March 6, 2026. https://developers.googleblog.com/whats-new-in-tensorflow-221/
12. Baylor, D., Haas, K., Katsiapis, K., Leong, S., Liu, R., Mewald, C., Miao, H., Polyzotis, N., Trott, M., Zinkevich, M. "Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform". 2019 USENIX Conference on Operational Machine Learning (OpML '19), May 2019. https://www.usenix.org/conference/opml19/presentation/baylor
13. Ramesh, A., Unruh, A. "Introducing Cloud AI Platform Pipelines". Google Cloud Blog, March 12, 2020. https://cloud.google.com/blog/products/ai-machine-learning/introducing-cloud-ai-platform-pipelines
14. Google Cloud. "Build a pipeline". Gemini Enterprise Agent Platform documentation (formerly Vertex AI documentation). https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/pipelines/build-pipeline
15. Baer, J., Ngahane, S. "The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow". Spotify Engineering, December 13, 2019. https://engineering.atspotify.com/2019/12/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow
16. Google Cloud. "Gemini Enterprise Agent Platform release notes" (entry of April 22, 2026). Google Cloud documentation. https://docs.cloud.google.com/gemini-enterprise-agent-platform/release-notes
17. TensorFlow. "TFX 1.16.0 Release". GitHub, December 11, 2024. https://github.com/tensorflow/tfx/releases/tag/v1.16.0

