Citation and evidence

TensorFlow Extended

15 min full readUpdated 17 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Developer ToolsGoogleMLOpsMachine Learning

Cite this article

TensorFlow Extended (TFX) is an open-source platform from Google for building and running production machine learning pipelines on top of TensorFlow. A TFX pipeline chains standard components for data ingestion, data validation, feature engineering, training, model evaluation and deployment, and records the artifacts and executions of each run in a metadata store, so that a model can be traced back to the data and pipeline run that produced it.[3][4][5] Google described the platform publicly at the KDD 2017 conference, launched it inside the company the same year, and released the public version in early 2019.[1][2][7] The code is published under the Apache 2.0 license on GitHub.[6]

Within Google, TFX succeeded an older platform called Sibyl and by 2020 had become, in the words of its developers, "the most widely used, general purpose E2E ML platform at Alphabet".[2] Outside Google it has been used directly and through Google Cloud services such as Cloud AI Platform Pipelines and Vertex AI Pipelines.[13][14] As of November 2026 the latest release is TFX 1.21.0 (June 2026), and Google has said that work on TFX and related TensorFlow libraries now concentrates on security fixes, critical bug fixes and dependency updates.[9][11]

History

Sibyl, the predecessor

Google's earlier end-to-end platform, Sibyl, was founded in 2007. It trained "wide" models (linear, logistic and Poisson regression, later factorization machines) and bundled tools for data ingestion, data analysis and validation, training, model analysis and training-serving skew detection into one integrated product.[2] According to a 2020 history written by members of the TFX team, Sibyl stayed in production for about 14 years before it was decommissioned, and most of its workloads moved to TFX.[2] Sibyl ran only on CPUs in Google's own data centers, on the Borg cluster manager, and its pipeline topology was largely fixed.[2]

The same account says that TensorFlow, released publicly in 2015, offered modeling flexibility but no complete production system, while Sibyl had end-to-end tooling but limited flexibility. Google built TFX to combine the two and launched it internally in 2017.[2] TFX reused many of Sibyl's system designs, and where Sibyl had relied on MapReduce and Flume for distributed data processing, TFX adopted their portable successor, Apache Beam.[2]

The KDD 2017 paper

The platform was first described in "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform", a paper by Denis Baylor and 21 co-authors at Google, presented in the Applied Data Science track of KDD 2017 in Halifax, Nova Scotia.[1] The paper argued that teams often wired together learners, data checks, model checks and serving systems with ad hoc glue code, which duplicated effort and produced fragile systems. It reported that integrating these pieces into one platform with a shared configuration reduced the time to production "from the order of months to weeks".[1]

The paper's case study was the ranking model behind app recommendations on the Google Play store, which at the time had more than one billion active users and more than one million apps.[1] The ranking model was retrained continuously on training sets of hundreds of billions of examples and served through TensorFlow Serving at thousands of queries per second with a latency budget of tens of milliseconds.[1] By comparing statistics of serving logs with training data from the same day, the data validation component found features that were always present in training but always missing at serving time; an online A/B experiment showed that removing this skew raised the app install rate on the store's main landing page by 2%.[1] Play also used warm-starting, initializing a subset of parameters such as embeddings from a previous model, so that it could push fresh models frequently without training from scratch.[1]

A follow-up paper at the 2019 USENIX Conference on Operational Machine Learning (OpML '19) described how TFX runs continuous pipelines in production at Google and the lessons learned from that deployment.[12]

Open-source release and TFX 1.0

The GitHub repository was created in February 2019, and the first stable package on the Python Package Index, version 0.12.0, was published on March 6, 2019, after release candidates that began on March 2.[6][7] The 2020 history paper says the public offering "was released in early 2019" and was adopted within a year in on-premises environments and on Google Cloud.[2]

TFX 1.0.0, released on July 22, 2021, was the first post-beta release.[3][8] It introduced the tfx.v1 public API namespace with stability guarantees for pipelines across upgrades, and added support in the TFX command-line tool for Vertex Pipelines through an --engine=vertex flag.[3][8]

Later releases and 2026 status

Releases slowed after 2024. Version 1.16.0 appeared in December 2024; it dropped support for the TensorFlow Estimator API and deprecated the runner for version 1 of the Kubeflow Pipelines SDK in favour of the version 2 runner.[17] Version 1.17.0 followed in March 2026, targeting TensorFlow 2.17.[10] After patch releases up to 1.17.3 in April 2026, the next release was 1.21.0 on June 18, 2026.[10] Its notes list support for TensorFlow 2.21.0 and Protobuf 6.x on Python 3.10 through 3.13, drop four outdated or incompatible dependencies (TensorFlow Decision Forests, TensorFlow Ranking, TensorFlow Text and TensorFlow.js) to avoid installation conflicts, skip legacy tests for the retired Google Cloud AI Platform integration, and add a pure-Python fallback for ML Metadata environments that lack the ZetaSQL engine.[9]

In a March 6, 2026 post on TensorFlow 2.21, Google said the TensorFlow team would focus on security and critical bug fixes, minor releases for dependency updates such as new Python versions, and accepting community bug fixes. The post listed TFX, TensorFlow Serving, TensorFlow Data Validation, TensorFlow Transform and TensorFlow Model Analysis among the libraries those commitments cover, and pointed developers starting new generative AI work toward Keras 3, JAX and PyTorch.[11]

DateEvent
2007Sibyl, TFX's predecessor, founded at Google[2]
2017TFX launched inside Google[2]
August 2017TFX paper presented at KDD 2017[1]
February 2019tensorflow/tfx repository created on GitHub[6]
March 2019First stable PyPI release, version 0.12.0[7]
March 2020Cloud AI Platform Pipelines announced with a TFX SDK in preview[13]
July 2021TFX 1.0.0, first post-beta release[8]
December 2024TFX 1.16.0[10]
March 2026TFX 1.17.0; TensorFlow 2.21 post narrows ecosystem work to fixes and dependency updates[10][11]
June 2026TFX 1.21.0, supporting TensorFlow 2.21[9]

Expanded article table

"What exactly is this TFX thing? (TensorFlow Extended)", official video from TensorFlow, published July 11, 2019. Watch on YouTube.

Architecture

Pipelines, components and artifacts

A TFX pipeline is a portable description of a machine learning workflow made of component instances and input parameters.[4] Each component has three parts: a component specification that declares its input artifacts, output artifacts and parameters; an executor that does the actual work; and a component interface that packages the two for use in a pipeline.[4] The outputs of components are called artifacts, and every artifact must have a type registered in the ML Metadata store.[4] TFX works out the execution order by building a directed acyclic graph from the artifact dependencies, so components whose inputs do not depend on each other, such as ExampleValidator and Transform, can run in parallel.[4] Users can write custom components alongside the standard ones.[4]

Standard components

The TFX user guide lists the following standard components.[3]

ComponentRole
ExampleGenIngests the input data, optionally splits it, and formats it as tf.Example records
StatisticsGenComputes statistics over the dataset
SchemaGenInfers a data schema from the statistics
ExampleValidatorLooks for anomalies, missing values and wrong types using the statistics and schema
TransformPerforms feature engineering and exports the transformation graph
TunerTunes model hyperparameters, using KerasTuner
TrainerTrains the model
EvaluatorComputes metrics, compares a candidate model with a baseline, and can block deployment when thresholds are not met
InfraValidatorMakes a canary deployment of the model in a sandbox to check that it can actually be served
PusherDeploys the model to serving infrastructure
BulkInferrerRuns batch inference on unlabeled data

Expanded article table

The guide stresses the role of Transform in avoiding training-serving skew: its output is a SavedModel containing the preprocessing graph, which the Trainer imports so that the same transformations run during training and at inference time.[3] The KDD paper made the same point, noting that integrating data analysis, training and serving lets the platform guarantee that identical transformations are applied at training and serving, which "prevents one form of training-serving skew".[1]

Libraries

Many standard components are built on libraries that provide their base functionality and can also be used on their own.[3] The 2020 history paper describes TFX as layered: managed cloud services at the top, TFX pipelines below them, the TensorFlow Serving binary, and then a set of libraries.[2]

LibraryPurpose
TensorFlow Data Validation (TFDV)Computes summary statistics, infers schemas and detects anomalies such as missing features, out-of-range values or wrong types; includes viewers for statistics and anomalies[3]
TensorFlow Transform (TFT)Preprocessing that needs a full pass over the data, such as normalizing by mean and standard deviation, building vocabularies or bucketizing values[3]
TensorFlow Model Analysis (TFMA)Distributed model evaluation, with metrics computed over slices of the data[3]
TensorFlow Metadata (TFMD)Standard formats for schemas and dataset statistics[3]
ML Metadata (MLMD)Records and retrieves metadata and lineage for pipeline artifacts and executions[3]
TFX Basic Shared Libraries (TFX-BSL)Shared library layer listed in the TFX architecture[2]
TensorFlow ServingServes trained models over REST and gRPC[3]

Expanded article table

Evaluation slices let users check whether a model that performs well on average does badly for a smaller group, for example a particular customer segment.[3] The history paper adds that the Evaluator supports custom metrics, confidence intervals and Fairness Indicators through TFMA.[2]

ML Metadata and lineage

ML Metadata is a library for recording and querying metadata about machine learning workflows. It is part of TFX but designed to be usable independently.[5] Its data model stores artifacts and their types, executions of components, events linking executions to the artifacts they consumed and produced, and contexts that group artifacts and executions, such as a pipeline run or an experiment.[5] With these records a user can answer questions such as which dataset a model was trained on, which hyperparameters were used, which pipeline run created a model, or when a failed model was pushed.[5] MLMD can also tell whether a component already ran on the same inputs, so that earlier outputs can be reused.[5] The storage backend is pluggable: MLMD ships reference implementations for SQLite and MySQL, and its documentation also shows how to connect to PostgreSQL.[5]

The history paper compares artifact lineage to a stack trace in ordinary software debugging, and says MLMD, together with the underlying storage layer, forms the foundation of TFX's compliance mechanisms.[2]

Data processing and orchestration

TFX uses Apache Beam for data-parallel processing, and the default installation includes Beam's DirectRunner for local execution; distributed runs can use back ends such as Apache Flink, Apache Spark and Google Cloud Dataflow.[3] For its compute-heavy libraries TFX adopted Apache Arrow as an in-memory columnar format.[2]

Pipelines are portable across orchestrators. The documentation names Apache Airflow, Apache Beam and Kubeflow Pipelines, and uses the term DagRunner for the code that connects a pipeline to a given orchestrator.[3][4] A command-line tool can create, update, run, list and delete pipelines on these orchestrators.[3]

Deployment targets

Trained models are exported as SavedModels and can be deployed to three classes of target: TensorFlow Serving for server-side inference, TensorFlow Lite for mobile and IoT devices, and TensorFlow.js for browsers and Node.js.[3] In a typical pipeline, InfraValidator first launches a canary TensorFlow Serving instance with the new model, and Pusher deploys it only if that check passes.[3]

Continuous training

TFX was designed for pipelines that keep running as new data arrives, not only for one-off training jobs.[1][2] The KDD paper describes continuation strategies built from two settings: data visitation, which can be static or can follow a rolling range of directories, and warm-starting, which initializes some model parameters from an earlier state.[1] The history paper says TFX was built from the ground up for continuous learning: pipelines detect new data and schedule updated models, recover from intermittent failures, and support "backfilling", where a pipeline is rerun on existing data with updated components or on corrected data. It noted in 2020 that the public release did not yet offer continuous pipelines.[2]

Google Cloud integration

In March 2020 Google Cloud announced Cloud AI Platform Pipelines, a beta service that offered two authoring options: the Kubeflow Pipelines SDK and the TFX SDK, which was then in preview. Google described the TFX SDK as the higher-level option, "with prescriptive, but customizable components", recommended for end-to-end pipelines built on TensorFlow.[13] Vertex AI Pipelines, the serverless pipeline service of Google's later Vertex AI platform, accepts pipelines written with TFX (version 0.30.0 or later) or the Kubeflow Pipelines SDK, and its documentation recommends TFX for TensorFlow workflows that process terabytes of structured or text data and the Kubeflow Pipelines SDK for other cases.[14] On April 22, 2026 Google made Vertex AI part of Gemini Enterprise Agent Platform and renamed the service Agent Platform Pipelines; the same TFX support is documented under the new name.[14][16]

Adoption

The 2020 history paper reported that inside Alphabet TFX was used by thousands of users and powered hundreds of products, including Cloud AI services on Google Cloud. It said that on a typical day thousands of TFX pipelines were running, processing exabytes of data and producing tens of thousands of models, which served hundreds of millions of inferences per second.[2]

Spotify described its move to TFX and Kubeflow Pipelines in a December 2019 engineering post. TensorFlow Data Validation was the first TFX component it used, to find skew, bad values and excess nulls, and its platform wrapped TFX components inside centrally managed Kubeflow Pipelines clusters. Spotify launched an alpha version of the platform in August 2019 and reported about 100 users and 18,000 runs by December; early analysis indicated that some teams were producing "7x more experiments".[15]

Trade-offs

The TFX team's own 2020 account lists the costs of several design choices.[2] Taking dependencies on external systems such as Kubeflow Pipelines or Airflow required effort that grew faster than linearly with the number of dependencies, and those costs sometimes reached users as version conflicts or incompatibilities between environments. Supporting many combinations of orchestrators, runners and clouds produced more deployment configurations than the team could test. Offering several public layers (libraries, pipelines and services) confused some users about which layer to use, and the platform's flexibility made it harder to build, maintain and support.[2] Dependency management remained a theme in 2026, when the 1.21.0 release dropped several TensorFlow libraries from TFX's dependency list to stop pip from backtracking through version conflicts during installation.[9]

Continue reading

More from See also

More to read

Topics:Developer ToolsGoogleMLOpsMachine Learning

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., Koo, C. Y., Lew, L., Mewald, C., Modi, A. N., Polyzotis, N., Ramesh, S., Roy, S., Whang, S. E., Wicke, M., Wilkiewicz, J., Zhang, X., Zinkevich, M. "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform". Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '17), pp. 1387-1395, August 2017. doi.org/...3097983.3098021
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20Katsiapis, K., Karmarkar, A., Altay, A., Zaks, A., Polyzotis, N., Ramesh, A., Mathes, B., Vasudevan, G., Giannoumis, I., Wilkiewicz, J., Simsa, J., Hong, J., Trott, M., Lutz, N., Dournov, P. A., Crowe, R., Sirajuddin, S., Warkentin, T. B., Li, Z. "Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX)". arXiv:2010.02013, September 2020. arxiv.org/...2010.02013
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18TensorFlow. "The TFX User Guide". TensorFlow documentation. tensorflow.org/...guide
  4. ^1 ^2 ^3 ^4 ^5 ^6 ^7TensorFlow. "Understanding TFX Pipelines". TensorFlow documentation. tensorflow.org/...understanding_tfx_pipelines
  5. ^1 ^2 ^3 ^4 ^5 ^6TensorFlow. "ML Metadata". TensorFlow documentation. tensorflow.org/...mlmd
  6. ^1 ^2 ^3TensorFlow. "tensorflow/tfx: TFX is an end-to-end platform for deploying production ML pipelines". GitHub. github.com/...tfx
  7. ^1 ^2 ^3Python Package Index. "tfx: release history". PyPI. pypi.org/...tfx
  8. ^1 ^2 ^3TensorFlow. "TFX 1.0.0 Release". GitHub, July 22, 2021. github.com/...v1.0.0
  9. ^1 ^2 ^3 ^4TensorFlow. "TFX 1.21.0 Release". GitHub, June 18, 2026. github.com/...v1.21.0
  10. ^1 ^2 ^3 ^4TensorFlow. "Releases: tensorflow/tfx". GitHub. github.com/...releases
  11. ^1 ^2 ^3Kuppala, P., Witcher, R. "What's new in TensorFlow 2.21". Google Developers Blog, March 6, 2026. developers.googleblog.com/...new-in-tensorflow-221
  12. ^Baylor, D., Haas, K., Katsiapis, K., Leong, S., Liu, R., Mewald, C., Miao, H., Polyzotis, N., Trott, M., Zinkevich, M. "Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform". 2019 USENIX Conference on Operational Machine Learning (OpML '19), May 2019. usenix.org/...baylor
  13. ^1 ^2 ^3Ramesh, A., Unruh, A. "Introducing Cloud AI Platform Pipelines". Google Cloud Blog, March 12, 2020. cloud.google.com/...ng-cloud-ai-platform-pipelines
  14. ^1 ^2 ^3Google Cloud. "Build a pipeline". Gemini Enterprise Agent Platform documentation (formerly Vertex AI documentation). docs.cloud.google.com/...build-pipeline
  15. ^Baer, J., Ngahane, S. "The Winding Road to Better Machine Learning Infrastructure Through Tensorflow Extended and Kubeflow". Spotify Engineering, December 13, 2019. engineering.atspotify.com/...extended-and-kubeflow
  16. ^Google Cloud. "Gemini Enterprise Agent Platform release notes" (entry of April 22, 2026). Google Cloud documentation. docs.cloud.google.com/...release-notes
  17. ^TensorFlow. "TFX 1.16.0 Release". GitHub, December 11, 2024. github.com/...v1.16.0

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 2,911 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 78 claims checked against 18 cited and independent sources; 1 material and 5 minor corrections applied before publication.

Cite this page: AI Wiki. "TensorFlow Extended." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/tfx

Edit article