TensorFlow Serving
See also: Machine learning terms
TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages their full lifecycle, including versioning and zero downtime hot swapping, inside long running inference servers.[1] First open sourced on February 16, 2016 as part of the TensorFlow ecosystem, it is built in C++ for low overhead and serves native TensorFlow SavedModel artifacts, though its plugin architecture also lets it serve lookup tables, embeddings, and arbitrary custom servables.[2][5] By the August 2017 release of version 1.0, Google reported more than 800 internal projects already running on top of the system.[3]
The reference paper is Christopher Olston et al., TensorFlow-Serving: Flexible, High-Performance ML Serving, presented at the 2017 NIPS Workshop on ML Systems, which states that the total number of Google projects using the system "is in the hundreds" and that overall Google traffic served through it runs to "tens of millions of inferences per second."[1] As a piece of MLOps infrastructure, TF Serving was the first widely adopted dedicated model server and remains the canonical path for taking a TensorFlow model into production.
When was TensorFlow Serving released?
Google published the initial open source release of TensorFlow Serving on February 16, 2016, three months after open sourcing the TensorFlow framework itself.[2] The original announcement on the Google Open Source Blog framed the project around three goals: model lifecycle management, support for running multiple algorithms concurrently, and efficient use of GPU resources for online inference.[2] The code was released under the Apache 2.0 license.[4]
Version 1.0 shipped on August 7, 2017.[3] That release stabilized the API, added a prebuilt binary installable through apt-get so users no longer had to compile the C++ server, made SavedModel the officially supported export format, and deprecated the legacy SessionBundle format; the gRPC ModelServer binary with its Predict API and a Kubernetes deployment example had been introduced earlier, between the 2016 launch and 1.0, and a Docker container was already available.[3] The 1.0 announcement reported: "Today there are over 800 projects within Google using TensorFlow Serving in production."[3]
Later that year, Olston and colleagues at Google published the technical paper TensorFlow-Serving: Flexible, High-Performance ML Serving (arXiv:1712.06139), which documents the design choices behind the Source, Loader, Manager, and Servable abstractions and reports microbenchmarks of around 100,000 queries per second per core on a 16 vCPU Intel Xeon E5 2.6 GHz server, excluding gRPC and TensorFlow inference time.[1]
The project has continued to ship regular releases. As of November 2026 the active line is still the 2.x series, with version 2.21.0 released on September 29, 2026 (following 2.20.0 in June 2026). The repository at github.com/tensorflow/serving lists more than 9,400 commits across 124 releases.[4]
| Milestone | Date | What it added |
|---|---|---|
| Initial open source release | February 16, 2016 | C++ server, lifecycle management, multi model serving |
| Version 1.0 | August 7, 2017 | Stable API, apt-get installable binary, SavedModel as the supported format, SessionBundle deprecated |
| Reference paper (arXiv:1712.06139) | December 2017 | Source, Loader, Manager, Servable design; ~100,000 QPS per core benchmark |
| Version 2.x line | 2019 onward | TensorFlow 2 compatibility; 2.21.0 in September 2026 |
How does TensorFlow Serving work?
TF Serving is written in C++ and structured as a small number of cleanly separated components. The same components can be used together as the standard tensorflow_model_server binary, or composed individually inside a custom server.[5]
Core abstractions
| Component | Role |
|---|---|
| Servable | The underlying object that clients use for computation. A servable can be a TensorFlow model, a single shard of a lookup table, an embedding table, a vocabulary, or a tuple of inference models. Servables do not manage their own lifecycle. |
| Servable Stream | A sequence of versions of the same servable, ordered by increasing version number. |
| Loader | Standardizes the API for loading and unloading a servable, including estimating its resource cost. New model backends are added by writing a Loader. |
| Source | A plugin module that discovers servables on disk or in remote storage and emits Loader instances. Sources publish the set of versions they want loaded as the aspired versions list. |
| Aspired Versions | The set of servable versions a Source currently wants the system to load. The Manager reconciles this against what is currently loaded. |
| Manager | Owns the lifecycle of all servables: it listens to Sources, applies a version policy, calls Loaders to load and unload servables, and hands clients short lived references to loaded versions. |
| Servable Handle | The narrow, reference counted handle a client receives from GetServableHandle() and uses to call into a loaded servable. |
This layering means that adding a new model format only requires writing a Source and a Loader. Everything else, including version management, batching, and the request APIs, is reused.[5]
Version policies
The Manager applies a version policy to decide how to transition between aspired version sets. Two policies ship out of the box:[5]
- Availability Preserving Policy: load the new version before unloading the old one. This avoids any window of zero loaded versions, at the cost of holding both versions in memory simultaneously.
- Resource Preserving Policy: unload the old version before loading the new one. This keeps peak memory low but creates a brief window during which the model is not served.
Multiple versions of the same model can be loaded at once, which is what enables zero downtime rollouts, canary releases, and A/B comparisons against a stable baseline.[5]
How does dynamic batching work?
TF Serving ships a request batching library that groups individual inference requests into a single batched call to the underlying graph.[5] The batch scheduler enforces both a maximum batch size and a maximum queueing latency, so callers can trade off throughput against tail latency. Scheduling is done globally across all models and model versions on the server to maximize utilization of the underlying hardware.[5] Batching is most valuable on accelerator hardware (GPUs and TPUs), where the per call fixed cost is high relative to the per example cost.
What is the SavedModel format?
The canonical input to TF Serving is a SavedModel directory: a self contained bundle of the TensorFlow graph, the trained weights, the asset files, and one or more named signatures that describe input and output tensors.[3] SavedModels are produced by tf.saved_model.save() in modern TensorFlow and by Keras model.save() when the SavedModel format is selected.
A typical on disk layout looks like this:
/models/my_model/
1/
saved_model.pb
variables/
2/
saved_model.pb
variables/
Each numeric subdirectory is a version. By default the file system Source watches the parent directory and exposes new numeric subdirectories as new aspired versions, which is how rolling deployments work without restarting the server.[5]
What APIs does TensorFlow Serving expose?
TF Serving exposes the same set of inference operations through two transports: a binary gRPC interface and a JSON over HTTP REST API.[6] The two endpoints can run side by side from a single server process.
| API | Default port | Wire format | Typical use |
|---|---|---|---|
| gRPC | 8500 | Protocol Buffers over HTTP/2 | Service to service calls where latency matters and the client can link a generated stub. |
| REST | 8501 | JSON over HTTP/1.1 | Browser, mobile, and language ecosystems without a convenient gRPC client. |
Both transports expose largely the same logical methods, mapped from the underlying TensorFlow Serving protobuf services (MultiInference is gRPC-only).
| Method | Purpose |
|---|---|
| Predict | Generic inference. Sends a dictionary of input tensors and receives a dictionary of output tensors. Works for any signature. |
| Classify | Convenience method for classification graphs. Returns labels with scores. |
| Regress | Convenience method for regression graphs. Returns numeric outputs. |
| MultiInference | Runs Classify and Regress against the same input batch in one call. |
| GetModelMetadata | Returns the signatures, input shapes, and dtypes for a loaded model version. |
| GetModelStatus | Returns load state for each version of a model (LOADING, AVAILABLE, UNLOADING, END). |
The REST URL pattern is POST /v1/models/{name}[/versions/{n}|/labels/{label}]:{predict|classify|regress}.[6] The gRPC equivalents are the PredictionService and ModelService services in the tensorflow.serving protobuf package. If neither a version nor a label is specified, the server routes to the latest available version.[6]
How is TensorFlow Serving deployed?
TF Serving is shipped as a static C++ binary, a Debian package, and an official Docker image at tensorflow/serving on Docker Hub.[7] The image comes in CPU and GPU (CUDA) variants.[7] A minimal launch command looks like this:
docker run -p 8500:8500 -p 8501:8501 \
-v /path/to/models:/models \
-e MODEL_NAME=my_model \
tensorflow/serving
For multiple models, the server reads a models.config file that lists each model name, base path, and version policy. The same file controls per model labels, which let callers refer to a logical version like prod or canary instead of a number.
On Kubernetes, TF Serving is typically deployed as a Deployment plus Service, often fronted by an ingress for the REST port and a separate gRPC service for internal callers. Higher level platforms wrap this pattern: KServe (formerly KFServing) has a first class tensorflow predictor that runs the official image, Google Cloud Vertex AI Prediction can host SavedModels through TF Serving under the hood, and Kubeflow Pipelines includes TF Serving deployment components.[8] For monitoring, the binary exposes Prometheus style metrics through a --monitoring_config_file flag.
The system has been demonstrated at very large scale: in a 2021 case study, ad exchange OpenX reported serving 2.5 million prediction requests per second through TensorFlow Serving on Google Kubernetes Engine, each under 15 milliseconds.[11]
Edge and on device serving is generally not done with TF Serving itself. The recommended path for mobile and embedded targets is TensorFlow Lite, which compiles SavedModels into a smaller flatbuffer format suitable for ARM CPUs, mobile GPUs, and microcontrollers.
How does TensorFlow Serving differ from TorchServe and Triton?
TF Serving was the first widely adopted dedicated model serving system, but the landscape has grown considerably since 2020, especially around large language models.
| Server | Maintainer | Primary frameworks | Notable strengths | Notable gaps |
|---|---|---|---|---|
| TensorFlow Serving | TensorFlow (SavedModel), pluggable | Mature versioning, dynamic batching, low overhead C++ core, deep TF integration | Designed for classic graph models, not optimized for autoregressive LLMs | |
| TorchServe | Meta and AWS (archived August 2025) | PyTorch | Native handler API for PyTorch, multi model endpoints | Limited-maintenance notice from February 2025 (no planned updates, bug fixes, or security patches); repository archived August 2025 |
| Triton Inference Server | NVIDIA | TensorFlow, PyTorch, ONNX, TensorRT, Python, vLLM backend, others | Multi framework, GPU dynamic batching, model ensembles, business logic scripting | Heavier deployment, more configuration knobs |
| vLLM | UC Berkeley plus open community | Hugging Face transformers, GGUF | PagedAttention, continuous batching, very high LLM throughput | LLM specific; no general CV or tabular serving story |
| Text Generation Inference (TGI) | Hugging Face | Hugging Face transformers | Token streaming, tight HF Hub integration, prefix caching for long contexts | LLM specific; repository archived March 2026[14] |
| ONNX Runtime Server | Microsoft | ONNX | Cross framework via ONNX export, broad hardware backends | Deprecated and no longer supported; ONNX models are now usually served through ONNX Runtime embedded in another server, such as Triton's ONNX Runtime backend[15] |
| Ray Serve | Anyscale | Any Python | Compose Python services with autoscaling and Ray cluster integration | Performance bound by Python; less specialized than Triton |
| BentoML, Seldon Core, KServe | Independent communities | Multi framework | Higher level orchestration, packaging, A/B routing on top of underlying servers | Often delegate the actual inference to one of the servers above |
In practice, large Docker shops with a TensorFlow heavy stack still reach for TF Serving because it is the lowest friction path for a SavedModel. PyTorch shops historically used TorchServe or Triton, but TorchServe's maintainers posted a notice in February 2025 that the project is "no longer actively maintained" and archived the repository in August 2025.[12] Teams that want a single server for many frameworks, particularly on NVIDIA GPUs, tend to standardize on Triton, which NVIDIA folded into its Dynamo platform in March 2025, renaming it NVIDIA Dynamo-Triton.[13] LLM workloads have largely migrated to vLLM, TensorRT-LLM, SGLang and, until Hugging Face archived it in March 2026, TGI, none of which TF Serving was designed for.[10][14]
What is TensorFlow Serving used for?
The sweet spot for TF Serving is production scoring of TensorFlow models that look like classic supervised graphs. Common deployments include:
- Computer vision inference: image classification, object detection, and segmentation models exported from Keras or
tf.keras.applications. - Tabular and recommendation models: wide and deep networks, ranking models, and embedding lookup services that combine a graph with large embedding tables.
- NLP encoders: BERT style classification and sentence embedding models where the input is a fixed length token sequence.
- A/B testing of model versions, using labels like
prodandcanaryso traffic can be split at the routing layer without changing client code. - Hot swapping models without restarting the server, useful for retrain pipelines that publish a new SavedModel directory every few hours.
The 2016 launch post framed the project against Google's own ML-powered features, such as speech recognition in the Google app, Smart Reply in Inbox and search in Google Photos, and the 2017 paper reports that hundreds of Google projects use TF Serving internally.[1][2]
Limitations
TF Serving is a mature, focused tool, and its limitations mostly come from where its focus is not.
The server is built around the TensorFlow runtime. Serving PyTorch or pure ONNX models requires writing a custom Loader and is rarely the path of least resistance. Triton or ONNX Runtime are usually a better fit for that case.
It is not optimized for autoregressive LLM inference. There is no PagedAttention, no continuous batching across decode steps, no KV cache management, and no native token streaming endpoint.[10] LLM workloads have moved to vLLM, TensorRT-LLM and SGLang (and, until its March 2026 archiving, TGI) for those reasons.[10][14]
The REST endpoint is JSON over HTTP/1.1, which is convenient but adds serialization overhead compared to gRPC.[6] For very large tensors, the JSON path can become a bottleneck before the model itself does.
For a quick prototype with one model and modest QPS, a Flask or FastAPI wrapper around tf.saved_model.load is often enough and avoids the operational cost of a separate server. TF Serving starts to pay off when there are multiple versions, multiple models, batching requirements, or zero downtime deployment needs.
Is TensorFlow Serving still used in 2026?
Yes. As of 2026, TF Serving remains widely deployed for non LLM TensorFlow workloads, particularly in computer vision pipelines and recommendation systems. It is part of the TFX production ML platform, the default predictor for TensorFlow models in KServe, and one of the underlying serving runtimes used by Google Cloud Vertex AI Prediction.[8] The release cadence has slowed compared to the 2017 to 2020 period but the project is still maintained.[4]
For general purpose multi framework serving on GPUs, NVIDIA Triton has taken much of the share that TF Serving once held by default, and for LLMs the conversation is dominated by dedicated engines such as vLLM (Hugging Face archived TGI in March 2026).[10][14] Within the TensorFlow ecosystem itself, Serving is still the canonical way to take a SavedModel into production without rebuilding the deployment story from scratch.
Explain like I'm 5
Imagine you trained a robot to recognize cats in photos. TF Serving is like a coat check counter for that robot. You leave the robot at the counter in a numbered slot, and any friend who has a photo can walk up, hand it through the window, and get back the answer "cat" or "not cat." If you train a smarter robot, you put it in a higher numbered slot, and the counter quietly starts handing photos to the new one without ever closing. The old robot stays in its slot for a little while, just in case the new one turns out to be worse, and then it goes home.
Continue reading
- TensorFlowTensorFlow is an open-source software platform for numerical computation and machine learning.Mentioned 38 times above
- GoogleGoogle LLC is a United States technology company and the largest operating business within Alphabet Inc. Its activities include internet search, online advertising, video, mobile and web platforms, consumer…Mentioned 16 times above
- InferenceIn machine learning, inference is the execution of a trained model to produce an output from an input.Mentioned 13 times above
- SavedModelA SavedModel is the standardized, language-neutral serialization format that TensorFlow uses to package a complete, trained model (its computation graph, its weights, and any supporting assets) as a…Mentioned 13 times above
- Machine LearningMachine learning (ML) is a branch of artificial intelligence that gives computers the ability to learn from data and improve their performance on tasks without being explicitly programmed.Linked in the introduction
- Machine learning termsMachine learning terms are the vocabulary used to describe data, models, training procedures, evaluation methods, and deployed behavior.Suggested at the top of this article
More to read
References
- ^1 ^2 ^3 ^4Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., and Soyke, J. (2017). *TensorFlow-Serving: Flexible, High-Performance ML Serving*. NIPS 2017 Workshop on ML Systems. arXiv:1712.06139.
- ^1 ^2 ^3 ^4Google Open Source Blog (2016, February 16). *Running your models in production with TensorFlow Serving*.
- ^1 ^2 ^3 ^4 ^5Google Developers Blog (2017, August 7). *TensorFlow Serving 1.0*.
- ^1 ^2 ^3TensorFlow Serving GitHub repository. github.com/tensorflow/serving.
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8TensorFlow Serving Architecture documentation. Architecture overview.
- ^1 ^2 ^3 ^4TensorFlow team. *RESTful API*. TFX serving documentation.
- ^1 ^2TensorFlow team. *TensorFlow Serving with Docker*. TFX serving documentation.
- ^1 ^2KServe project. *TensorFlow predictor*.
- NVIDIA. *Triton Inference Server* project documentation.
- ^1 ^2 ^3 ^4vLLM project. *PagedAttention and continuous batching* documentation.
- ^The TensorFlow Blog (2021, February). *How OpenX Trains and Serves for a Million Queries per Second in under 15 Milliseconds*.
- ^pytorch/serve GitHub repository. *TorchServe limited maintenance notice*.
- ^NVIDIA. *NVIDIA Dynamo-Triton* (formerly NVIDIA Triton Inference Server).
- ^1 ^2 ^3 ^4huggingface/text-generation-inference GitHub repository (archived March 21, 2026). *Text Generation Inference*.
- ^Microsoft. *ONNX Runtime Server usage (deprecated)*. onnxruntime GitHub repository.
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
11 revisions · v11 · 2,910 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Reviewer note: Independent fact-check of the full article against its cited sources (85 claims, 22 sources fetched); 7 material + 11 minor fixed
Cite this page: AI Wiki. "TensorFlow Serving." aiwiki.ai, updated 23 Jun 2026, fact-checked 4 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/tensorflow_serving


