TensorFlow Serving

See also: Machine learning terms

TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages their full lifecycle, including versioning and zero downtime hot swapping, inside long running inference servers.^[1] First open sourced on February 16, 2016 as part of the TensorFlow ecosystem, it is built in C++ for low overhead and serves native TensorFlow SavedModel artifacts, though its plugin architecture also lets it serve lookup tables, embeddings, and arbitrary custom servables.^[2][5] By the August 2017 release of version 1.0, Google reported more than 800 internal projects already running on top of the system.^[3]

The reference paper is Christopher Olston et al., TensorFlow-Serving: Flexible, High-Performance ML Serving, presented at the 2017 NIPS Workshop on ML Systems, which states that the total number of Google projects using the system "is in the hundreds" and that overall Google traffic served through it runs to "tens of millions of inferences per second."^[1] As a piece of MLOps infrastructure, TF Serving was the first widely adopted dedicated model server and remains the canonical path for taking a TensorFlow model into production.

When was TensorFlow Serving released?

Google published the initial open source release of TensorFlow Serving on February 16, 2016, three months after open sourcing the TensorFlow framework itself.^[2] The original announcement on the Google Open Source Blog framed the project around three goals: model lifecycle management, support for running multiple algorithms concurrently, and efficient use of GPU resources for online inference.^[2] The code was released under the Apache 2.0 license.^[4]

Version 1.0 shipped on August 7, 2017.^[3] That release introduced the gRPC ModelServer binary with the Predict API, added Kubernetes deployment examples, made SavedModel the officially supported export format, and deprecated the legacy SessionBundle format.^[3] It also added apt installable binaries and Docker images so users no longer had to build the C++ server from source.^[3] The 1.0 announcement reported that "there are over 800 projects within Google using TensorFlow Serving in production today."^[3]

Later that year, Olston and colleagues at Google published the technical paper TensorFlow-Serving: Flexible, High-Performance ML Serving (arXiv:1712.06139), which documents the design choices behind the Source, Loader, Manager, and Servable abstractions and reports microbenchmarks of around 100,000 queries per second per core on a 16 vCPU Intel Xeon E5 2.6 GHz server, excluding gRPC and TensorFlow inference time.^[1]

The project has continued to ship regular releases. As of mid 2025 the active line is the 2.x series, with version 2.19.1 released in August 2025. The repository at github.com/tensorflow/serving lists more than 9,000 commits across 122 releases.^[4]

Scroll sideways for more →

How does TensorFlow Serving work?

TF Serving is written in C++ and structured as a small number of cleanly separated components. The same components can be used together as the standard tensorflow_model_server binary, or composed individually inside a custom server.^[5]

Core abstractions

Scroll sideways for more →

This layering means that adding a new model format only requires writing a Source and a Loader. Everything else, including version management, batching, and the request APIs, is reused.^[5]

Version policies

The Manager applies a version policy to decide how to transition between aspired version sets. Two policies ship out of the box:^[5]

• Availability Preserving Policy: load the new version before unloading the old one. This avoids any window of zero loaded versions, at the cost of holding both versions in memory simultaneously.
• Resource Preserving Policy: unload the old version before loading the new one. This keeps peak memory low but creates a brief window during which the model is not served.

Multiple versions of the same model can be loaded at once, which is what enables zero downtime rollouts, canary releases, and A/B comparisons against a stable baseline.^[5]

How does dynamic batching work?

TF Serving ships a request batching library that groups individual inference requests into a single batched call to the underlying graph.^[5] The batch scheduler enforces both a maximum batch size and a maximum queueing latency, so callers can trade off throughput against tail latency. Scheduling is done globally across all models and model versions on the server to maximize utilization of the underlying hardware.^[5] Batching is most valuable on accelerator hardware (GPUs and TPUs), where the per call fixed cost is high relative to the per example cost.

What is the SavedModel format?

The canonical input to TF Serving is a SavedModel directory: a self contained bundle of the TensorFlow graph, the trained weights, the asset files, and one or more named signatures that describe input and output tensors.^[3] SavedModels are produced by tf.saved_model.save() in modern TensorFlow and by Keras model.save() when the SavedModel format is selected.

A typical on disk layout looks like this:

/models/my_model/
  1/
    saved_model.pb
    variables/
  2/
    saved_model.pb
    variables/

Each numeric subdirectory is a version. By default the file system Source watches the parent directory and exposes new numeric subdirectories as new aspired versions, which is how rolling deployments work without restarting the server.^[5]

What APIs does TensorFlow Serving expose?

TF Serving exposes the same set of inference operations through two transports: a binary gRPC interface and a JSON over HTTP REST API.^[6] The two endpoints can run side by side from a single server process.

Scroll sideways for more →

Both transports expose the same logical methods, mapped from the underlying TensorFlow Serving protobuf services.

Scroll sideways for more →

The REST URL pattern is POST /v1/models/{name}[/versions/{n}|/labels/{label}]:{predict|classify|regress}.^[6] The gRPC equivalents live in the tensorflow_serving.PredictionService package. If neither a version nor a label is specified, the server routes to the latest available version.^[6]

How is TensorFlow Serving deployed?

TF Serving is shipped as a static C++ binary, a Debian package, and an official Docker image at tensorflow/serving on Docker Hub.^[7] The image comes in CPU and GPU (CUDA) variants.^[7] A minimal launch command looks like this:

docker run -p 8500:8500 -p 8501:8501 \
  -v /path/to/models:/models \
  -e MODEL_NAME=my_model \
  tensorflow/serving

For multiple models, the server reads a models.config file that lists each model name, base path, and version policy. The same file controls per model labels, which let callers refer to a logical version like prod or canary instead of a number.

On Kubernetes, TF Serving is typically deployed as a Deployment plus Service, often fronted by an ingress for the REST port and a separate gRPC service for internal callers. Higher level platforms wrap this pattern: KServe (formerly KFServing) has a first class tensorflow predictor that runs the official image, Google Cloud Vertex AI Prediction can host SavedModels through TF Serving under the hood, and Kubeflow Pipelines includes TF Serving deployment components.^[8] For monitoring, the binary exposes Prometheus style metrics through a --monitoring_config_file flag.

The system has been demonstrated at very large scale: in a 2021 case study, ad exchange OpenX reported serving 2.5 million prediction requests per second through TensorFlow Serving on Google Kubernetes Engine, each under 15 milliseconds.^[11]

Edge and on device serving is generally not done with TF Serving itself. The recommended path for mobile and embedded targets is TensorFlow Lite, which compiles SavedModels into a smaller flatbuffer format suitable for ARM CPUs, mobile GPUs, and microcontrollers.

How does TensorFlow Serving differ from TorchServe and Triton?

TF Serving was the first widely adopted dedicated model serving system, but the landscape has grown considerably since 2020, especially around large language models.

Scroll sideways for more →

In practice, large Docker shops with a TensorFlow heavy stack still reach for TF Serving because it is the lowest friction path for a SavedModel. PyTorch shops generally prefer TorchServe or Triton, though the PyTorch team stated in 2024 that TorchServe is "no longer actively maintained."^[12] Teams that want a single server for many frameworks, particularly on NVIDIA GPUs, tend to standardize on Triton, which NVIDIA folded into its Dynamo platform in early 2025.^[9] LLM workloads have largely migrated to vLLM, TGI, TensorRT-LLM, or SGLang, none of which TF Serving was designed for.^[10]

What is TensorFlow Serving used for?

The sweet spot for TF Serving is production scoring of TensorFlow models that look like classic supervised graphs. Common deployments include:

• Computer vision inference: image classification, object detection, and segmentation models exported from Keras or tf.keras.applications.
• Tabular and recommendation models: wide and deep networks, ranking models, and embedding lookup services that combine a graph with large embedding tables.
• NLP encoders: BERT style classification and sentence embedding models where the input is a fixed length token sequence.
• A/B testing of model versions, using labels like prod and canary so traffic can be split at the routing layer without changing client code.
• Hot swapping models without restarting the server, useful for retrain pipelines that publish a new SavedModel directory every few hours.

Google has used variants of the same internal serving stack across products including Search, Photos, Translate, and Gmail Smart Reply, although the exact internal version is not the same binary as the open source release.^[2]

Limitations

TF Serving is a mature, focused tool, and its limitations mostly come from where its focus is not.

The server is built around the TensorFlow runtime. Serving PyTorch or pure ONNX models requires writing a custom Loader and is rarely the path of least resistance. Triton or ONNX Runtime are usually a better fit for that case.

It is not optimized for autoregressive LLM inference. There is no PagedAttention, no continuous batching across decode steps, no KV cache management, and no native token streaming endpoint.^[10] LLM workloads have moved to vLLM, TGI, TensorRT-LLM, and SGLang for those reasons.^[10]

The REST endpoint is JSON over HTTP/1.1, which is convenient but adds serialization overhead compared to gRPC.^[6] For very large tensors, the JSON path can become a bottleneck before the model itself does.

For a quick prototype with one model and modest QPS, a Flask or FastAPI wrapper around tf.saved_model.load is often enough and avoids the operational cost of a separate server. TF Serving starts to pay off when there are multiple versions, multiple models, batching requirements, or zero downtime deployment needs.

Is TensorFlow Serving still used in 2026?

Yes. As of 2026, TF Serving remains widely deployed for non LLM TensorFlow workloads, particularly in computer vision pipelines and recommendation systems. It is part of the TFX production ML platform, the default predictor for TensorFlow models in KServe, and one of the underlying serving runtimes used by Google Cloud Vertex AI Prediction.^[8] The release cadence has slowed compared to the 2017 to 2020 period but the project is still maintained.^[4]

For general purpose multi framework serving on GPUs, NVIDIA Triton has taken much of the share that TF Serving once held by default, and for LLMs the conversation is dominated by vLLM and TGI.^[9][10] Within the TensorFlow ecosystem itself, Serving is still the canonical way to take a SavedModel into production without rebuilding the deployment story from scratch.

Explain like I'm 5

Imagine you trained a robot to recognize cats in photos. TF Serving is like a coat check counter for that robot. You leave the robot at the counter in a numbered slot, and any friend who has a photo can walk up, hand it through the window, and get back the answer "cat" or "not cat." If you train a smarter robot, you put it in a higher numbered slot, and the counter quietly starts handing photos to the new one without ever closing. The old robot stays in its slot for a little while, just in case the new one turns out to be worse, and then it goes home.

References

1. Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., and Soyke, J. (2017). *TensorFlow-Serving: Flexible, High-Performance ML Serving*. NIPS 2017 Workshop on ML Systems. arXiv:1712.06139. ↩
2. Google Open Source Blog (2016, February 16). *Running your models in production with TensorFlow Serving*. ↩
3. Google Developers Blog (2017, August 7). *TensorFlow Serving 1.0*. ↩
4. TensorFlow Serving GitHub repository. github.com/tensorflow/serving. ↩
5. TensorFlow Serving Architecture documentation. Architecture overview. ↩
6. TensorFlow team. *RESTful API*. TFX serving documentation. ↩
7. TensorFlow team. *TensorFlow Serving with Docker*. TFX serving documentation. ↩
8. KServe project. *TensorFlow predictor*. ↩
9. NVIDIA. *Triton Inference Server* project documentation. ↩
10. vLLM project. *PagedAttention and continuous batching* documentation. ↩
11. The TensorFlow Blog (2021, February). *How OpenX Trains and Serves for a Million Queries per Second in under 15 Milliseconds*. ↩
12. pytorch/serve GitHub repository. *TorchServe limited maintenance notice*. ↩