Static inference

See also: Inference, Offline inference, Dynamic inference

Static inference is a machine learning serving pattern in which a model generates a batch of predictions offline, ahead of time, and caches them so that applications read the precomputed answer from storage instead of running the model on each request. Google's Machine Learning Glossary defines it as the process where "a model generating a batch of predictions and then caching (saving) those predictions," and notes that static inference is a synonym for offline inference and is also called batch inference ^[1][9]. It is the direct opposite of dynamic inference (also called online inference or real-time inference), where the model only runs on demand when a request arrives ^[1]. Its two headline advantages are that you "don't need to worry much about cost of inference" and that you "can do post-verification of predictions before pushing" them; its two headline costs are that the system can serve only cached predictions for a known set of inputs and that update latency is "likely measured in hours or days" ^[1].

What is static inference?

Static inference is a serving pattern in which a machine learning model computes predictions ahead of time, writes them to a storage layer, and then serves each user request by looking up the cached prediction instead of running the model live. The term is used as a synonym for offline inference and batch inference, and it is the direct opposite of dynamic inference (also called online inference), where the model only runs when a request arrives ^[1].

Google's Machine Learning Crash Course defines the pattern in the production systems chapter on "Static versus dynamic inference": "Static inference (also called offline inference or batch inference) means the model makes predictions on a bunch of common unlabeled examples and then caches those predictions somewhere" ^[1]. That single sentence captures the whole idea. The model still runs, but it runs on a schedule rather than on demand, and the application reads predictions from a key-value store, a database table, or a file.

This article is about that serving pattern. It is sometimes confused with two unrelated ideas: "static computation graph" (the TensorFlow 1 versus PyTorch debate over define-and-run versus define-by-run graphs) and pre-trained models that are no longer being updated. Neither of those is what static inference means in production machine learning systems ^[8]. The disambiguation section at the end of this article covers the difference.

What are the synonyms for static inference (offline vs batch inference)?

Three terms describe the same pattern, with small shifts in emphasis. Google's glossary lists static inference as a direct "synonym for offline," and defines offline inference as "the process of a model generating a batch of predictions and then caching (saving) those predictions" so that "apps can then access the inferred prediction from the cache rather than rerunning the model" ^[9][1].

Scroll sideways for more →

When people say "static," they usually mean the prediction is fixed at the moment it was computed and will not change until the next batch job runs. "Offline" emphasizes that the work happens outside the request path. "Batch" emphasizes that many predictions are computed together in a single job. In practice the three words are interchangeable, and you will see them mixed within the same engineering blog or vendor doc ^[2][3].

The contrast pair is dynamic inference, online inference, or real-time inference. Google's glossary defines online inference as "generating predictions on demand" and explicitly says to "contrast with offline inference" ^[9]. In a dynamic system, a request comes in, the model runs, and the prediction is returned in the same network round trip. In a static system, the request comes in and a lookup is performed against a precomputed table.

How does static inference work?

A typical static inference pipeline has four pieces: an input source, a batch prediction job, a storage layer, and a serving layer.

The input source is usually a data warehouse, a feature store, or files on object storage. It contains the entities that need predictions: every user, every product, every search query that appeared more than ten times last week. The set is finite and known in advance, which is what makes static inference possible ^[1].

The batch job loads the input, runs the model over each row, and writes the output. Jobs are scheduled with an orchestrator like Airflow, Dagster, or Vertex AI Pipelines. The job itself runs on Spark, Apache Beam, Ray, or a model-server batch endpoint such as Vertex AI Batch Prediction or SageMaker Batch Transform ^[2][4]. Because the unit cost matters more than wall-clock latency, engineers tune for throughput and often use spot or preemptible instances.

The storage layer holds the predictions. Common choices are a key-value store (Redis, DynamoDB, Bigtable), a relational table indexed by the lookup key, or a document store. The schema is simple: a key, a prediction value, and a timestamp. Some teams write predictions back to a feature store so they can be joined with other features later.

The serving layer takes a request, extracts the lookup key (a user ID, a product ID, a session ID), and reads the prediction from storage. There is no model in the request path, so latency is whatever the storage layer can deliver, often single-digit milliseconds.

[input data] -> [batch job: load model, predict] -> [key-value store] -> [serving layer]
      ^                  scheduled                       fast lookup        request handler
      |
   feature store / warehouse

What are the advantages and disadvantages of static inference?

Static and dynamic inference make opposite choices on almost every axis. The Google Crash Course lists two advantages and two disadvantages for each side, and most other MLOps references say roughly the same thing ^[1][3]. For static inference, the listed advantages are: you "don't need to worry much about cost of inference," and you "can do post-verification of predictions before pushing" them. The listed disadvantages are: you "can only serve cached predictions, so the system might not be able to serve predictions for uncommon input examples," and "update latency is likely measured in hours or days" ^[1]. The table below maps those tradeoffs against dynamic inference.

Scroll sideways for more →

The single biggest advantage of static inference is the freedom it buys you on the model side. Because the model runs offline, the latency budget is measured in minutes per million examples instead of milliseconds per request. That lets teams use larger models, ensembles, or multi-pass pipelines that would never be acceptable in a real-time path. It also lets the team inspect predictions before they go live, which Google calls "post-verification" and which is useful for safety review, fairness audits, and basic sanity checks ^[1].

The single biggest disadvantage is the coverage problem. A static system can only serve predictions for keys that were in the batch. If a new user signs up after the nightly job ran, there is no row for them in the lookup table. The same problem hits any system with a long tail of rare inputs. Google notes that dynamic inference, by contrast, "can infer a prediction on any new item as it comes in, which is great for long tail (less common) predictions" ^[1]. Free-form text queries, for example, are a poor fit for static inference because the input space is effectively unbounded.

Freshness is the other recurring complaint. Predictions in the table reflect the world at the moment the batch job started, not the moment the request arrives, which is why Google warns that update latency "is likely measured in hours or days" ^[1]. For slow-moving signals like long-term user preferences this gap does not matter much. For fast-moving signals like fraud risk on a live transaction it matters a great deal.

When should you use static inference?

Static inference fits well when three conditions hold at the same time. First, the set of entities you need predictions for is finite and known. Second, the predictions do not need to reflect events from the last few minutes. Third, the cost or complexity of running the model live would be prohibitive.

The table below lists common production use cases that meet those conditions.

Scroll sideways for more →

Dynamic inference is the right choice when the input is unbounded, the prediction must reflect the current request context, or the cost of being wrong about a stale prediction is high. Search ranking on novel queries, fraud detection on live card swipes, ad bidding, and chatbot replies all live on the dynamic side ^[1].

What is the difference between static and dynamic inference?

The difference is when the model runs and what gets served at request time. In static inference the model runs offline on a schedule and the request is answered from a cache; in dynamic inference the model runs on demand and computes a fresh prediction for each request. Google's glossary frames the two as a contrast pair: offline inference "generates a batch of predictions and then caches" them, while online inference "generates predictions on demand" ^[9][1]. The practical consequences cascade from there: static inference trades freshness and coverage for low serving cost, low request-time latency, and the chance to verify predictions before they ship, while dynamic inference trades cost and model-complexity headroom for always-current predictions and unlimited input coverage ^[1].

What are hybrid static and dynamic inference patterns?

Real systems rarely sit at one extreme. A common compromise is the cache-with-fallback pattern: serve a cached prediction when one exists, and run the model live when the lookup misses. This handles the cold-start problem for new users while keeping average latency low. Netflix, Uber, and most large recommender systems use some form of this pattern ^[3].

Another hybrid is the lambda architecture, borrowed from streaming data systems. A batch layer produces canonical predictions on a daily or hourly schedule, while a streaming layer updates predictions for recent events. The serving layer merges the two, usually by preferring the streaming value when it exists. This keeps the freshness of online inference for the long tail of recent activity while letting the bulk of traffic hit the cheap cached values.

A third pattern, sometimes called "precompute the hard part," splits the model itself. The expensive piece, often an embedding lookup over a large corpus, runs offline and writes vectors to a store. The cheap piece, often a small ranker or classifier on top of those vectors, runs online. Two-tower retrieval models in search and recommendations are the canonical example.

What tools and platforms run static inference?

Static inference is not tied to any specific framework. The table below lists the most common tools as of 2026, grouped by what they handle.

Scroll sideways for more →

The last three rows show a relatively new development: provider-side batch APIs for LLM workloads. OpenAI's Batch API and Anthropic's Message Batches API let you submit large numbers of prompts asynchronously and get results back within twenty-four hours, typically at half the per-token price of the synchronous endpoints ^[6]. They are static inference for generative models. Use cases include bulk classification, document tagging, evaluation runs, content rewrites, and any pipeline where you can wait a day for the answer ^[7].

What are the production considerations for static inference?

A few practical issues come up often enough to be worth listing.

Key design matters. The lookup key has to be available at request time and stable enough that the batch job can compute the same key offline. User IDs and content IDs are easy. Anything derived from session state or live signals is harder, and may force you toward dynamic inference.

The cold-start problem is unavoidable for new entities. Most teams handle it with a default prediction (the global average, the most popular item, a heuristic) for unknown keys, then upgrade to a real prediction the next time the batch job runs. For latency-critical paths, a small online model can fill the gap.

Versioning the prediction table is important. When the model changes, the table needs to be regenerated and the serving layer needs to switch over atomically. The standard pattern is to write new predictions to a fresh table, validate, and flip a pointer. Rolling back is just flipping the pointer back.

Monitoring static systems is easier than monitoring online systems, but it has its own quirks. Job-success rate and time-since-last-refresh are the two metrics teams care about most. A successful job that is forty-eight hours stale is often worse than a failed job detected within an hour, so many teams add a freshness SLO to the prediction table itself.

Storage cost can become a real factor. A retailer with one hundred million products and ten million users producing a personalized score for every pair would need a trillion rows. Teams avoid the dense cross-product by precomputing only top-K results per user, by using approximate nearest-neighbor indexes over embeddings, or by accepting a lower hit rate with a smaller candidate set.

What is the difference between static inference and a static computation graph?

The phrase "static" gets attached to several distinct ideas in machine learning, and they are easy to mix up.

Static computation graph is a property of a deep-learning framework, not a serving pattern. TensorFlow 1.x required you to define the full graph before running it, which made the graph "static." PyTorch builds the graph on the fly during each forward pass, which makes it "dynamic." TensorFlow 2.x added eager execution to behave more like PyTorch. Both static and dynamic computation graphs can be used for either static or dynamic inference. The two distinctions are independent ^[8].

Static in the sense of "frozen" or "pre-trained without further updates" is closer to the everyday English meaning, but it is not what static inference means either. A model that is no longer being trained can still be served either statically (predictions cached) or dynamically (predictions on demand).

Static features in feature engineering are features whose value does not change over time, like a user's birth year. They can be computed offline and stored once, which makes them a natural fit for static inference, but the two terms are not synonyms.

Explain like I'm 5

Imagine you are running a small restaurant for kids. There are two ways to handle dinner.

The first way: every kid orders, you cook their plate, and you bring it out. Each plate is fresh and exactly what they asked for, but you have to keep the kitchen running the whole night and people sometimes have to wait. That is dynamic inference.

The second way: in the afternoon, you cook a plate for every kid you know is coming. You write each kid's name on a sticker, put the plates in the warmer, and when a kid sits down you grab their plate off the rack and bring it out. The plates were ready before anyone arrived, so service is fast. That is static inference.

The second way is great if you know who is coming and what they like. It is bad if a new kid wanders in and there is no plate with their name on it.

See also

• Inference
• Offline inference
• Dynamic inference
• Online inference
• Batch processing
• Feature store
• Recommendation system
• Spark
• Airflow
• LLM

References

1. Google for Developers. "Production ML systems: Static versus dynamic inference." Machine Learning Crash Course. https://developers.google.com/machine-learning/crash-course/production-ml-systems/static-vs-dynamic-inference ↩
2. Google Cloud. "What is batch inference?" https://cloud.google.com/discover/what-is-batch-inference ↩
3. Hapke, Hannes, and Catherine Nelson. "Batch Inference vs. Online Inference." ML in Production. https://mlinproduction.com/batch-inference-vs-online-inference/ ↩
4. Amazon Web Services. "Batch transform for inference with Amazon SageMaker AI." https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html ↩
5. Google for Developers. "Machine Learning Glossary." https://developers.google.com/machine-learning/glossary
6. Anthropic. "Introducing the Message Batches API." October 2024. https://www.anthropic.com/news/message-batches-api ↩
7. vLLM project. "Offline Inference with the OpenAI Batch file format." https://docs.vllm.ai/en/latest/examples/offline_inference/openai_batch/ ↩
8. GeeksforGeeks. "Dynamic vs Static Computational Graphs: PyTorch and TensorFlow." (Used for the disambiguation section, not for the main definition.) https://www.geeksforgeeks.org/deep-learning/dynamic-vs-static-computational-graphs-pytorch-and-tensorflow/ ↩
9. Google for Developers. "Machine Learning Glossary: ML Fundamentals." (Entries for static inference, offline inference, online inference, dynamic inference.) https://developers.google.com/machine-learning/glossary/fundamentals ↩