Inference

In machine learning, inference is the execution of a trained model to produce an output from an input. A classifier may return class scores, a regressor may return a number, and a generative model may return a sequence. The defining boundary is that the learned parameters are used rather than updated by that invocation. The input does not have to be new or previously unseen, and an inference call can occur during validation, testing, production service, or an interactive application.^[1][2]

Inference is often described as a forward pass through a model, but a deployed inference system usually does more. It must reproduce the required preprocessing, execute the correct model version, interpret the raw outputs, enforce application rules, and expose enough telemetry to detect failures. Its performance depends on the model, input shape, numerical precision, runtime, hardware, request pattern, and quality requirements. Consequently, a result such as latency or tokens per second is meaningful only when its workload and measurement conditions are stated.^[8][9]

Scope and terminology

The word inference has several established meanings. They are related by the idea of deriving an output or conclusion, but they are not interchangeable.

Scroll sideways for more →

Statistical inference concerns the relation between samples and a wider population.^[3] In probabilistic models, inference commonly means evaluating posterior distributions or related expectations, which can itself require a substantial algorithm.^[4] This article focuses on executing trained machine learning models and on the systems that support that execution.

For a deterministic model with parameters theta, the central computation can be written as y = f_theta(x). A stochastic model or decoding procedure instead samples or otherwise selects an output from a distribution conditioned on x. Fixed model parameters therefore do not guarantee identical outputs. Random sampling, nondeterministic device kernels, concurrency, numerical precision, and preprocessing differences can all affect reproducibility.^[6][7]

Relationship to training, evaluation, and serving

Inference is one stage of a larger model lifecycle.

Scroll sideways for more →

Evaluation normally performs inference and then compares the outputs with labels or other reference judgments. Production inference normally lacks an immediate ground-truth answer. This difference matters because an online service can report stable latency and error rates while its task quality is deteriorating.

Some frameworks separate gradient recording from model behavior. In PyTorch, for example, inference mode disables autograd-related work, while evaluation mode changes the behavior of modules such as dropout and batch normalization. The two controls are orthogonal: disabling gradients does not by itself put a model in evaluation mode.^[5] Equivalent details vary by framework, so a deployed artifact should be tested through the same execution path used in production.

Execution pipeline

A complete inference path commonly contains the following stages:

1. Request and input validation. The system checks data type, shape, range, encoding, required fields, and request limits. Invalid inputs should fail explicitly rather than being silently coerced.
2. Preprocessing. Raw text, images, audio, tables, or sensor values are converted to the representation expected by the model. Examples include tokenization, resizing, normalization, feature construction, and padding.
3. Model execution. The runtime loads or reuses the model parameters, binds input and output buffers, and schedules operators on one or more devices. For a neural network, this is the forward computation.
4. Postprocessing and decision logic. Raw tensors may be decoded, calibrated, thresholded, ranked, filtered, or mapped to application objects. Generative systems may apply stopping rules and output constraints.
5. Response and observability. The service returns the result and records appropriately limited telemetry about timing, errors, model version, resource use, and data health.

The model alone is therefore not a reproducible inference system. A tokenizer version, image-resize convention, category mapping, threshold, or missing-value rule can change the user-visible result without changing the parameter file. A release should version these dependencies with the model and validate the assembled pipeline.

Model artifacts and compilation

An inference artifact may be a framework checkpoint, an exported computation graph, a portable interchange model, or a hardware-specific compiled engine. Export and compilation can perform constant folding, operator fusion, layout changes, precision conversion, and kernel selection. These transformations can reduce overhead, but they also create a new object that must be checked against the accepted model on representative and edge-case inputs.

NVIDIA TensorRT illustrates the distinction between a build phase and a runtime phase: the builder optimizes a network and chooses kernels for a target GPU, while the runtime loads and executes the resulting engine.^[10] ONNX provides a portable model format, and ONNX Runtime assigns supported parts of a graph to execution providers for particular hardware or libraries.^[11] Portability of a format does not imply that every operator, shape, numerical result, or performance characteristic is identical across providers.

Stateful and stateless execution

Many conventional prediction calls are stateless at the model boundary: the output depends on the supplied input and fixed parameters. The surrounding service can still be stateful because it maintains caches, sessions, feature histories, rate limits, or routing decisions. Recurrent models, streaming models, and autoregressive generators may explicitly carry state from one invocation or iteration to the next.

State changes the unit of correctness. A test of one isolated request may not reveal errors caused by state reuse, sequence boundaries, cancellation, eviction, or concurrent access. Systems that reuse memory or cache model states should test isolation between users and requests as well as numerical correctness.

Workload and deployment patterns

The same model can be executed under different service patterns:

• Online inference handles individual requests or small groups under a latency objective. Queueing and tail latency matter because users experience the complete request path.
• Offline or batch inference processes a finite dataset, often emphasizing total completion time and throughput. Larger batches may be possible because results need not be returned immediately.
• Streaming inference consumes an ongoing sequence and may preserve temporal state. End-to-end delay includes windowing, buffering, and any ordered processing.
• On-device inference runs near the data source, such as on a phone, vehicle, or embedded system. It can reduce network dependence, but it is constrained by local memory, supported operators, power, and thermal limits.
• Centralized service inference runs in a data center and can pool accelerators, autoscale, and serve multiple model versions. It introduces network, queueing, isolation, and capacity-planning concerns.

These patterns are not mutually exclusive. An application may perform a small edge AI model locally, send selected inputs to a larger service, and later run a batch job for audit or analytics.

Serving systems add functions that a bare runtime generally does not provide: model discovery and loading, version policies, request protocols, batching, health checks, traffic routing, metrics, and resource management. TensorFlow Serving, for example, is a production model-serving system with versioned model loading and network APIs.^[12] The engineering discipline around these functions is commonly treated as part of MLOps.

Performance measurement

Inference performance is multidimensional. Reporting only an average latency or a peak arithmetic rate can hide the constraint that controls an actual service.

Scroll sideways for more →

Throughput and latency interact. A larger batch can improve device utilization and throughput but also makes a request wait for a batch to form and can increase the duration of each scheduled unit. Dynamic batching groups compatible requests that arrive close together; Triton Inference Server documents queue-delay and batch-size controls for this tradeoff.^[13] The best setting depends on the arrival pattern and service objective rather than on model size alone.

Average latency is insufficient for an interactive service because queueing and long-running inputs can create a heavy tail. Measurements should include warm-up, steady-state offered load, request-length distribution, concurrency, failures, and high-percentile latency. Cold-start behavior, model loading, compilation, or scale-up delay should be measured separately when it can affect users.

MLPerf Inference was designed to compare systems under defined tasks, quality targets, rules, and deployment scenarios. Its original benchmark described single-stream, multistream, server, and offline scenarios, each with different metrics.^[8] The maintained benchmark suite continues to define versioned workloads and rules.^[9] Published benchmark results are not transferable to a different model, precision, input distribution, or service configuration without a new measurement.

Inference for autoregressive language models

Autoregressive large language models repeatedly predict a distribution for the next token, choose a token, append it to the sequence, and continue until a stopping condition is met. Decoder-only transformers prevent each position from attending to later positions, preserving the autoregressive dependency.^[14]

The token-selection rule is part of inference behavior. Greedy decoding selects the highest-probability token at each step. Sampling draws from a transformed probability distribution. Beam search retains multiple candidate sequences and prunes them as generation proceeds.^[7] Temperature, top-k or nucleus filtering, beam width, repetition controls, stop strings, and random seeds can change the output while the model parameters remain fixed.

Prefill and decode

Serving literature often divides one request into two phases:

• Prefill, also called prompt processing, processes the supplied input tokens and produces the first-token state.
• Decode, also called token generation, executes successive iterations to produce later tokens.

The distinction is operationally useful because the phases expose different parallelism. On a given model and system, sufficiently large prefill work is often compute intensive, while single-token decode at modest batch size is often limited by moving model parameters and cached state through memory. These are tendencies, not universal laws. Short prompts, large batches, mixture-of-experts routing, cache placement, quantization, and hardware balance can change the bottleneck. Splitwise measured the contrast and evaluated separating the phases onto different machine pools for its workloads.^[19]

Common latency measures include time to first token (TTFT) and time per output token (TPOT), sometimes supplemented by inter-token latency and complete request latency. TTFT is not simply prefill kernel time. In an end-to-end service it can include admission, queueing, preprocessing, prompt transfer, prefill, the first decode step, and response transfer. Similarly, TPOT depends on scheduling and co-runners as well as the model computation.

Key-value cache

During autoregressive decoding, an attention layer can reuse keys and values computed for earlier tokens instead of recomputing the entire prefix at every step. This stored state is the key-value cache, or KV cache. It grows with the number of cached tokens and concurrent sequences, and it can become a major memory constraint.^[16]

For a standard cached key and value at every decoder layer, an approximate per-token storage calculation is:

2 * number_of_layers * number_of_KV_heads * head_dimension * bytes_per_element

The factor 2 represents one key and one value. Total cache storage then scales with cached tokens and active sequences. This expression is not a universal capacity guarantee. Grouped-query or multi-query attention changes the number of KV heads; cache quantization changes bytes per element; alignment, allocator blocks, metadata, replication, paging, offload, and sliding-window policies add or remove storage.

Naively reserving one contiguous maximum-length cache for every request can waste memory when actual sequence lengths vary. PagedAttention divides a request's cache into blocks that need not be physically contiguous, allowing allocation as generation advances and sharing where sequences have common state. The associated vLLM paper evaluated this approach under specific models and workloads; its reported speedups are results of those experiments, not a general multiplier for every server.^[16]

Scheduling and phase separation

Traditional request-level batching can hold a short generation behind a longer one. Orca introduced iteration-level scheduling, allowing the scheduler to reconsider the active batch between token iterations and return completed requests earlier.^[15] Modern systems often call related behavior continuous or in-flight batching. It can improve utilization and responsiveness, but scheduling policy still determines fairness, preemption cost, memory pressure, and tail latency.

Prefill and decode can also be separated across devices or workers. This permits phase-specific provisioning and can reduce interference, but it requires transferring request state, including the KV cache, between phases. The transfer consumes bandwidth and adds latency, so disaggregation is beneficial only under configurations where its scheduling and utilization gains exceed those costs.^[19]

Optimization techniques

Inference optimization should begin with a reproducible baseline: fixed model artifact, preprocessing, dataset or request trace, quality metric, hardware, software versions, warm-up policy, and load. Each change should then be measured against both system objectives and output quality. Techniques at different layers can interact, so their individual gains do not generally multiply.

Model transformations

Scroll sideways for more →

Jacob and colleagues demonstrated integer-only inference with quantized weights and activations on the models and mobile processors in their study.^[20] That result supports quantization as a practical technique, but it does not justify a fixed speedup for arbitrary hardware. Likewise, Han and colleagues pruned and retrained particular convolutional networks while preserving their measured accuracy.^[21] Real latency gains from pruning depend on whether the runtime can exploit the resulting structure. Distillation transfers information from a teacher's outputs into a student during training; it is not a runtime compression switch applied without further learning.^[22]

Graph, kernel, and memory optimization

Compilers and runtimes can fold constants, eliminate redundant operations, fuse compatible operators, choose layouts, reuse buffers, and select kernels for known shapes. The resulting benefit depends on graph structure and target hardware. Dynamic shapes can reduce the compiler's opportunities or require multiple specialized plans.

Data movement can cost more time and energy than arithmetic, especially when operands repeatedly cross memory levels. Surveys of efficient deep learning processing therefore analyze dataflow and memory hierarchy alongside operation count.^[23] FlashAttention is an exact attention algorithm designed around this principle: it tiles the computation to reduce reads and writes between GPU high-bandwidth memory and on-chip memory.^[17] Its paper's end-to-end results apply to the tested models, sequence lengths, and hardware, while the general lesson is that fewer floating-point operations is not the only route to lower latency.

Buffer reuse and preallocated memory pools can reduce allocation overhead, but they must preserve request isolation and respect peak concurrent state. Copy avoidance can reduce latency when the same memory representation is accepted across components. It can also complicate ownership and lifetime rules, so correctness under cancellation and concurrency must be tested.

Batching, caching, and routing

Batching amortizes fixed costs and can reuse loaded parameters across requests. Static batching is natural for offline work. Dynamic or continuous batching adapts to arrivals in an online service. Larger batches consume more activation and cache memory, and they can violate latency objectives before a device reaches its theoretical peak throughput.

Result caching can bypass model execution for an identical request when the model, preprocessing, postprocessing, permissions, and relevant external state are all part of the cache key. It is unsafe when outputs are intentionally random, personalized, time-dependent, access-controlled, or affected by hidden state. Prefix caching for language models reuses computed prompt state rather than the final answer; its value depends on prefix repetition and the cost of retaining and isolating cached state.

A multi-model system may route a request to a smaller or specialized model before escalating to a larger one. This can reduce average work, but the router becomes part of the quality and safety behavior. Its errors, thresholds, fallback rate, and distribution shifts need separate evaluation.

Speculative and parallel decoding

Speculative decoding uses a faster approximation model to propose several tokens and the target model to verify them in parallel. The algorithm of Leviathan and colleagues preserves the target model's output distribution while reducing serial target-model steps when proposals are accepted.^[18] This guarantee belongs to the specified acceptance and correction procedure. An implementation that simply accepts draft tokens without that procedure does not inherit it. Speed depends on draft cost, acceptance rate, verification efficiency, batch conditions, and target-model bottlenecks.

Tensor parallelism, pipeline parallelism, and expert parallelism can make a model fit or distribute computation across devices. They also add collective communication, synchronization, and failure domains. More devices do not automatically reduce latency, particularly for small batches or slow interconnects. Replicating a complete model instead increases independent serving capacity but does not make one request's model computation faster.

Hardware and system design

Inference can run on general-purpose CPUs, GPU computing platforms, or domain-specific AI accelerators. No device class is universally best. Selection begins with workload requirements:

• Capacity: parameters, activations, workspaces, and request state must fit in local or distributed memory.
• Bandwidth: sustained movement of weights, activations, and caches can limit execution even when arithmetic units are not full.
• Compute support: operator mix and numerical precision must map to efficient, validated kernels.
• Latency behavior: launch overhead, queueing, batch requirements, and synchronization affect small or interactive workloads.
• Interconnect: multi-device execution depends on device-to-device and host-to-device communication.
• Power and thermal envelope: sustained performance can differ from short benchmark bursts, especially on edge devices.
• Software compatibility: exporter, compiler, runtime, drivers, observability, and supported operators determine whether hardware can be used correctly.

Jouppi and colleagues described the first-generation Tensor Processing Unit as a domain-specific inference ASIC centered on a matrix multiply unit and on-chip buffers.^[24] The work is historically important evidence for inference-specific hardware design, but its comparisons involve 2015-era systems and should not be used as current product rankings.

The fastest device execution can still yield a slow service if tokenization, data copies, queues, networking, or postprocessing dominate. Profiling should therefore start end to end and then attribute time to layers of the stack. A roofline-style analysis can help distinguish compute throughput from memory-bandwidth limits, but measured operator traces and memory use are needed to confirm the bottleneck.

Software stack

Inference software is layered, and one product may span more than one layer.

Scroll sideways for more →

TensorRT's engine build is target-specific, ONNX Runtime can partition a graph across execution providers, and TensorFlow Serving focuses on managed network serving.^[10][11][12] Triton's batching features operate above supported model backends.^[13] These components are not interchangeable merely because each can appear in an inference deployment.

A production stack should record the model checksum, preprocessing and postprocessing versions, runtime and driver versions, precision mode, build options, and hardware target. It should also define rollback behavior when a model or compiled artifact fails validation after deployment.

Validation and monitoring

Pre-deployment validation should cover more than aggregate task accuracy:

• Compare the deployed artifact with an accepted reference implementation on representative, boundary, malformed, and high-impact examples.
• Test preprocessing and postprocessing independently, including locale, encoding, missing values, shape changes, and category mappings.
• Measure numerical differences introduced by export, precision conversion, fusion, or hardware-specific kernels.
• Run load tests with the expected arrival pattern and input-size distribution, then inspect tail latency, queue growth, memory peaks, timeouts, and recovery.
• Exercise cancellation, retries, model reloads, process restarts, device errors, and fallback paths.
• Verify access control, data retention, logging redaction, and isolation of per-request state.

For probabilistic classifiers, confidence and accuracy are different properties. A model is calibrated when, over an appropriate set of predictions, stated confidence corresponds to observed correctness frequency. Modern neural classifiers can be miscalibrated, and calibration must be measured on relevant data rather than inferred from accuracy alone.^[25] Calibration can also change after quantization, model updates, or a shift in inputs.

Production inputs should be checked against an expected schema and distribution. Google's data-validation work describes detecting anomalies and training-serving skew in machine-learning pipelines.^[26] Such checks can reveal missing fields or distribution changes, but a distribution alert does not by itself prove that task performance has degraded. Conversely, stable marginal feature statistics do not prove that the relationship between inputs and outcomes is unchanged.

Post-deployment monitoring should combine:

• Operational signals: request rate, errors, queue time, end-to-end and stage latency, saturation, memory, restarts, and fallback rate.
• Input and output signals: schema violations, missingness, length, drift indicators, prediction distribution, confidence, refusal or abstention rate, and policy-rule triggers.
• Quality signals: delayed labels, sampled human review, controlled evaluations, user-impact measures, and subgroup analysis where appropriate.
• Version signals: model, data, feature, runtime, configuration, and hardware identity for every comparable measurement.

NIST AI 800-4 groups post-deployment monitoring challenges around functionality, operational behavior, security, human factors, compliance, and broader impacts. It also emphasizes that methods and terminology remain an active area of work rather than a solved checklist.^[27] Earlier systems research similarly identified data dependencies, feedback loops, changing external conditions, and monitoring gaps as sources of technical debt in machine-learning systems.^[28]

Monitoring does not repair a model by itself. Alerts need an owner, investigation context, and a defined response such as rollback, traffic reduction, fallback, threshold review, or retraining. Automatic intervention should be bounded and tested because it changes the deployed system's behavior.

Common interpretation errors

• Equating inference with production serving. Inference is the model execution; serving includes the surrounding request, version, scheduling, and observability system.
• Calling every input unseen. A model can infer on training, validation, repeated, cached, synthetic, or genuinely new inputs.
• Assuming inference is deterministic. Sampling and some kernels can produce different outputs even with fixed parameters.
• Treating parameter count as a latency predictor. Operator structure, shapes, precision, memory traffic, batch size, runtime, and hardware also matter.
• Comparing throughput without quality constraints. A faster result produced by a different model, precision, decoding rule, or accuracy target is not a like-for-like comparison.
• Reporting only kernel time. User-visible latency may be dominated by queues, transfers, preprocessing, or postprocessing.
• Assuming an optimization has a universal multiplier. Published speedups describe particular baselines and experimental conditions.
• Skipping post-deployment quality checks. A service can remain operational while data or task performance changes.

See also

• Computer Vision
• Natural Language Processing
• TensorRT

References

1. ^Crankshaw, Daniel, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. "Clipper: A Low-Latency Online Prediction Serving System." *14th USENIX Symposium on Networked Systems Design and Implementation* (2017). usenix.org/...crankshaw
2. ^Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. "Deep Feedforward Networks." In *Deep Learning* (MIT Press, 2016). deeplearningbook.org/...mlp
3. ^NIST/SEMATECH. "Populations and Sampling." *e-Handbook of Statistical Methods*. Accessed July 28, 2026. itl.nist.gov/...ppc134
4. ^Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. "Approximate Inference." In *Deep Learning* (MIT Press, 2016). deeplearningbook.org/...inference
5. ^PyTorch documentation. "Autograd mechanics." Accessed July 28, 2026. docs.pytorch.org/...autograd
6. ^PyTorch documentation. "Reproducibility." Accessed July 28, 2026. docs.pytorch.org/...randomness
7. ^Hugging Face documentation. "Generation strategies." Accessed July 28, 2026. huggingface.co/...generation_strategies
8. ^Reddi, Vijay Janapa, et al. "MLPerf Inference Benchmark." arXiv:1911.02549 (2020 revision). arxiv.org/...1911.02549
9. ^MLCommons. "MLPerf Inference Benchmark Suite." Accessed July 28, 2026. docs.mlcommons.org/...index_gh
10. ^NVIDIA. "How TensorRT Works." Accessed July 28, 2026. docs.nvidia.com/...how-trt-works
11. ^ONNX Runtime documentation. "Execution Providers." Accessed July 28, 2026. onnxruntime.ai/...execution-providers
12. ^TensorFlow documentation. "Serving Models." Accessed July 28, 2026. tensorflow.org/...serving
13. ^NVIDIA Triton Inference Server documentation. "Batchers." Accessed July 28, 2026. docs.nvidia.com/...batcher
14. ^Vaswani, Ashish, et al. "Attention Is All You Need." *Advances in Neural Information Processing Systems* 30 (2017). proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
15. ^Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." *16th USENIX Symposium on Operating Systems Design and Implementation* (2022). usenix.org/...yu
16. ^Kwon, Woosuk, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." *29th Symposium on Operating Systems Principles* (2023). doi.org/...3600006.3613165
17. ^Dao, Tri, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." *Advances in Neural Information Processing Systems* 35 (2022). arxiv.org/...2205.14135
18. ^Leviathan, Yaniv, Matan Kalman, and Yossi Matias. "Fast Inference from Transformers via Speculative Decoding." *Proceedings of the 40th International Conference on Machine Learning*, PMLR 202 (2023). proceedings.mlr.press/...leviathan23a
19. ^Patel, Pratyush, et al. "Splitwise: Efficient Generative LLM Inference Using Phase Splitting." arXiv:2311.18677 (2023; revised 2024). arxiv.org/...2311.18677
20. ^Jacob, Benoit, et al. "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference." *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition* (2018). openaccess.thecvf.com/..._Training_CVPR_2018_paper
21. ^Han, Song, Jeff Pool, John Tran, and William J. Dally. "Learning both Weights and Connections for Efficient Neural Networks." *Advances in Neural Information Processing Systems* 28 (2015). arxiv.org/...1506.02626
22. ^Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the Knowledge in a Neural Network." arXiv:1503.02531 (2015). arxiv.org/...1503.02531
23. ^Sze, Vivienne, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. "Efficient Processing of Deep Neural Networks: A Tutorial and Survey." *Proceedings of the IEEE* 105(12), 2295-2329 (2017). doi.org/...JPROC.2017.2761740
24. ^Jouppi, Norman P., et al. "In-Datacenter Performance Analysis of a Tensor Processing Unit." *Proceedings of the 44th Annual International Symposium on Computer Architecture* (2017). doi.org/...3079856.3080246
25. ^Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On Calibration of Modern Neural Networks." *Proceedings of the 34th International Conference on Machine Learning*, PMLR 70 (2017). proceedings.mlr.press/...guo17a
26. ^Breck, Eric, Martin Zinkevich, Neoklis Polyzotis, Steven Whang, and Sudip Roy. "Data Validation for Machine Learning." *Proceedings of SysML* (2019). research.google/...validation-for-machine-learning
27. ^Rao, Anita, et al. "Challenges to the Monitoring of Deployed AI Systems: Center for AI Standards and Innovation." NIST AI 800-4 (2026). doi.org/...NIST.AI.800-4
28. ^Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." *Advances in Neural Information Processing Systems* 28 (2015). proceedings.neurips.cc/...674f757a2463eba-Abstract