Mixture of Experts (MoE)
A mixture of experts (MoE) is a machine-learning architecture that contains multiple component functions called experts and a gating or routing mechanism that determines how their outputs contribute to a…
Explore AI Inference through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of AI Inference.
Showing 61-120 of 131 articles
A mixture of experts (MoE) is a machine-learning architecture that contains multiple component functions called experts and a gating or routing mechanism that determines how their outputs contribute to a…
Model compression is a family of techniques that reduce the size, memory footprint, and computational cost of machine learning models while preserving as much of their original accuracy as possible.
The model-as-a-service (MaaS) market in China is the commercial market for access to large language models and other generative models through public-cloud APIs, where enterprises and developers pay per token…
Modular (legally Modular Inc, also referred to as Modular AI) is an American AI infrastructure software company that builds tools for running AI models on chips from many vendors without rewriting code for…
Multi-token prediction (often abbreviated MTP) is a language modeling training objective in which the model is trained to predict several future tokens at each context position rather than only the next token.
NOSA (Native and Offloadable Sparse Attention) is a trainable sparse attention mechanism designed so that most of a language model's KV cache can live in CPU memory during decoding without the CPU-to-GPU…
NVIDIA BioNeMo Inference Runtime (BioIR) is a GPU-accelerated Python library for biomolecular structure prediction inference.
NVIDIA DGX Spark is a compact, deskside AI development system, marketed by NVIDIA as a "personal AI supercomputer," that puts a Grace Blackwell superchip on a desktop and delivers up to 1 petaFLOP (1,000 TOPS)…
NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.
NVIDIA Groq 3 LPX is a rack-scale inference accelerator that NVIDIA introduced at GTC 2026, built around 256 Groq Language Processing Units and designed to sit beside Vera Rubin NVL72 racks as a dedicated…
NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…
NVIDIA Picasso is a cloud-based generative AI foundry from NVIDIA for building, training, and deploying visual generative models that produce images, video, and 3D content from text prompts.
NVIDIA Riva is a GPU-accelerated software development kit and family of containerized inference services for speech and translation AI, built by NVIDIA.
NVIDIA Rubin CPX is a class of GPU announced by NVIDIA on September 9, 2025, purpose-built to accelerate the compute-heavy "context" phase of large-model inference.
NVIDIA TensorRT-LLM is an open-source library developed by nvidia for high-performance inference of large language models on NVIDIA GPUs.
NVIDIA Triton Inference Server is open-source model deployment software that lets teams run trained models from any machine learning or deep learning framework on any processor (GPU, CPU, or other accelerator)…
NanoDC (Nano Data Center) is a factory-built data center developed by Liquid Energy for AI inference.
A neural processing unit (NPU) is a processor, or a block inside a larger chip, designed specifically to run neural network inference at high throughput and low power.
Neywa Labs is a startup that builds software for running artificial intelligence models locally on personal devices, with a focus on Macs and other Apple silicon hardware.
NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer…
OctoAI (originally OctoML) was an American artificial intelligence infrastructure company that operated a generative-AI inference platform and, before its pivot
Offline inference (also called batch inference, static inference, or bulk scoring) is the practice of running a trained machine learning model over a known set of inputs ahead of time and storing the resulting…
On-device AI is the practice of running machine learning models on the phone, laptop, watch, or embedded board a person is actually using, instead of sending the input to a remote data center.
Online inference (also called dynamic inference, real-time inference, or on-demand prediction) is the practice of running a trained machine learning model synchronously inside a request path
OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning inference across a heterogeneous set of Intel hardware…
Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight and activation quantization across multiple devices…
OrcaRouter is a hosted API gateway and routing service for large language models and other generative models.
PagedAttention is a KV-cache memory management algorithm for serving large language models that applies the virtual-memory paging technique used by operating systems to the GPU, eliminating memory…
Positron AI is an American semiconductor startup headquartered in Reno, Nevada, that designs and manufactures purpose-built hardware for transformer inference.
In machine learning, post-processing is any operation applied to a model's raw outputs after the prediction step but before the results reach a downstream system or end user.
Prime Inference is a model-serving service from Prime Intellect. Its OpenAI-compatible API gives applications access to large language models through a common endpoint and model catalog.
PrismML (also styled Prism ML; legal name Prism ML, Inc.) is an American artificial intelligence company, based in Pasadena, California
Product quantization (PQ) is a vector-compression technique for approximate nearest-neighbor (ANN) search that splits each high-dimensional vector into M equal sub-vectors and quantizes each sub-vector with…
Project Maya is open-source software for running large mixture-of-experts language models on local hardware.
Prompt caching is a large language model (LLM) inference optimization that stores the computed key-value (KV) state of a repeated prompt prefix so it can be reused across API calls, cutting both cost and…
Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
Qualcomm AI200 is a rack-scale data-center accelerator for artificial intelligence inference, announced by Qualcomm on 27 October 2025 and slated for commercial availability in 2026.
Qualcomm AI250 is a planned data-center artificial intelligence inference accelerator and rack-scale system announced by Qualcomm in late October 2025.
Quantization is the representation of values from a large or continuous set by values from a smaller, usually finite set.
Reinforcement Learning with Verifiable Rewards (RLVR) is a post-training paradigm for large language models in which the reward signal comes from a deterministic
RadixAttention is a KV cache management technique introduced in SGLang that uses a radix tree data structure to automatically share and reuse cached key-value tensors across inference requests.
REBEL-Quad is a chiplet-based AI inference accelerator developed by Rebellions, a South Korean AI-chip company.
SRAM, or static random-access memory, is a semiconductor memory that stores each bit in a latch built from cross-coupled inverters.
Skeleton-of-Thought (SoT) is a prompting technique for large language models that reduces end-to-end generation latency by first eliciting a short outline of the answer (the "skeleton") and then expanding each…
SmoothQuant is a training-free, accuracy-preserving post-training quantization (PTQ) method that enables 8-bit weight and 8-bit activation (W8A8) integer inference for large language models without retraining…
Snapdragon AR1 is a family of Qualcomm system-on-chip platforms built specifically for smart glasses.
SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026.
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…
Splash is an open-source large language model inference engine for Macs with Apple silicon, released by the inference company Inco AI on 18 September 2026 under the Apache-2.0 license.
Static inference is a machine learning serving pattern in which a model generates a batch of predictions offline, ahead of time, and caches them so that applications read the precomputed answer from storage…
StreamingLLM is an inference-time technique that allows pretrained transformer language models, originally trained with a finite attention window
Temperature sampling is the most common decoding control in large language models: a single hyperparameter, written T, that divides the model's output logits before the softmax function so that the formula…
TensorFlow Serving (often shortened to TF Serving) is Google's open source system for serving machine learning models in production: it loads trained models, exposes them over gRPC and REST APIs, and manages…
Tenstorrent Galaxy Blackhole is an AI inference server built by Tenstorrent, the fabless semiconductor company led by chief executive Jim Keller.
Test-Time Training (TTT) is a family of machine learning techniques in which a model updates some of its own parameters at inference time
Text Generation Inference (TGI) is an open-source toolkit developed by Hugging Face for deploying and serving large language models in production.
The Biological Computing Company (TBC; styled The Biological Computing Co. and formerly called Biological Black Box) is an American startup that grows living neurons on multi-electrode arrays, records how the…
Top-k sampling is a decoding strategy for autoregressive language models that restricts each generation step to the k most probable next tokens.
Ultrafast is OpenAI's premium speed tier, a processing option that generates output tokens faster than the company's Standard, Fast, Flex and Batch tiers at a correspondingly higher price.