AI Accelerator
An AI accelerator is hardware designed or configured to execute artificial intelligence and machine learning workloads more efficiently than a general-purpose processor executing the same workload without…
AI Inference
Explore AI Inference through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (41)
Articles that also belong to these categories. Counts cover all of AI Inference.
All articles (97)AI Hardware (30)AI Infrastructure (26)Large Language Models (16)Developer Tools (15)NVIDIA (11)Machine Learning (10)Deep Learning (9)Open Source AI (9)AI Companies (8)Training & Optimization (8)MLOps (6)Model Architecture (4)Reasoning Models (4)Algorithms (3)Chinese AI (3)Reinforcement Learning (3)AI Alignment (2)AI Models (2)Anthropic (2)Artificial Intelligence (2)Data Centers (2)Edge computing (2)Google (2)Information Retrieval (2)Meta AI (2)AI Benchmarks (1)AI Research (1)AI Safety (1)Computer Science (1)Enterprise AI (1)Generative AI (1)Image Generation (1)Microsoft (1)Mixture of Experts (1)Natural Language Processing (1)Prompt Engineering (1)Small Language Models (1)Speech & Audio AI (1)Statistics (1)Transformer Models (1)Video Generation (1)
Showing 1-60 of 97 articles
AMD Instinct MI350P
The AMD Instinct MI350P is a PCIe add-in card built on AMD's CDNA 4 architecture, introduced on May 7, 2026 as the third member of the Instinct MI350 series and the first AMD Instinct product in a conventional…
AWQ (Activation-aware Weight Quantization)
Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…
AWS Graviton
AWS Graviton is a family of Arm-based server processors designed by Amazon Web Services for use in its own cloud computing fleet.
AWS Inferentia
AWS Inferentia is a family of custom application specific integrated circuits (ASICs) designed by Amazon Web Services for machine learning inference in the cloud, built to deliver, in AWS's words, "high…
Adaptive thinking
Adaptive thinking is an inference-time reasoning mode in the Anthropic Messages API in which a claude model decides, on a per-request basis, whether to use extended thinking at all and how much of it to spend
Attention-FFN disaggregation
Attention-FFN disaggregation (AFD), also written attention-feedforward disaggregation or A/F disaggregation, is a large language model serving architecture that runs the attention sublayers and the…
Beam search
Beam search is a heuristic search algorithm for sequence generation that, at each decoding step, keeps only the top-K highest-scoring partial sequences (where K is the beam width) and discards the rest
Constitutional Classifiers
Constitutional Classifiers are a machine learning-based safety technique developed by Anthropic to defend large language models against universal jailbreak attacks.
Context caching
Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…
Continuous Batching
Continuous batching is a scheduling technique for large language model (LLM) inference servers that inserts new requests into a running batch at the granularity of individual model iterations rather than…
DeepInfra
DeepInfra is a serverless AI inference cloud that hosts open-source and open-weight AI models and serves them to developers through a single pay-per-token API.
Disaggregated serving
Disaggregated serving is an LLM inference architecture that physically separates the prefill phase and the decode phase of text generation onto different sets of GPU hardware.
Dynamic inference
Dynamic inference, also called input-adaptive inference, conditional computation, or adaptive computation
EAGLE (speculative decoding)
EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a lossless speculative decoding method that speeds up large language model (LLM) inference by 2x to 6.5x by doing autoregression at the…
EAGLE-2
EAGLE-2 ("Faster Inference of Language Models with Dynamic Draft Trees") is the second generation of the EAGLE family of speculative decoding methods for accelerating large language model inference, introduced…
Edge TPU
The Edge TPU is a small application-specific integrated circuit (ASIC) designed by Google to run machine learning inference on low-power devices.
Etched Sohu
Sohu is a transformer-specialized application-specific integrated circuit (ASIC) built by Etched, a Silicon Valley AI hardware startup founded in 2022 by Harvard dropouts Gavin Uberti, Chris Zhu, and Robert…
ExLlamaV2 (EXL2)
EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single consumer-class NVIDIA GPU.
FP4 (4-bit floating point)
FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep learning.
FPGA
A field-programmable gate array (FPGA) is an integrated circuit whose logic functions and internal wiring are set by the customer after the chip has been manufactured, and can be reset later.
Fireworks AI
Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), image generation models, audio…
Flash-Decoding
Flash-Decoding is an inference-time variant of the FlashAttention algorithm that targets the decoding (autoregressive generation) phase of large language model inference
FlashInfer
FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference.
GPTQ
GPTQ (Generative Pre-trained Transformer Quantization) is a one-shot post-training quantization method that compresses the weights of large language models to 3 or 4 bits using approximate second-order…
GRPO
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for fine-tuning large language models that eliminates the separate critic (value) network used by PPO
Google TPU 8i
Google TPU 8i is an eighth-generation Tensor Processing Unit from Google, built specifically for AI inference rather than model training.
GraphRAG
GraphRAG is a graph-based approach to retrieval-augmented generation developed by Microsoft Research, first described publicly on February 13, 2024 and formalized in the paper "From Local to Global: A Graph…
Groq Hardware
Groq hardware is a family of artificial-intelligence accelerators and multi-chip systems built around a statically scheduled streaming architecture.
H2O (Heavy-Hitter Oracle for KV Cache)
H2O (Heavy-Hitter Oracle) is a training-free, runtime KV cache eviction policy for autoregressive large language model inference.
Inferact
Inferact is an artificial intelligence infrastructure company founded by creators and core maintainers of vLLM, the open-source inference engine that emerged from the University of California, Berkeley in 2023.
Inference-time scaling
Inference-time scaling (also called test-time compute scaling) is the practice of improving an AI model's output quality by allocating more computational resources during inference rather than during training.
InferenceX
InferenceX, launched in October 2025 under the name InferenceMAX, is an open-source benchmark that continuously measures large language model inference performance across AI accelerators and serving software.
Intel Crescent Island
Intel Crescent Island is a data-center GPU from Intel built for artificial-intelligence inference workloads.
Jev (AI model)
Jev is an AI model released in early access on 15 September 2026 by TypeSafe AI, a San Francisco startup founded in 2024 by former OpenAI researcher Diogo Almeida with Erik Gafni and Sasha Sheng.
KTO
KTO (Kahneman-Tversky Optimization) is a method for aligning large language models with human feedback using only a binary signal of whether a model output is desirable or undesirable, rather than the paired…
KV Cache
KV cache, short for key-value cache, is transient model state used during Transformer generation.
KV cache offloading
KV cache offloading is the practice of moving part or all of a Transformer model's KV cache out of accelerator memory (GPU HBM) into a larger, slower tier such as CPU DRAM, local NVMe storage, or remote…
Knowledge Distillation
Knowledge distillation is a training method in which a student model learns from signals produced by a teacher model.
LLM inference engine
An LLM inference engine (also called an LLM serving engine or LLM inference server) is the systems software stack that loads trained large language model weights into GPU or CPU memory and answers user…
LLM.int8()
LLM.int8() is an 8-bit matrix multiplication scheme for large language model inference that preserves accuracy across models up to 175 billion parameters by combining vector-wise quantization with a…
Llama API
The Llama API is Meta's first-party hosted cloud service for running Llama models.
Lookahead Decoding
Lookahead Decoding is a parallel decoding algorithm for accelerating inference in large language models, introduced in November 2023 by Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang from the Hao AI Lab at…
Luminal
Luminal (Luminal AI Inc.) is a San Francisco company that builds an open-source AI inference compiler and sells inference capacity and licensed deployments built on it.
MTIA
MTIA (Meta Training and Inference Accelerator) is a family of custom silicon chips that Meta designs for use in its own data centers rather than for sale.
MediaTek
MediaTek Inc. is a Taiwanese fabless semiconductor company headquartered in Hsinchu, Taiwan.
Medusa
Medusa is a large language model inference acceleration framework that speeds up text generation by adding multiple lightweight decoding heads on top of an existing model to predict several future tokens in…
Model Compression
Model compression is a family of techniques that reduce the size, memory footprint, and computational cost of machine learning models while preserving as much of their original accuracy as possible.
NOSA (Native and Offloadable Sparse Attention)
NOSA (Native and Offloadable Sparse Attention) is a trainable sparse attention mechanism designed so that most of a language model's KV cache can live in CPU memory during decoding without the CPU-to-GPU…
NVIDIA BioNeMo Inference Runtime
NVIDIA BioNeMo Inference Runtime (BioIR) is a GPU-accelerated Python library for biomolecular structure prediction inference.
NVIDIA Dynamo
NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.
NVIDIA Groq LPX Rack
NVIDIA Groq 3 LPX is a rack-scale inference accelerator that NVIDIA introduced at GTC 2026, built around 256 Groq Language Processing Units and designed to sit beside Vera Rubin NVL72 racks as a dedicated…
NVIDIA NIM
NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…
NVIDIA Picasso
NVIDIA Picasso is a cloud-based generative AI foundry from NVIDIA for building, training, and deploying visual generative models that produce images, video, and 3D content from text prompts.
NVIDIA Riva
NVIDIA Riva is a GPU-accelerated software development kit and family of containerized inference services for speech and translation AI, built by NVIDIA.
NVIDIA Rubin CPX
NVIDIA Rubin CPX is a class of GPU announced by NVIDIA on September 9, 2025, purpose-built to accelerate the compute-heavy "context" phase of large-model inference.
NVIDIA TensorRT-LLM
NVIDIA TensorRT-LLM is an open-source library developed by nvidia for high-performance inference of large language models on NVIDIA GPUs.
NVIDIA Triton Inference Server
NVIDIA Triton Inference Server is open-source model deployment software that lets teams run trained models from any machine learning or deep learning framework on any processor (GPU, CPU, or other accelerator)…
Neural Processing Unit
A neural processing unit (NPU) is a processor, or a block inside a larger chip, designed specifically to run neural network inference at high throughput and low power.
NormalFloat 4-bit (NF4)
NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer…