AI Inference

AI Accelerator

An AI accelerator is hardware designed or configured to execute artificial intelligence and machine learning workloads more efficiently than a general-purpose...

AI HardwareAI Infrastructure

AMD Instinct MI350P

The AMD Instinct MI350P is a PCIe add-in card built on AMD's CDNA 4 architecture, introduced on May 7, 2026 as the third member of the Instinct MI350 series...

AI Hardware

AWQ (Activation-aware Weight Quantization)

Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally...

Deep LearningLarge Language Models

AWS Graviton

AWS Graviton is a family of Arm-based server processors designed by Amazon Web Services for use in its own cloud computing fleet. The first generation launched...

AI HardwareAI Infrastructure

AWS Inferentia

AWS Inferentia is a family of custom application specific integrated circuits (ASICs) designed by Amazon Web Services for machine learning inference in the...

AI Hardware

Adaptive thinking

Adaptive thinking is an inference-time reasoning mode in the Anthropic Messages API in which a claude model decides, on a per-request basis, whether to use...

AnthropicReasoning Models

Beam search

Beam search is a heuristic search algorithm for sequence generation that, at each decoding step, keeps only the top-K highest-scoring partial sequences (where...

Constitutional Classifiers

Constitutional Classifiers are a machine learning-based safety technique developed by Anthropic to defend large language models against universal jailbreak...

AI AlignmentAI Safety

Context caching

Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier...

Developer ToolsLarge Language Models

Continuous Batching

Continuous batching is a scheduling technique for large language model (LLM) inference servers that inserts new requests into a running batch at the...

AI Infrastructure

DeepInfra

DeepInfra is a serverless AI inference cloud that hosts open-source and open-weight AI models and serves them to developers through a single pay-per-token API....

AI CompaniesAI Infrastructure

Disaggregated serving

Disaggregated serving is an LLM inference architecture that physically separates the prefill phase and the decode phase of text generation onto different sets...

AI InfrastructureArtificial Intelligence

Dynamic inference

Dynamic inference, also called input-adaptive inference, conditional computation, or adaptive computation, is the family of techniques that adapts a neural...

Mixture of Experts

EAGLE (speculative decoding)

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a lossless speculative decoding method that speeds up large language model (LLM)...

AI Infrastructure

EAGLE-2

EAGLE-2 ("Faster Inference of Language Models with Dynamic Draft Trees") is the second generation of the EAGLE family of speculative decoding methods for...

Large Language Models

Edge TPU

The Edge TPU is a small application-specific integrated circuit (ASIC) designed by Google to run machine learning inference on low-power devices. Google's...

AI HardwareEdge computing

Etched Sohu

--- Manufacturer United States Announced Pre-production (not yet shipping as of May 2026) Architecture TSMC 4nm (N4 family) Memory Approximately...

AI Hardware

ExLlamaV2 (EXL2)

EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single...

Developer ToolsOpen Source AI

FP4 (4-bit floating point)

FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep...

AI HardwareTraining & Optimization

FPGA

A field-programmable gate array (FPGA) is an integrated circuit whose logic functions and internal wiring are set by the customer after the chip has been...

AI HardwareAI Infrastructure

Fireworks AI

Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language...

AI CompaniesDeveloper Tools

Flash-Decoding

Flash-Decoding is an inference-time variant of the FlashAttention algorithm that targets the decoding (autoregressive generation) phase of large language model...

Algorithms

GPTQ

GPTQ (Generative Pre-trained Transformer Quantization) is a one-shot post-training quantization method that compresses the weights of large language models to...

Deep Learning

GRPO

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for fine-tuning large language models that eliminates the separate critic...

Chinese AIReasoning Models

Google TPU 8i

Google TPU 8i is an eighth-generation Tensor Processing Unit from Google, built specifically for AI inference rather than model training. It was previewed at...

AI HardwareGoogle

GraphRAG

GraphRAG is a graph-based approach to retrieval-augmented generation developed by Microsoft Research, first described publicly on February 13, 2024 and...

Information RetrievalMicrosoft

Groq Hardware

Groq hardware is a family of artificial-intelligence accelerators and multi-chip systems built around a statically scheduled streaming architecture. Groq's...

AI HardwareAI Infrastructure

H2O (Heavy-Hitter Oracle for KV Cache)

H2O (Heavy-Hitter Oracle) is a training-free, runtime KV cache eviction policy for autoregressive large language model inference. It identifies a small subset...

Large Language Models

Inferact

Value --- Inferact Inc. Type January 22, 2026 Named founding members Simon Mo Origin $150 million seed (announced January 22, 2026) Reported...

AI CompaniesAI Infrastructure

Inference-time scaling

Inference-time scaling (also called test-time compute scaling) is the practice of improving an AI model's output quality by allocating more computational...

AI ResearchArtificial Intelligence

InferenceX

InferenceX, launched in October 2025 under the name InferenceMAX, is an open-source benchmark that continuously measures large language model inference...

AI BenchmarksAI Hardware

Intel Crescent Island

Intel Crescent Island is a data-center GPU from Intel built for artificial-intelligence inference workloads. Intel announced it on October 14, 2025 at the Open...

AI Hardware

KTO

KTO (Kahneman-Tversky Optimization) is a method for aligning large language models with human feedback using only a binary signal of whether a model output is...

AI AlignmentReinforcement Learning

KV Cache

KV cache, short for key-value cache, is transient model state used during Transformer generation. At each attention layer, it retains the key and value vectors...

Deep LearningMachine Learning

Knowledge Distillation

Knowledge distillation is a training method in which a student model learns from signals produced by a teacher model. The teacher may be larger, an ensemble, a...

Deep LearningMachine Learning

LLM inference engine

An LLM inference engine (also called an LLM serving engine or LLM inference server) is the systems software stack that loads trained large language model...

AI InfrastructureLarge Language Models

LLM.int8()

LLM.int8() is an 8-bit matrix multiplication scheme for large language model inference that preserves accuracy across models up to 175 billion parameters by...

Large Language Models

Llama API

The Llama API is Meta's first-party hosted cloud service for running Llama models. Announced on April 29, 2025 at Meta's inaugural LlamaCon developer...

Developer ToolsMeta AI

Lookahead Decoding

Lookahead Decoding is a parallel decoding algorithm for accelerating inference in large language models, introduced in November 2023 by Yichao Fu, Peter...

AlgorithmsLarge Language Models

MTIA

MTIA (Meta Training and Inference Accelerator) is a family of custom silicon chips that Meta designs for use in its own data centers rather than for sale. Meta...

AI HardwareAI Infrastructure

MediaTek

MediaTek Inc. is a Taiwanese fabless semiconductor company headquartered in Hsinchu, Taiwan. It designs system-on-chip (SoC) processors and does not operate...

AI CompaniesAI Hardware

Medusa

Medusa is a large language model inference acceleration framework that speeds up text generation by adding multiple lightweight decoding heads on top of an...

AI Infrastructure

Model Compression

Model compression is a family of techniques that reduce the size, memory footprint, and computational cost of machine learning models while preserving as much...

NVIDIA Dynamo

NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across...

AI InfrastructureDeveloper Tools

NVIDIA Groq LPX Rack

NVIDIA Groq 3 LPX is a rack-scale inference accelerator that NVIDIA introduced at GTC 2026, built around 256 Groq Language Processing Units and designed to sit...

AI HardwareNVIDIA

NVIDIA NIM

NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model,...

AI InfrastructureDeveloper Tools

NVIDIA Picasso

See also: Model Deployment and artificial intelligence applications See also: Image generation, Video generation, and 3D generation NVIDIA Picasso is a...

AI HardwareAI Infrastructure

NVIDIA Riva

NVIDIA Riva is a GPU-accelerated software development kit and family of containerized inference services for speech and translation AI, built by NVIDIA. Riva...

Developer ToolsEnterprise AI

NVIDIA Rubin CPX

NVIDIA Rubin CPX is a class of GPU announced by NVIDIA on September 9, 2025, purpose-built to accelerate the compute-heavy "context" phase of large-model...

AI HardwareNVIDIA

NVIDIA TensorRT-LLM

NVIDIA TensorRT-LLM is an open-source library developed by nvidia for high-performance inference of large language models on NVIDIA GPUs. It provides a Python...

NVIDIAOpen Source AI

NVIDIA Triton Inference Server

See also: Model Deployment, artificial intelligence applications, and GPU Computing NVIDIA Triton Inference Server is open-source model deployment software...

Deep LearningDeveloper Tools

Neural Processing Unit

A neural processing unit (NPU) is a processor, or a block inside a larger chip, designed specifically to run neural network inference at high throughput and...

AI HardwareEdge computing

NormalFloat 4-bit (NF4)

NormalFloat 4-bit (NF4) is a 4-bit numerical data type for storing the weights of deep neural networks, introduced in the 2023 QLoRA paper by Tim Dettmers,...

Training & Optimization

OctoAI

OctoAI (originally OctoML) was an American artificial intelligence infrastructure company that operated a generative-AI inference platform and, before its...

AI CompaniesAI Infrastructure

Offline inference

See also: online inference, static inference, dynamic inference, inference, machine learning terms Offline inference (also called batch inference, static...

MLOps

On-device AI

On-device AI is the practice of running machine learning models on the phone, laptop, watch, or embedded board a person is actually using, instead of sending...

AI HardwareMachine Learning

Online inference

See also: offline inference, static inference, dynamic inference, inference, machine learning terms Online inference (also called dynamic inference, real-time...

MLOps

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit developed by Intel for optimizing and deploying deep learning...

Developer ToolsOpen Source AI

Optimum-Quanto

Optimum Quanto, commonly referred to as Quanto, is a PyTorch-based quantization toolkit developed and maintained by Hugging Face that provides linear weight...

Developer ToolsOpen Source AI

PagedAttention

PagedAttention is a KV-cache memory management algorithm for serving large language models that applies the virtual-memory paging technique used by operating...

AI InfrastructureModel Architecture