AI Inference

AI Accelerator

An AI accelerator is hardware designed or configured to execute artificial intelligence and machine learning workloads more efficiently than a general-purpose processor executing the same workload without…

AI ComputeAI Hardware

AMD Instinct MI350P

The AMD Instinct MI350P is a PCIe add-in card built on AMD's CDNA 4 architecture, introduced on May 7, 2026 as the third member of the Instinct MI350 series and the first AMD Instinct product in a conventional…

AI ComputeAI Hardware

AWQ (Activation-aware Weight Quantization)

Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…

Deep LearningLarge Language Models

AWS Graviton

AWS Graviton is a family of Arm-based server processors designed by Amazon Web Services for use in its own cloud computing fleet.

AI HardwareAI Infrastructure

AWS Inferentia

AWS Inferentia is a family of custom application specific integrated circuits (ASICs) designed by Amazon Web Services for machine learning inference in the cloud, built to deliver, in AWS's words, "high…

AI ComputeAI Hardware

Adaptive thinking

Adaptive thinking is an inference-time reasoning mode in the Anthropic Messages API in which a claude model decides, on a per-request basis, whether to use extended thinking at all and how much of it to spend

AnthropicReasoning Models

Atlas (inference engine)

Atlas is an open-source large language model inference engine written in Rust and CUDA, built for desktop "unified memory" machines such as the NVIDIA DGX Spark (GB10) and AMD Strix Halo (Ryzen AI Max+ 395)…

Developer ToolsOpen Source AI

Attention-FFN disaggregation

Attention-FFN disaggregation (AFD), also written attention-feedforward disaggregation or A/F disaggregation, is a large language model serving architecture that runs the attention sublayers and the…

AI InfrastructureMachine Learning

Beam search

Beam search is a heuristic search algorithm for sequence generation that, at each decoding step, keeps only the top-K highest-scoring partial sequences (where K is the beam width) and discards the rest

Constitutional Classifiers

Constitutional Classifiers are a machine learning-based safety technique developed by Anthropic to defend large language models against universal jailbreak attacks.

AI AlignmentAI Safety

Context caching

Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…

Developer ToolsLarge Language Models

Continuous Batching

Continuous batching is a scheduling technique for large language model (LLM) inference servers that inserts new requests into a running batch at the granularity of individual model iterations rather than…

AI Infrastructure

DFlash

DFlash is a speculative decoding method in which the draft model is a small block-diffusion network rather than an autoregressive one.

Diffusion ModelsLarge Language Models

Decision 2.0

Decision 2.0 is a family of open-weight models for structured decisions from the vLLM Semantic Router project.

AI ModelsOpen Source AI

DeepInfer

DeepInfer is an open-source inference framework developed by DeepSeek for running its models across accelerator clusters. DeepSeek announced its public release on October 20, 2026.

Chinese AIOpen Source AI

DeepInfra

DeepInfra is a serverless AI inference cloud that hosts open-source and open-weight AI models and serves them to developers through a single pay-per-token API.

AI CompaniesAI Infrastructure

DeepSelect

DeepSelect is an open-source library of top-k selection kernels published by the Chinese AI company DeepSeek.

AI InfrastructureChinese AI

Disaggregated serving

Disaggregated serving is an LLM inference architecture that physically separates the prefill phase and the decode phase of text generation onto different sets of GPU hardware.

AI InfrastructureArtificial Intelligence

Dynamic inference

Dynamic inference, also called input-adaptive inference, conditional computation, or adaptive computation

Mixture of Experts

EAGLE (speculative decoding)

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a lossless speculative decoding method that speeds up large language model (LLM) inference by 2x to 6.5x by doing autoregression at the…

AI Infrastructure

EAGLE-2

EAGLE-2 ("Faster Inference of Language Models with Dynamic Draft Trees") is the second generation of the EAGLE family of speculative decoding methods for accelerating large language model inference, introduced…

Large Language Models

Edge TPU

The Edge TPU is a small application-specific integrated circuit (ASIC) designed by Google to run machine learning inference on low-power devices.

AI HardwareEdge computing

Etched Sohu

Sohu is a transformer-specialized application-specific integrated circuit (ASIC) built by Etched, a Silicon Valley AI hardware startup founded in 2022 by Harvard dropouts Gavin Uberti, Chris Zhu, and Robert…

AI ComputeAI Hardware

ExLlamaV2 (EXL2)

EXL2 (ExLlamaV2 format) is an open-source, mixed-bit weight-quantization format for compressing large language models so they run fast on a single consumer-class NVIDIA GPU.

Developer ToolsOpen Source AI

FP4 (4-bit floating point)

FP4 (4-bit floating point) is a numerical format that stores a real number in just 4 bits, the smallest floating-point type in mainstream use for deep learning.

AI ComputeAI Hardware

FPGA

A field-programmable gate array (FPGA) is an integrated circuit whose logic functions and internal wiring are set by the customer after the chip has been manufactured, and can be reset later.

AI HardwareAI Infrastructure

Fireworks AI

Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), image generation models, audio…

AI CompaniesDeveloper Tools

Flash-Decoding

Flash-Decoding is an inference-time variant of the FlashAttention algorithm that targets the decoding (autoregressive generation) phase of large language model inference

Algorithms

FlashInfer

FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference.

AI InfrastructureDeveloper Tools

FlashMLA

FlashMLA is an open-source library of attention kernels from DeepSeek, built around Multi-head Latent Attention (MLA), the attention variant DeepSeek uses to shrink the KV cache in its large language models.

AI InfrastructureDeveloper Tools

GPTQ

GPTQ (Generative Pre-trained Transformer Quantization) is a one-shot post-training quantization method that compresses the weights of large language models to 3 or 4 bits using approximate second-order…

Deep Learning

GRPO

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for fine-tuning large language models that eliminates the separate critic (value) network used by PPO

Chinese AIReasoning Models

Google TPU 8i

Google TPU 8i is an eighth-generation Tensor Processing Unit from Google, built specifically for AI inference rather than model training.

AI ComputeAI Hardware

GraphRAG

GraphRAG is a graph-based approach to retrieval-augmented generation developed by Microsoft Research, first described publicly on February 13, 2024 and formalized in the paper "From Local to Global: A Graph…

Information RetrievalMicrosoft

Groq Hardware

Groq hardware is a family of artificial-intelligence accelerators and multi-chip systems built around a statically scheduled streaming architecture.

AI ComputeAI Hardware

H2O (Heavy-Hitter Oracle for KV Cache)

H2O (Heavy-Hitter Oracle) is a training-free, runtime KV cache eviction policy for autoregressive large language model inference.

Large Language Models

Humming

Humming is an open-source GPU kernel library for matrix multiplication in quantized inference.

AI ComputeAI Infrastructure

Inco AI

Inco AI is an AI inference company that builds speculative decoding draft models, quantized checkpoints, a hosted inference platform and an open-source local inference engine for Macs.

AI CompaniesOpen Source AI

Inferact

Inferact is an artificial intelligence infrastructure company founded by creators and core maintainers of vLLM, the open-source inference engine that emerged from the University of California, Berkeley in 2023.

AI CompaniesAI Infrastructure

Inference-time scaling

Inference-time scaling (also called test-time compute scaling) is the practice of improving an AI model's output quality by allocating more computational resources during inference rather than during training.

AI ResearchArtificial Intelligence

InferenceX

InferenceX, launched in October 2025 under the name InferenceMAX, is an open-source benchmark that continuously measures large language model inference performance across AI accelerators and serving software.

AI BenchmarksAI Compute

Intel Crescent Island

Intel Crescent Island is a data-center GPU from Intel built for artificial-intelligence inference workloads.

AI ComputeAI Hardware

Jev (AI model)

Jev is an AI model released in waitlisted early access on 15 September 2026, and opened to all developers on 20 September, by TypeSafe AI, a San Francisco startup founded in 2024 by former OpenAI researcher…

AI ModelsMachine Learning

KTO

KTO (Kahneman-Tversky Optimization) is a method for aligning large language models with human feedback using only a binary signal of whether a model output is desirable or undesirable, rather than the paired…

AI AlignmentReinforcement Learning

KV Cache

KV cache, short for key-value cache, is transient model state used during Transformer generation.

Deep LearningMachine Learning

KV cache offloading

KV cache offloading is the practice of moving part or all of a Transformer model's KV cache out of accelerator memory (GPU HBM) into a larger, slower tier such as CPU DRAM, local NVMe storage, or remote…

AI InfrastructureLarge Language Models

Knowledge Distillation

Knowledge distillation is a training method in which a student model learns from signals produced by a teacher model.

Deep LearningMachine Learning

LLM Model Routing

LLM model routing is the selection of a large language model to handle a request from a pool of candidate models.

Large Language ModelsMLOps

LLM inference engine

An LLM inference engine (also called an LLM serving engine or LLM inference server) is the systems software stack that loads trained large language model weights into GPU or CPU memory and answers user…

AI InfrastructureLarge Language Models

LLM.int8()

LLM.int8() is an 8-bit matrix multiplication scheme for large language model inference that preserves accuracy across models up to 175 billion parameters by combining vector-wise quantization with a…

Large Language Models

Llama API

The Llama API is Meta's first-party hosted cloud service for running Llama models.

Developer ToolsMeta AI

Lookahead Decoding

Lookahead Decoding is a parallel decoding algorithm for accelerating inference in large language models, introduced in November 2023 by Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang from the Hao AI Lab at…

AlgorithmsLarge Language Models

Luminal

Luminal (Luminal AI Inc.) is a San Francisco company that builds an open-source AI inference compiler and sells inference capacity and licensed deployments built on it.

AI CompaniesDeveloper Tools

MTIA

MTIA (Meta Training and Inference Accelerator) is a family of custom silicon chips that Meta designs for use in its own data centers rather than for sale.

AI ComputeAI Hardware

Magnitude (inference engine)

Magnitude is an open-source inference engine for running open-weight language models on local hardware, developed by Magnitude AI Inc., a San Francisco startup from the Y Combinator Summer 2025 batch.

AI Tools & ProductsDeveloper Tools

MediaTek

MediaTek Inc. is a Taiwanese fabless semiconductor company headquartered in Hsinchu, Taiwan.

AI CompaniesAI Hardware

Medusa

Medusa is a large language model inference acceleration framework that speeds up text generation by adding multiple lightweight decoding heads on top of an existing model to predict several future tokens in…

AI Infrastructure

Meta Private Processing

Private Processing is Meta's confidential computing infrastructure for running AI workloads on user data in its data centers.

AI InfrastructureMeta AI

Model Compression

Model compression is a family of techniques that reduce the size, memory footprint, and computational cost of machine learning models while preserving as much of their original accuracy as possible.

Modular (company)

Modular (legally Modular Inc, also referred to as Modular AI) is an American AI infrastructure software company that builds tools for running AI models on chips from many vendors without rewriting code for…

AI CompaniesAI Infrastructure