AI Accelerator
An AI accelerator is hardware designed or configured to execute artificial intelligence and machine learning workloads more efficiently than a general-purpose processor executing the same workload without…
AI Inference
Explore AI Inference through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: AI Infrastructure
Articles that also belong to these categories. Counts cover all of AI Inference.
All articles (97)AI Hardware (30)AI Infrastructure (26)Large Language Models (16)Developer Tools (15)NVIDIA (11)Machine Learning (10)Deep Learning (9)Open Source AI (9)AI Companies (8)Training & Optimization (8)MLOps (6)Model Architecture (4)Reasoning Models (4)Algorithms (3)Chinese AI (3)Reinforcement Learning (3)AI Alignment (2)AI Models (2)Anthropic (2)Artificial Intelligence (2)Data Centers (2)Edge computing (2)Google (2)Information Retrieval (2)Meta AI (2)AI Benchmarks (1)AI Research (1)AI Safety (1)Computer Science (1)Enterprise AI (1)Generative AI (1)Image Generation (1)Microsoft (1)Mixture of Experts (1)Natural Language Processing (1)Prompt Engineering (1)Small Language Models (1)Speech & Audio AI (1)Statistics (1)Transformer Models (1)Video Generation (1)
Showing 1-26 of 26 articles
AWS Graviton
AWS Graviton is a family of Arm-based server processors designed by Amazon Web Services for use in its own cloud computing fleet.
Attention-FFN disaggregation
Attention-FFN disaggregation (AFD), also written attention-feedforward disaggregation or A/F disaggregation, is a large language model serving architecture that runs the attention sublayers and the…
Continuous Batching
Continuous batching is a scheduling technique for large language model (LLM) inference servers that inserts new requests into a running batch at the granularity of individual model iterations rather than…
DeepInfra
DeepInfra is a serverless AI inference cloud that hosts open-source and open-weight AI models and serves them to developers through a single pay-per-token API.
Disaggregated serving
Disaggregated serving is an LLM inference architecture that physically separates the prefill phase and the decode phase of text generation onto different sets of GPU hardware.
EAGLE (speculative decoding)
EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a lossless speculative decoding method that speeds up large language model (LLM) inference by 2x to 6.5x by doing autoregression at the…
FPGA
A field-programmable gate array (FPGA) is an integrated circuit whose logic functions and internal wiring are set by the customer after the chip has been manufactured, and can be reset later.
FlashInfer
FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference.
Groq Hardware
Groq hardware is a family of artificial-intelligence accelerators and multi-chip systems built around a statically scheduled streaming architecture.
Inferact
Inferact is an artificial intelligence infrastructure company founded by creators and core maintainers of vLLM, the open-source inference engine that emerged from the University of California, Berkeley in 2023.
KV cache offloading
KV cache offloading is the practice of moving part or all of a Transformer model's KV cache out of accelerator memory (GPU HBM) into a larger, slower tier such as CPU DRAM, local NVMe storage, or remote…
LLM inference engine
An LLM inference engine (also called an LLM serving engine or LLM inference server) is the systems software stack that loads trained large language model weights into GPU or CPU memory and answers user…
MTIA
MTIA (Meta Training and Inference Accelerator) is a family of custom silicon chips that Meta designs for use in its own data centers rather than for sale.
MediaTek
MediaTek Inc. is a Taiwanese fabless semiconductor company headquartered in Hsinchu, Taiwan.
Medusa
Medusa is a large language model inference acceleration framework that speeds up text generation by adding multiple lightweight decoding heads on top of an existing model to predict several future tokens in…
NVIDIA Dynamo
NVIDIA Dynamo is an open-source, low-latency distributed inference serving framework designed to deploy and scale generative AI and reasoning models across large GPU clusters.
NVIDIA NIM
NVIDIA NIM (NVIDIA Inference Microservices) is a set of containerized, prebuilt-and-optimized model-serving microservices from NVIDIA that package an AI model, an optimized inference engine, and an…
NVIDIA Picasso
NVIDIA Picasso is a cloud-based generative AI foundry from NVIDIA for building, training, and deploying visual generative models that produce images, video, and 3D content from text prompts.
OctoAI
OctoAI (originally OctoML) was an American artificial intelligence infrastructure company that operated a generative-AI inference platform and, before its pivot
PagedAttention
PagedAttention is a KV-cache memory management algorithm for serving large language models that applies the virtual-memory paging technique used by operating systems to the GPU, eliminating memory…
Product quantization
Product quantization (PQ) is a vector-compression technique for approximate nearest-neighbor (ANN) search that splits each high-dimensional vector into M equal sub-vectors and quantizes each sub-vector with…
RadixAttention
RadixAttention is a KV cache management technique introduced in SGLang that uses a radix tree data structure to automatically share and reuse cached key-value tensors across inference requests.
SRAM
SRAM, or static random-access memory, is a semiconductor memory that stores each bit in a latch built from cross-coupled inverters.
d-Matrix
d-Matrix is a privately held American semiconductor company headquartered in Santa Clara, California that builds accelerators, I/O cards and software for AI inference in data centers.
d-Matrix Raptor
Raptor is the second-generation AI inference accelerator from d-Matrix, a Santa Clara semiconductor startup, and the first commercial chip built on the company's 3D stacked digital in-memory compute technology