AI Inference

Explore AI Inference through related topics and the articles other pages reference most.

97 articlesSearch and filter Follow updates by RSS

Explore articles

Browse subtopics: Large Language Models

Articles that also belong to these categories. Counts cover all of AI Inference.

Showing 1-16 of 16 articles

AWQ (Activation-aware Weight Quantization)

Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…

Deep Learning Large Language Models

Context caching

Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…

Developer Tools Large Language Models

EAGLE-2

EAGLE-2 ("Faster Inference of Language Models with Dynamic Draft Trees") is the second generation of the EAGLE family of speculative decoding methods for accelerating large language model inference, introduced…

Large Language Models

Fireworks AI

Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), image generation models, audio…

AI Companies Developer Tools

H2O (Heavy-Hitter Oracle for KV Cache)

H2O (Heavy-Hitter Oracle) is a training-free, runtime KV cache eviction policy for autoregressive large language model inference.

Large Language Models

KV cache offloading

KV cache offloading is the practice of moving part or all of a Transformer model's KV cache out of accelerator memory (GPU HBM) into a larger, slower tier such as CPU DRAM, local NVMe storage, or remote…

AI Infrastructure Large Language Models

LLM inference engine

An LLM inference engine (also called an LLM serving engine or LLM inference server) is the systems software stack that loads trained large language model weights into GPU or CPU memory and answers user…

AI Infrastructure Large Language Models

LLM.int8()

LLM.int8() is an 8-bit matrix multiplication scheme for large language model inference that preserves accuracy across models up to 175 billion parameters by combining vector-wise quantization with a…

Large Language Models

Lookahead Decoding

Lookahead Decoding is a parallel decoding algorithm for accelerating inference in large language models, introduced in November 2023 by Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang from the Hao AI Lab at…

Algorithms Large Language Models

NOSA (Native and Offloadable Sparse Attention)

NOSA (Native and Offloadable Sparse Attention) is a trainable sparse attention mechanism designed so that most of a language model's KV cache can live in CPU memory during decoding without the CPU-to-GPU…

Chinese AI Large Language Models

QLoRA

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…

Deep Learning Large Language Models

SmoothQuant

SmoothQuant is a training-free, accuracy-preserving post-training quantization (PTQ) method that enables 8-bit weight and 8-bit activation (W8A8) integer inference for large language models without retraining…

Large Language Models

SparDA

SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026.

Large Language Models Model Architecture

Speculative Decoding

Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…

Deep Learning Large Language Models

StreamingLLM

StreamingLLM is an inference-time technique that allows pretrained transformer language models, originally trained with a finite attention window

Large Language Models

Top-k sampling

Top-k sampling is a decoding strategy for autoregressive language models that restricts each generation step to the k most probable next tokens.

Algorithms Large Language Models