AWQ (Activation-aware Weight Quantization)
Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…
Large Language Models
Explore language models, how they work, and the techniques used to build applications with them.
Explore articles
Browse subtopics: AI Inference
Articles that also belong to these categories. Counts cover all of Large Language Models.
All articles (545)AI Models (184)Open Source AI (165)Chinese AI (92)Natural Language Processing (58)AI Companies (53)Machine Learning (49)Mixture of Experts (46)Multimodal AI (41)Reasoning Models (39)Artificial Intelligence (37)OpenAI (35)AI Benchmarks (33)Anthropic (31)Google DeepMind (26)AI Agents (22)AI Safety (21)Developer Tools (20)Meta AI (20)AI Code Generation (18)Small Language Models (18)Training & Optimization (18)AI Inference (16)Model Architecture (14)Generative AI (13)NVIDIA (13)Prompt Engineering (13)AI Research (12)Deep Learning (12)Google (12)Model Evaluation (12)Data & Datasets (11)AI Tools & Products (9)Conversational AI (9)Microsoft (9)Transformer Models (9)AI Infrastructure (7)Diffusion Models (7)AI Alignment (5)Enterprise AI (5)Research Organizations (5)AI Hardware (4)Healthcare AI (4)Information Retrieval (4)Interpretability (4)People (4)Mathematics (3)AI History (2)AI for Science (2)Algorithms (2)Computer Vision (2)Legal AI (2)Reinforcement Learning (2)Speech & Audio AI (2)AI Policy (1)AI Policy & Regulation (1)AI in Cybersecurity (1)AI in Gaming (1)Autonomous Vehicles (1)ChatGPT (1)China AI (1)Drug Discovery (1)Neural Networks (1)Robotics (1)Software Development (1)Video Generation (1)
Showing 1-16 of 16 articles
Context caching
Context caching is a large-language-model API feature that stores parts of a request's input (system prompts, instructions, attached documents, or earlier conversation turns) on the provider's infrastructure…
EAGLE-2
EAGLE-2 ("Faster Inference of Language Models with Dynamic Draft Trees") is the second generation of the EAGLE family of speculative decoding methods for accelerating large language model inference, introduced…
Fireworks AI
Fireworks AI is an artificial intelligence infrastructure company that runs a high-performance inference platform for deploying and serving open large language models (LLMs), image generation models, audio…
H2O (Heavy-Hitter Oracle for KV Cache)
H2O (Heavy-Hitter Oracle) is a training-free, runtime KV cache eviction policy for autoregressive large language model inference.
KV cache offloading
KV cache offloading is the practice of moving part or all of a Transformer model's KV cache out of accelerator memory (GPU HBM) into a larger, slower tier such as CPU DRAM, local NVMe storage, or remote…
LLM inference engine
An LLM inference engine (also called an LLM serving engine or LLM inference server) is the systems software stack that loads trained large language model weights into GPU or CPU memory and answers user…
LLM.int8()
LLM.int8() is an 8-bit matrix multiplication scheme for large language model inference that preserves accuracy across models up to 175 billion parameters by combining vector-wise quantization with a…
Lookahead Decoding
Lookahead Decoding is a parallel decoding algorithm for accelerating inference in large language models, introduced in November 2023 by Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang from the Hao AI Lab at…
NOSA (Native and Offloadable Sparse Attention)
NOSA (Native and Offloadable Sparse Attention) is a trainable sparse attention mechanism designed so that most of a language model's KV cache can live in CPU memory during decoding without the CPU-to-GPU…
QLoRA
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
SmoothQuant
SmoothQuant is a training-free, accuracy-preserving post-training quantization (PTQ) method that enables 8-bit weight and 8-bit activation (W8A8) integer inference for large language models without retraining…
SparDA
SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026.
Speculative Decoding
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…
StreamingLLM
StreamingLLM is an inference-time technique that allows pretrained transformer language models, originally trained with a finite attention window
Top-k sampling
Top-k sampling is a decoding strategy for autoregressive language models that restricts each generation step to the k most probable next tokens.