AWQ (Activation-aware Weight Quantization)
Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task…
AI Inference
Explore AI Inference through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of AI Inference.
All articles (97)AI Hardware (30)AI Infrastructure (26)Large Language Models (16)Developer Tools (15)NVIDIA (11)Machine Learning (10)Deep Learning (9)Open Source AI (9)AI Companies (8)Training & Optimization (8)MLOps (6)Model Architecture (4)Reasoning Models (4)Algorithms (3)Chinese AI (3)Reinforcement Learning (3)AI Alignment (2)AI Models (2)Anthropic (2)Artificial Intelligence (2)Data Centers (2)Edge computing (2)Google (2)Information Retrieval (2)Meta AI (2)AI Benchmarks (1)AI Research (1)AI Safety (1)Computer Science (1)Enterprise AI (1)Generative AI (1)Image Generation (1)Microsoft (1)Mixture of Experts (1)Natural Language Processing (1)Prompt Engineering (1)Small Language Models (1)Speech & Audio AI (1)Statistics (1)Transformer Models (1)Video Generation (1)
Showing 1-9 of 9 articles
GPTQ
GPTQ (Generative Pre-trained Transformer Quantization) is a one-shot post-training quantization method that compresses the weights of large language models to 3 or 4 bits using approximate second-order…
KV Cache
KV cache, short for key-value cache, is transient model state used during Transformer generation.
Knowledge Distillation
Knowledge distillation is a training method in which a student model learns from signals produced by a teacher model.
NVIDIA Triton Inference Server
NVIDIA Triton Inference Server is open-source model deployment software that lets teams run trained models from any machine learning or deep learning framework on any processor (GPU, CPU, or other accelerator)…
QLoRA
QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that finetunes a 65-billion-parameter large language model on a single 48 GB GPU while matching the quality of full 16-bit…
Quantization
Quantization is the representation of values from a large or continuous set by values from a smaller, usually finite set.
Speculative Decoding
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…
YaRN
YaRN (Yet another RoPE extensioN) is a compute-efficient method for extending the context window of large language models that use Rotary Position Embeddings (RoPE).