Last reviewed
Sources
No citations yet
Review status
Needs citations
Revision
v4 ยท 3,958 words
Add missing citations, update stale details, or suggest a clearer explanation.
Activation-aware Weight Quantization (AWQ) is a post-training quantization method for large language models that compresses weights to 4-bit (and optionally 3-bit) integers while keeping near-FP16 task accuracy. Its central finding is that protecting only the top 1% of "salient" weight channels, identified by activation magnitude rather than weight magnitude, removes most of the quantization error, so AWQ applies a per-channel scaling transformation that keeps every weight in the same 4-bit format and adds no inference overhead.[1] The paper states the core claim directly: "not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To identify salient weight channels, we should refer to the activation distribution, not weights."[1] AWQ was introduced in a paper by Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han of MIT, Tsinghua University, and the MIT-IBM Watson AI Lab, first posted to arXiv on June 1, 2023.[1][2] The method received the Best Paper Award at the Seventh Conference on Machine Learning and Systems (MLSys 2024) in Santa Clara, California.[3] The companion TinyChat inference engine implements W4A16 (4-bit weight, 16-bit activation) GPU kernels that the paper reports deliver "more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs."[1][2]
The growth of decoder-only transformer language models created a substantial deployment problem. A model with 70 billion parameters in 16-bit floating point requires roughly 140 GB of storage for its weights alone; a 7-billion-parameter model requires about 14 GB, which exceeds the VRAM on most consumer GPUs and many cloud instances. Lower-precision representations reduce both the storage footprint and the DRAM bandwidth consumed during inference, with the latter being the dominant cost during single-request decoding on a memory-bound accelerator.[1]
Quantization for neural networks has a long history, but the regime of interest for large language models (LLMs) is weight-only post-training quantization (PTQ) in 3- to 4-bit integer formats. Two prior baselines defined the space when AWQ was introduced:
AWQ takes a different path. Rather than correcting quantization error after it occurs, AWQ asks which weights, if quantized poorly, would most damage downstream computation. The proposed answer is "those weights multiplied by large activations during inference," a signal that does not require gradient information.[1][2]
The arXiv preprint "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" was first posted as version 1 on June 1, 2023.[5] Subsequent revisions on arXiv updated the title in v5 to "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration" to reflect the strong on-device results added to the camera-ready version.[5] The paper was accepted to the MLSys 2024 conference, where it received the Best Paper Award; the official proceedings list the same author roster as the arXiv submission.[3][6]
Lead author Ji Lin completed the research as a PhD student in Song Han's group at MIT. Song Han is principal investigator of the HAN Lab at MIT (also distinguished scientist at NVIDIA). Author affiliations across the paper and the MIT HAN Lab project page span MIT, Tsinghua University, and the MIT-IBM Watson AI Lab.[2]
The official reference implementation lives at the GitHub repository mit-han-lab/llm-awq, released under the MIT License and including the TinyChat inference engine.[7]
The core empirical finding of the paper is illustrated by a controlled ablation. Consider quantizing a linear layer y = W x to INT4. Suppose 1% of the columns of W (corresponding to 1% of input channels) are left in FP16 while the remaining 99% are quantized with standard RTN. Three strategies for choosing the protected columns are compared on LLaMA-7B:[1]
Strategies 1 and 2 give marginal gains. Strategy 3 closes most of the gap between INT4-RTN and FP16 on WikiText perplexity. The paper reports that mixed-precision protection of the top 1% of activation-selected channels reduces LLaMA-7B INT4 perplexity from about 43.2 (RTN, 3-bit demonstration) toward the FP16 baseline, while random and weight-magnitude selection do not.[1]
The intuition is that the output error of a quantized linear layer is (W minus Q(W)) times x. The contribution of column i to that error is (w_i minus Q(w_i)) times x_i. A weight column multiplied by an input channel that is always near zero contributes little error regardless of how poorly it is quantized; a column multiplied by an input channel with large magnitude contributes proportionally more. Activation magnitude, averaged over a calibration set, is therefore a cleaner importance signal than weight magnitude.[1]
Keeping 1% of columns in FP16 and the rest in INT4 is hardware-unfriendly because the resulting layer requires mixed-precision matrix multiplication. AWQ therefore replaces explicit precision protection with an equivalent per-channel scaling transformation that keeps every weight in the same INT4 format.[1][2]
The mathematical core of AWQ is the following observation. For a single weight w in a salient channel and an input activation x, the output contribution before quantization is w times x. If w is multiplied by a scale s > 1 before quantization, the scaled weight is w' = s times w. Quantizing w' to INT4 produces some absolute error delta, but the relative error delta / w' equals delta / (s times w), which is 1/s times the relative error of the unscaled weight. Larger s shrinks the relative error on this channel.[1]
For the layer output to remain unchanged, the corresponding input activation x must be divided by s. The substitution (Q(s times w) / s) times x equals Q(s times w) times (x / s), so dividing the activation cancels the weight scale. In a transformer block, the upstream operation feeding the linear layer is typically a layer normalization (or a prior linear projection), and the 1/s scaling can be absorbed into the affine parameters of that upstream operation at offline-quantization time. This means inference adds no extra arithmetic; the inverse scale is baked into the preceding layer.[1]
AWQ formulates the search for per-channel scales s = (s_1, ..., s_C) as the minimization of the per-layer output reconstruction error over a calibration set:
L(s) = || Q(W diag(s)) diag(s)^{-1} X \u2212 W X ||
where X is the matrix of input activations gathered from the calibration set and W is the FP16 weight matrix. The candidate scales are parameterized as s_i = (mean absolute activation of channel i) raised to a power alpha, with alpha searched over a small one-dimensional grid (the paper uses 20 candidate values).[1] This is a fast forward-pass search rather than a gradient-based optimization. The paper emphasizes that "AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set," and reports that the full quantization procedure runs in minutes for a 7B model on a single GPU.[1][2]
A complication is that increasing s indefinitely is not free. The scaled weight column now has a larger dynamic range, which compresses the INT4 grid spacing for the rest of the weights in its quantization group and can introduce error elsewhere. The grid search picks the alpha that balances protecting salient channels against disturbing unprotected ones.[1]
The paper also discusses a clipping step that adjusts the maximum value used to compute the per-group scale, trading a small additional clipping error for a tighter integer grid on the bulk of the weights.[1]
AWQ uses group-wise integer quantization for the actual INT4 encoding. With a group size of 128 (the standard setting), every 128 consecutive weights along the input-channel dimension share one FP16 scale and one zero-point. This is fine enough to track local distribution shape but coarse enough to limit storage overhead.[1][7]
For a single group of 128 4-bit weights:
Total: about 68 bytes per group, versus 256 bytes per group in FP16. The effective compression ratio is roughly 3.76x, somewhat less than the theoretical 4x because of the scale and zero-point overhead.[7]
The Hugging Face documentation refers to this configuration as bits: 4, group_size: 128, with zero_point: true in the model config.json. Group size 64 is also supported as a higher-fidelity variant, and version: "gemm", version: "gemv", or version: "exllama" selects the underlying CUDA kernel implementation.[8]
The shorthand W4A16g128 appears in model cards and serving stacks: 4-bit weights, 16-bit activations, group size 128. This distinguishes AWQ from weight-and-activation quantization schemes such as SmoothQuant (W8A8) or QuaRot (W4A4).[9]
AWQ requires a small calibration set to estimate per-channel activation magnitudes. The reference implementation defaults to a few hundred sequences drawn from the Pile or a similar web-text corpus; published recipes commonly use 128 to 512 samples.[1][7] Each sample is run through the model to record the mean absolute value of activations on each input channel of each linear layer. These statistics are averaged across the calibration set to yield channel importance scores used to compute s.[1]
Because AWQ only needs first-order activation statistics (mean absolute values), not gradients or Hessians, its calibration set is far smaller than GPTQ's (which is typically 2,048 samples) and its sensitivity to the calibration domain is lower.[1] The paper reports that under deliberate calibration-domain mismatch, AWQ perplexity rises by roughly 0.5 to 0.6 points on the evaluation distribution while GPTQ perplexity rises by 2.3 to 4.9 points under the same setup.[1] This calibration-domain tolerance has practical consequences for proprietary models where representative calibration data is hard to obtain.
AWQ-quantized checkpoints by themselves do not run faster than FP16; they require an inference kernel that dequantizes INT4 weights and multiplies them by FP16 activations efficiently. The MIT HAN Lab released TinyChat as the companion runtime.[2][7]
TinyChat implements fused W4A16 CUDA kernels that:
Through kernel fusion and platform-aware weight packing, TinyChat reports more than 3x speedup over the Hugging Face FP16 baseline; the paper also reports that its 4-bit kernels run roughly 1.45x faster than GPTQ and about 1.85x faster than the cuBLAS FP16 implementation.[1][2] End-to-end results reported in the paper and project page include 2.7x speedup on the NVIDIA RTX 4090 desktop GPU and 2.9x speedup on the NVIDIA Jetson Orin embedded GPU for LLaMA-class models running with TinyChat AWQ kernels relative to the Hugging Face FP16 baseline.[1][2] The system runs a quantized Llama 2 70B model on a Jetson Orin 64 GB platform at interactive generation speed, where the FP16 variant would not fit in memory at all.[2]
A subsequent release, TinyChat 2.0 (December 2024), extended the engine to vision-language models, added VILA and NVILA support, and reports 1.5x to 1.7x faster prefill compared to the original TinyChat through revised kernel scheduling.[2] The HAN Lab also publishes TinyChatEngine, a C++ counterpart targeting on-device deployment on Apple silicon, ARM CPUs, and other accelerators where CUDA is not available.[7]
The AWQ paper reports zero-shot perplexity and downstream accuracy across LLaMA-1, LLaMA-2, OPT, Mistral, and several vision-language models. Headline numbers from the paper at W4A16 group-size-128 include the following ranges (all on WikiText-2 perplexity, lower is better):[1]
At 3-bit, the gap widens but AWQ retains a large advantage over RTN. The paper reports LLaMA-7B WikiText-2 at INT3 of about 13.0 with AWQ versus 43.2 with RTN.[1]
Beyond perplexity, the paper evaluates common-sense reasoning suites (HellaSwag, PIQA, ARC, WinoGrande, and others) and the instruction-tuned Vicuna chatbot benchmark, with AWQ showing minimal degradation versus FP16 at 4-bit precision.[1] The work also reports successful low-bit quantization of vision-language models, including LLaVA-13B and OpenFlamingo-9B, evaluated across eleven multimodal benchmarks, making AWQ one of the first weight-only PTQ methods to demonstrate this generality.[1][2]
The following table summarizes how AWQ relates to the most-cited alternative weight-only and weight-and-activation quantization methods discussed in the AWQ paper and the surrounding literature.
| Method | Format | Approach | Calibration | Notes |
|---|---|---|---|---|
| RTN | W4A16 | Round-to-nearest | None | Trivial; degrades on small models, fails at 3-bit. |
| GPTQ | W4A16 | Hessian-based column update | About 2,048 samples | Strong 4-bit accuracy; hours to run; sensitive to calibration domain.[4] |
| AWQ | W4A16 | Activation-aware channel scaling | About 128 to 512 samples | Minutes to run; tolerant of calibration mismatch.[1] |
| SmoothQuant | W8A8 | Per-channel migration of activation outliers to weights | Small set | Targets INT8 tensor-core compute, complementary to AWQ.[9] |
| QuaRot / SpinQuant | W4A4 | Rotation matrices to suppress activation outliers | Calibration set | Enables full INT4 compute at extra calibration cost. |
| AQLM, QuIP# | sub-4-bit | Vector-quantization codebooks | Calibration set | Targets 2-3 bit regime where AWQ degrades. |
| NF4 (used by QLoRA) | W4A16 | Non-uniform quantile grid | None | Used with bitsandbytes; slower inference kernels in practice. |
The single biggest difference from GPTQ is timing: GPTQ corrects quantization error after it occurs using inverse-Hessian column updates, while AWQ prevents it beforehand by scaling the channels that activations mark as important, using only first-order statistics and no backpropagation. AWQ and GPTQ are the two methods most often compared head-to-head in production-serving contexts. The community comparisons consistently find that:
Hugging Face TGI and the broader Hugging Face Transformers library added support for loading AWQ-quantized checkpoints in late 2023. Users invoke standard AutoModelForCausalLM.from_pretrained on any model with quant_method: "awq" in its config.json; the library inspects the configuration and selects the appropriate AWQ kernel.[8] Supported backends include the original llm-awq, the community AutoAWQ library, and the ExLlamaV2 kernels (selected with AwqConfig(version="exllama"), which works on both NVIDIA and AMD GPUs).[8] Hugging Face also implements fused AWQ modules for LLaMA and Mistral architectures: the Q, K, V, and O projections plus MLP layers are combined into single fused operations, which the documentation reports doubles decode throughput at batch size 1 for benchmark workloads on Mistral-7B-OpenOrca.[8]
AutoAWQ is a community library by Casper Hansen, started in August 2023, that simplified the end-to-end AWQ workflow and extended support to additional architectures beyond the original llm-awq set.[10] AutoAWQ was for several years the most common path for producing community AWQ checkpoints: by mid-2024, over 7,000 AWQ-format models existed on the Hugging Face Hub, many produced by AutoAWQ users.[10] The library is released under the MIT License. The project README now states that AutoAWQ is officially deprecated and no longer maintained, with users directed toward vLLM's llm-compressor package and MLX-LM as recommended successors.[10]
vLLM supports AWQ natively as a recommended INT4 format for production serving.[11] AWQ models can be loaded by passing quantization="awq" to the LLM constructor, with vLLM dispatching the appropriate W4A16 kernel for the host hardware. The llm-compressor package, maintained by the vLLM project, includes an AWQ calibration recipe so that producing a quantized checkpoint and serving it use a coherent workflow.[11]
NVIDIA's TensorRT-LLM supports INT4 AWQ as a native quantization recipe. The NVIDIA TensorRT Model Optimizer ships with an AWQ calibration utility that emits checkpoints consumable directly by TensorRT-LLM. This integration brings AWQ into enterprise serving pipelines including Amazon SageMaker JumpStart, Google Cloud Vertex AI, and NVIDIA NIM microservices.[12]
Production-grade systems that have shipped AWQ support include SGLang, LMDeploy (Shanghai AI Lab), Hugging Face TGI, Intel Neural Compressor, and AMD Quark.[7][8][11] llama.cpp uses its own GGUF-format Q4_K and Q5_K quantizations rather than AWQ; the two ecosystems mostly do not interoperate at the kernel level, though both serve the same practical goal of running 4-bit LLMs on commodity hardware.[7]
By mid-2024, the AWQ ecosystem on the Hugging Face Hub had accumulated over six million downloads of AWQ-format checkpoints.[2] Models for which AWQ-quantized variants are routinely published include:
The MIT HAN Lab maintains a model zoo of pre-computed AWQ search results so that users can reproduce or extend the original quantization configurations without re-running the full search.[7]
The memory reduction from W4A16g128 storage is roughly 3.5x to 3.8x relative to FP16. In rule-of-thumb form:
These figures cover the weight payload only; activation memory and the KV cache remain in FP16 and scale with context length and batch size.[1][2]
Throughput at single-request decoding follows from the memory reduction. Decoding is memory-bandwidth-bound on most accelerators at batch size 1: each generated token requires reading the entire weight matrix from DRAM. Reducing weight precision from 16 bits to 4 bits cuts DRAM traffic by roughly 4x, with the observed end-to-end speedup typically landing in the 2.7x to 3.3x range once dequantization overhead and the unchanged FP16 activation traffic are accounted for.[1][2]
At larger batch sizes, each loaded weight is reused across more tokens, the arithmetic intensity rises, and the relative benefit of weight-only quantization shrinks. AWQ's W4A16 design is therefore most effective for latency-bound single-request inference and for memory-constrained deployments, less so for high-throughput offline batch serving where compute throughput dominates.[1]
AWQ is a widely deployed method, but it has identifiable limitations: