Humming

Humming is an open-source GPU kernel library for matrix multiplication in quantized inference. It supports dense layers and mixture-of-experts workloads, generating specialized kernels through just-in-time compilation.^[1] Ant Group and the vLLM community announced the project on October 21, 2026.^[2] The Python package is named humming-kernels and uses the Apache 2.0 license.^[3]

Role in quantized inference

Humming implements general matrix multiplication, usually abbreviated GEMM, for weights and activations represented at different precisions. Its interface separates the format used to store a model's parameters from the instructions and scheduling used to execute a layer. The library can therefore run several quantization representations through a common execution system.^[1][4]

A quantization algorithm and an inference kernel perform different jobs. For example, the GPTQ research method uses approximate second-order information to choose compressed model weights.^[5] Humming includes an adapter for GPTQ checkpoints alongside adapters for AWQ, AutoRound, compressed-tensors, ModelOpt, Quark, FP8 and MXFP4 representations. The presence of an adapter establishes a loading path for compatible configurations, rather than support for every checkpoint produced by the corresponding tool.^[6]

Quantization representation

The library describes weights with a schema that records their numeric type, scale type and grouping, optional zero points, and optional secondary scales. Channel scales assign a scale to each output channel; group scales divide a channel along the reduction dimension; block scales share a scale across both matrix dimensions. Tensor scaling uses one scale for the weight matrix. The representation also accounts for an expert dimension in MoE layers.^[7]

These distinctions matter even when two checkpoints both contain four-bit weights. Their scale granularity, integer or floating-point encoding, and zero-point representation can require different storage and conversion operations. Humming's schema describes these properties explicitly instead of treating bit width as a complete format specification.^[7]

Weight preparation and execution are separate operations. The HummingLayer interface can load existing quantized tensors or quantize supplied floating-point weights. Its transform() method converts supported source schemas, prepares the layer configuration, and transforms weights and associated parameters for execution. Forward execution requires that transformation to have completed.^[8]

The transformed representation is tied to a target GPU architecture. Moving an original checkpoint between architectures can require preparing its weights again; a transformed layer is required to match its configured SM version.^[4]

Kernel configuration and compilation

Humming divides configuration into three objects. Here, M is the number of input rows, N is the output feature count, and K is the reduction dimension.^[4]

Changing a storage-related layer parameter requires transforming the corresponding tensors again. Compute and tuning choices can vary at invocation time when they remain compatible with the prepared layouts.^[4]

Device and workload selection

The default execution settings come from architecture-specific heuristics. The selector includes separate handling for H20 within SM90, as well as policies for Turing, Ampere, Ada, Hopper and Blackwell devices. It can produce a configuration for a particular M or a collection of configurations covering input-size ranges.^[9]

The common heuristic code estimates where computation begins to dominate weight transfer using the layer's weight size, matrix dimensions, activation type, GPU memory bandwidth and Tensor Core throughput. It also adjusts tiles using the number of experts, estimated work per expert, available SMs and resource constraints. These are rules for choosing configurations; they do not imply that every candidate kernel is benchmarked during ordinary execution.^[10]

MoE input organization is part of the configuration. An indexed path gathers input rows through routing indices. Grouped-contiguous execution places each expert's rows together; grouped-masked execution allocates storage per expert with separate valid-row counts. Changing between these forms requires the corresponding input layout and routing information.^[4]

JIT compilation and caching

Humming supports both NVIDIA's NVRTC runtime compiler and NVCC. HUMMING_COMPILER=nvrtc or HUMMING_COMPILER=nvcc explicitly selects one. The runtime otherwise attempts to use NVRTC on NVIDIA devices, with an NVCC fallback; its PPU path selects NVCC. Multiple kernels can be prepared through a thread pool.^[11]

The compiler constructs a cache key from the compiler identity, options, generated source, kernel expression and included implementation files. A cached CUBIN can then be reused. Compilation uses a file lock to coordinate access, retains source and diagnostic logs, and reports failures with the cache location. The CUDA module is subsequently loaded into the execution context.^[11][12]

This design places kernel preparation in the startup or warmup path when the required specialization is absent from the cache. A warmed kernel timing therefore measures a different part of the system from a first invocation that must compile and load code.^[11][12]

Numeric formats and hardware

Humming's README advertises NVIDIA support beginning at Turing, or SM75. Individual activation paths have stricter requirements. Its documented matrix lists FP16 and INT8 from SM75, BF16 and INT4 from SM80, and FP8 E4M3/E5M2 from SM89.^[1] The launch announcement also reports support for T-Head's CUDA-compatible Zhenwu 810E.^[2]

Compatibility depends on more than the GPU generation and operand width. The implementation checks scale formats, group sizes and available instruction paths. Its configuration code includes native UMMA FP4 paths for supported SM10x/SM11x configurations, in addition to the SM12x block-scaled paths. A broad architecture-support claim does not mean that every supported format uses the same instructions or reaches the same throughput on every device.^[13]

The vLLM adapter can change the activation representation when the requested path is incompatible or disfavored on the device. Its fallback logic considers weight and input schemas, scale groups and architecture, and can return to FP16/BF16 computation. Consequently, the checkpoint's weight precision alone does not establish the precision used for GEMM activations.^[14]

Installation and vLLM integration

The repository documents installation from PyPI or directly from source:^[1]

pip install humming-kernels
pip install git+https://github.com/vllm-project/humming.git

CUDA compilation dependencies can be installed with the extra matching the PyTorch environment's CUDA major version:^[1][3]

pip install "humming-kernels[cu12]"
pip install "humming-kernels[cu13]"

The current package metadata requires Python 3.10 or newer and PyTorch 2.7 or newer. It also declares Triton, NumPy, safetensors, Jinja2, CUDA bindings and supporting utilities; Windows uses a separate Triton dependency. The CUDA extras supply runtime, CCCL, NVCC and NVRTC packages. Specific instruction backends can impose additional compiler requirements.^[3][4]

Backend selection

vLLM exposes Humming separately for dense linear layers and MoE expert computation. The corresponding options are --linear-backend humming and --moe-backend humming. A deployment can request both:^[15]

vllm serve deepseek-ai/DeepSeek-V4.1-Flash-1017 \
    --tensor-parallel-size 8 \
    --linear-backend humming \
    --moe-backend humming

The linear-backend documentation states that layer types without an implementation from the requested backend use automatic selection. Selecting Humming therefore does not establish that every operation in a model runs through Humming.^[15]

Activation quantization is a separate setting. vLLM's quantization configuration has distinct linear and moe specifications, each with weight and activation fields. Its named activation choices include mxfp8, allowing an MoE activation override such as:^[16]

vllm serve openai/gpt-oss-20b \
    --moe-backend humming \
    --quantization-config.moe.activation mxfp8

The separate --quantization humming compatibility route should not be confused with backend selection. For GPT-OSS MXFP4 checkpoints, the current Humming integration explicitly rejects that route and directs users to --moe-backend humming.^[17]

Reported performance

The launch article reports roughly 5x Marlin performance for selected large-batch B300 MoE grouped GEMMs. These are kernel results. Its microbenchmarks use warmed kernels, a cold L2 cache, and M values from 1 to 8192 in powers of two; MoE comparisons combine gate/up and down GEMM latencies.^[2]

Separately, the developers report 14-43% higher end-to-end throughput and 25-32% lower mean time to first token for DeepSeek-V4.1-Flash on eight NVIDIA H20 GPUs with tensor parallelism of eight. MXFP4 weights remained unchanged, but the comparison switched from Marlin W4A16 with BF16 activations to Humming W4A8 with tokenwise FP8 activations.^[2]

That serving comparison changes activation precision as well as the implementation. It does not isolate a backend-only gain at identical precision or establish equal model quality. The reported kernel and serving gains apply to the tested configurations, rather than a universal speedup.^[2]

Precision and deployment considerations

Accumulation precision is distinct from both input and output types. Humming can use FP16 accumulation on eligible paths, but its documentation notes greater rounding error and overflow risk than FP32 accumulation. Its heuristics reject FP16 accumulation for block-scaled layers.^[4][9]

Batch-invariant execution imposes constraints on reduction order and configuration selection, including disabling Stream-K and keeping the same MMA type across batch sizes. Those constraints can reduce performance.^[4] A useful deployment comparison therefore needs to identify the checkpoint representation, activation and accumulation types, hardware, routing layout, batch or concurrency range, and whether compilation is included in the measurement.^[4][11][14]

References

1. ^1 ^2 ^3 ^4 ^5Humming contributors. Humming repository and README. Installation and support matrix. Accessed October 23, 2026.
2. ^1 ^2 ^3 ^4 ^5Ant Group and the vLLM Community. Announcing Humming: Fast, Flexible Kernels for Quantized Inference. October 21, 2026.
3. ^1 ^2 ^3Humming contributors. Package metadata and dependencies, pyproject.toml. Accessed October 23, 2026.
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Humming contributors. Humming configuration guide. Accessed October 23, 2026.
5. ^Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
6. ^Humming contributors. Quantization schema registry. Accessed October 23, 2026.
7. ^1 ^2Humming contributors. Humming weight and input schemas. Accessed October 23, 2026.
8. ^Humming contributors. HummingLayer loading and transformation implementation. Accessed October 23, 2026.
9. ^1 ^2Humming contributors. Architecture-specific heuristic selection. Accessed October 23, 2026.
10. ^Humming contributors. Common device heuristics. Accessed October 23, 2026.
11. ^1 ^2 ^3 ^4Humming contributors. Kernel runtime and compiler selection. Accessed October 23, 2026.
12. ^1 ^2Humming contributors. JIT compiler and cache implementation. Accessed October 23, 2026.
13. ^Humming contributors. Layer and compute configuration implementation. Accessed October 23, 2026.
14. ^1 ^2vLLM contributors. Humming schema mapping and activation fallback. Accessed October 23, 2026.
15. ^1 ^2vLLM contributors. vllm serve command reference. KernelConfig backend options. Accessed October 23, 2026.
16. ^vLLM contributors. Quantization configuration implementation. Accessed October 23, 2026.
17. ^vLLM contributors. Humming quantization integration. Accessed October 23, 2026.