FlashInfer

FlashInfer is an open-source GPU kernel library and code-generation system for large language model inference. It supplies optimized implementations of attention, matrix multiplication, mixture-of-experts computation, sampling, and communication operations. Serving systems can use these kernels as components while retaining their own request schedulers, model runners, and network interfaces.^[1][2]

The project is distinct from FlashAttention. FlashAttention names a family of input/output-aware attention algorithms and implementations. FlashInfer is a broader inference library that can dispatch multiple attention backends and includes operators outside attention. The FlashInfer system paper was published at MLSys 2025, and the source repository uses the Apache License 2.0.^[1][2][3]

Purpose and scope

Inference workloads present kernel problems that are less uniform than many training workloads. A serving batch can contain requests with different prompt lengths, generated lengths, cache layouts, masks, and numbers of query tokens. The shape also changes as requests enter and leave the batch. A single fixed kernel schedule can leave GPU work unevenly distributed or spend time compiling variants that do not match the active workload.^[1]

FlashInfer addresses this layer of the stack. It does not manage the complete lifecycle of an inference service. Projects such as vLLM, SGLang, TensorRT-LLM, and MLC-Engine integrate FlashInfer kernels or interfaces while providing higher-level scheduling and serving functions.^[1][2]

The library's name also does not imply that every operation uses the same implementation. Its current attention interfaces can select among backends based on the requested operation, GPU architecture, data type, head dimension, mask, and other constraints. Unsupported combinations can require another backend or fail validation.^[2][6]

System design

The MLSys paper organizes FlashInfer around three serving problems: different KV cache layouts, customized attention rules, and dynamic request shapes.^[1]

Scroll sideways for more →

Block-sparse cache representation

PagedAttention stores cache blocks non-contiguously so a serving system does not need one contiguous allocation for every request. Prefix trees, page tables, and speculative branches create other non-contiguous access patterns. FlashInfer models these arrangements through block-sparse structures and composable formats instead of maintaining a separate attention implementation for every cache organization.^[1]

The block size controls granularity. Larger query-row blocks can improve shared-memory and register reuse among requests placed in the same block, but can increase fragmentation. Smaller blocks support finer-grained sparsity but give up some of that cross-request reuse. Composable formats allow more than one granularity in the same logical operation. This is an implementation abstraction, not compression of the model's semantic context.^[1]

Custom attention templates

Modern models vary the calculation around the query-key product. They may use grouped-query heads, sliding windows, attention sinks, logit soft caps, custom masks, or multi-head latent attention. FlashInfer exposes a template that specializes these choices and generates CUDA code for a requested variant.^[1][2]

Just-in-time compilation avoids shipping every possible combination as one binary, but it moves some work to installation or first use. The project therefore distributes a core Python package plus optional packages containing precompiled binaries and a prebuilt JIT cache. The exact artifact selected depends on the FlashInfer, Python, CUDA, and GPU versions.^[2][5]

Plan and run

Several batched APIs separate plan() from run(). Planning inspects sequence lengths and cache metadata, selects a schedule, and prepares workspace information. Running applies that plan to tensors with the promised shapes and layouts. Reusing a plan avoids repeating all host-side work when the relevant geometry stays unchanged.^[1][6]

This split also supports CUDA Graph capture. A graph needs stable addresses and a repeatable launch structure, while live request lengths remain dynamic. FlashInfer can keep workspace buffers stable and pass updated scheduling data to a captured run. This does not make every dynamic Python call graph-capturable; the guarantee belongs to APIs and configurations that explicitly document graph support.^[1][6]

Attention states

FlashInfer's recursive-attention interface represents attention over part of a cache with an output state and a log-sum-exp normalization state. Merge operations combine states from disjoint cache regions into the result that their union would produce. A serving system can therefore evaluate a shared prefix separately from request-specific suffixes, or split a long cache across GPU work units, then merge the partial results.^[7]

The merge is mathematically useful because softmax normalization is global to a row. Partial outputs cannot be averaged directly. Their log-sum-exp values supply the rescaling needed to combine them under a common normalization constant.^[7]

Operations and hardware coverage

By August 28, 2026, the official documentation was labelled version 0.6.18. It described a library that had expanded beyond its original attention engine.^[2][4]

Scroll sideways for more →

These rows describe the project as a whole, not a compatibility promise for one machine. The current repository lists NVIDIA GPU targets from Turing through several Blackwell compute capabilities and warns that feature support differs across them.^[2]

The version 0.6.18 installation guide lists Linux, Python 3.10 through 3.14, and CUDA 12.9 or 13.0 for its regular supported wheel paths. It also lists CUDA 13.4 wheels built with a preview toolkit and PyTorch nightly, while stating that runtime continuous integration covers CUDA 12.9 and 13.0.^[5] Those requirements change across releases. Installation instructions should be matched to the intended FlashInfer version rather than copied from an undated example.

Evaluation

The peer-reviewed paper evaluates FlashInfer v0.2, not the 2026 library. Its experiments used NVIDIA A100 40 GB SXM and H100 80 GB SXM GPUs with CUDA 12.4, PyTorch 2.4.0, and FP16 storage and computation.^[1]

Scroll sideways for more →

The serving experiments used specific models, request distributions, arrival-rate constraints, and backend versions. They show that the paper's design improved those tested configurations. They do not establish a fixed speedup for later releases, every model, or every GPU. Backend projects also change independently, so a current deployment needs matched-version benchmarks with its own prompt and generation-length distributions.^[1]

Correctness and numerical behavior

FlashAttention is designed as an exact-attention algorithm rather than an approximation, but GPU implementations use finite-precision values and change the order of operations. Bitwise agreement with a materialized reference is therefore not guaranteed. A 2024 study of FlashAttention found roughly an order of magnitude more numerical deviation than baseline attention in its isolated BF16 forward-pass comparison. Its weight-space analysis estimated that deviation to be 2-5 times smaller than the effect of low-precision training in the studied setup.^[11] That work concerns general rounding behavior, not the separate implementation defect described below.

Extreme-negative-logit correction

On August 20, 2026, the FlashInfer maintainers merged pull request 4401 into the repository's main branch. The affected FlashAttention-2 paths had used the finite value 50,000 as the magnitude of an infinity sentinel. A valid raw query-key dot product below -50,000 could then fail to replace the online softmax's initialized running maximum, causing the valid row to return a zero output with a sentinel log-sum-exp value.^[9]

Replacing the sentinel with IEEE negative infinity exposed a second edge case. A fully masked tile can subtract negative infinity from negative infinity, yielding NaN before exponentiation. The merged patch clamps those differences before the exponential update and defines a fully masked row as zero output with negative-infinity log-sum-exp. It applies the correction across listed FlashAttention-2 prefill and decode paths, split-KV state merging, and related MLA sites, with deterministic regression tests added for the reported cases.^[9]

The pull request reports testing on an RTX 4090 and compile verification for sm_90a; its author did not have an SM90 GPU for that part of the test. The pull request also notes that the Hopper FlashAttention-3 path was left unchanged because it needed separate empty-row handling.^[9] The verified publication state is that commit d7f2c64 entered main. The pull request alone does not identify the first stable package release that contains it.

Josh Tobin of Recursive later attributed discovery of the finite-sentinel edge case to an automated-research workflow that used a reward-hacking judge while optimizing inference kernels.^[10] The public pull request independently documents the defect, review, tests, and merged correction, but it does not document that discovery provenance.

Machine-generated kernel workflow

FlashInfer-Bench is a related January 2026 preprint about evaluating and deploying GPU kernels produced by language-model agents. Its FlashInfer Trace schema records a kernel's contract, workloads, implementation, and evaluation. The benchmark uses workloads derived from serving traces, checks correctness and speed, and describes process isolation intended to stop candidates from manipulating performance measurements.^[8]

The preprint's apply() mechanism substitutes a validated implementation into an inference engine such as SGLang or vLLM. This closes part of the gap between a benchmark submission and an engine integration, but validation remains tied to the recorded operation and workload. A passing generated kernel is not automatically correct for untested shapes, data types, masks, or hardware.^[8]

Deployment considerations

FlashInfer is a component inside a larger software and hardware stack. Reproducible use requires pinning the library release or commit, optional binary and JIT-cache packages, PyTorch build, CUDA runtime, GPU architecture, and serving-engine version. It also requires confirming that the requested combination of head dimensions, data types, cache layouts, masks, and graph capture appears in the matching documentation.^[2][5][6]

Performance tests should separate prefill from decode and record batch size, query lengths, cache lengths, concurrency, and warmup state. Kernel microbenchmarks can reveal scheduling and bandwidth behavior, but only end-to-end tests show whether compilation, request scheduling, cache management, and communication erase or amplify that difference.

Correctness tests need adversarial cases as well as ordinary random tensors. Fully masked rows, empty cache partitions, extreme logits, short ragged requests, unusual head ratios, and boundary page lengths can reach branches that typical throughput benchmarks rarely exercise. The August 2026 correction is a concrete example: the affected paths could produce silent zero outputs or non-finite values only in particular numerical and masking conditions.^[9]

References

1. ^Zihao Ye et al. "FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving." Proceedings of Machine Learning and Systems 7, MLSys 2025. proceedings.mlsys.org/...8ab3a-Abstract-Conference
2. ^FlashInfer Project. "High-Performance GPU Kernels for Inference." Official repository, accessed August 28, 2026. github.com/...flashinfer
3. ^FlashInfer Project. "Apache License, Version 2.0." Official repository license, accessed August 28, 2026. github.com/...LICENSE
4. ^FlashInfer Project. "FlashInfer 0.6.18 documentation." Accessed August 28, 2026. docs.flashinfer.ai
5. ^FlashInfer Project. "Installation." FlashInfer 0.6.18 documentation, accessed August 28, 2026. docs.flashinfer.ai/installation
6. ^FlashInfer Project. "FlashInfer Attention Kernels." FlashInfer 0.6.18 documentation, accessed August 28, 2026. docs.flashinfer.ai/...attention
7. ^FlashInfer Project. "Attention States and Recursive Attention." FlashInfer 0.6.18 documentation, accessed August 28, 2026. docs.flashinfer.ai/...recursive_attention
8. ^Shanli Xing et al. "FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems." arXiv:2601.00227v1, January 1, 2026. arxiv.org/...2601.00227
9. ^FlashInfer Project. "fix(attention): handle extreme negative logits in masked softmax." Pull request 4401, merged August 20, 2026. github.com/...4401
10. ^Josh Tobin. "A fun small win for automated research." X post, August 27, 2026. x.com/...2093107857793462678
11. ^Alicia Golden et al. "Is Flash Attention Stable?" arXiv:2405.02803, May 5, 2024. arxiv.org/...2405.02803