KV Cache

KV cache, short for key-value cache, is transient model state used during Transformer generation. At each attention layer, it retains the key and value vectors already computed for prompt tokens and earlier output tokens. An autoregressive model can then process only the newest token instead of running every earlier token through the full model again.^[1][2]

The cache is central to Transformer-based large language model inference, but it is not a cache of text answers or database records. It contains model-specific tensors. Their shape, numerical format, positional treatment, and allocation layout depend on the model and runtime. A cache created for one set of weights, adapter parameters, or token positions is not generally valid for another.^[2][3]

KV caching removes repeated computation for the old tokens' projections and feed-forward layers. It does not make dense attention independent of context length: each new query must still attend to the retained keys and values. Cache memory therefore grows linearly with the number of stored tokens, and the amount read during each dense decode step grows with the active context.^[3][4]

That cost scales with every deployed request, so cache size per token has become a published design target rather than an implementation detail. Vendors now chart it across their own model generations: DeepSeek's figure for its own releases runs from 389,120 bytes per token in November 2023 to 890 in September 2026, a 437-fold reduction by the company's own measurement of a specific part of its cache.^[33]

How it works

Causal self-attention

In a causal self-attention layer, hidden state x_t at position t is projected into a query q_t, key k_t, and value v_t. The query is compared with keys at allowed positions, and the resulting attention weights combine their values. Causal masking prevents position t from using a later position. The original Transformer established this attention structure, although it did not define a standard runtime cache interface.^[1]

During incremental generation, keys and values for earlier positions do not change when a later token is appended. A runtime can retain those tensors, compute only q_t, k_t, and v_t for the new position at each layer, append the new key and value, and evaluate attention over the accumulated state. Implementations often expose this state as past_key_values, a cache object, or an engine-managed block pool.^[2][3]

This article primarily describes decoder-only causal self-attention. Encoder-decoder models can maintain separate decoder self-attention state and cross-attention state derived from the encoder output. Models without attention may use recurrent or state-space caches instead of key-value tensors.^[2]

Caching is exact only when the retained tensors have the same numerical meaning as recomputation would produce. Lower-precision storage, token eviction, cross-layer merging, or approximate retrieval can save memory but may change attention results. Cache allocation and reuse are therefore separate questions from cache compression.^[15][17][20]

Prefill and decode

An inference request usually has two phases. During prefill, the model processes the prompt and creates key-value state for its tokens. Prompt tokens are available together, so their matrix operations can be parallelized, although full causal attention still has quadratic arithmetic in prompt length. During decode, the model produces one or more new tokens sequentially and extends the cache after each accepted token.^[4][21]

The distinction matters because the phases stress hardware differently. Prefill can expose substantial matrix-multiplication parallelism. Decode has a sequential dependency between output tokens, and at low or moderate batch sizes it often spends substantial time moving weights and KV state from high-bandwidth memory. Whether weights or cache traffic dominates depends on model size, batch size, sequence length, data type, parallel layout, and hardware.^[3][4]

What computation the cache saves

Without a cache, producing a new token by rerunning the entire growing prefix would repeat the old tokens' projections, attention layers, and feed-forward layers. With a cache, those old key and value projections are reused. Holding model dimensions fixed, dense attention for one new token still reads and scores a number of cached positions proportional to the current sequence length.^[3][4]

For a prompt of P tokens followed by N generated tokens, cached dense decode attention has sequence-length work proportional to N x P + N x (N - 1) / 2 at each layer. The non-attention work for old positions is not repeated. This is why describing cached decode as simply O(1) is misleading: the newly projected token is constant in count, but the dense attention span is not.^[1][3]

Memory accounting

For a decoder-only model with the same attention shape in every layer, the raw KV tensor payload can be estimated as:^[3][4][11]

payload bytes = 2 x sum(T_i) x L x H_KV x D_head x S

Here, T_i is the stored length of active sequence i, L is the number of cache-producing layers, H_KV is the number of key-value heads, D_head is the dimension of each head, and S is bytes per stored element. The factor of two represents keys and values. For a uniform batch, sum(T_i) becomes batch size times stored tokens per sequence.^[3][4]

This formula describes tensor payload, not necessarily reserved device memory. Real runtimes may also hold block tables, alignment padding, quantization scales and zero points, unquantized residual windows, beam or speculative branches, allocator slack, and duplicated state caused by sharding or replication. Conversely, exact-prefix sharing can let several requests reference the same physical blocks.^[9][13][15]

Corrected Llama 2 13B example

Llama 2 13B has a 4,096-token training context and does not use grouped-query attention. Its inherited LLaMA architecture has 40 layers, hidden dimension 5,120, and 40 attention heads, giving a head dimension of 128. Under the explicit assumptions of multi-head attention, one sequence, no prefix sharing, and two-byte key and value elements, the calculation is 2 x 40 x 40 x 128 x 2 bytes per token.^[5][30]

These are binary units and exclude model weights, activations, allocator overhead, and runtime metadata. A quantized cache or a model with fewer key-value heads would use a different amount. Extending the arithmetic past 4,096 tokens would be a hypothetical memory calculation, not evidence that the unmodified model reliably supports a longer context window.^[5][15]

Architectural ways to reduce the cache

Multi-query and grouped-query attention

In standard multi-head attention, every query head has its own key and value head, so H_KV equals the number of query heads. Multi-query attention uses one shared key head and one shared value head. This can reduce raw cache payload and key-value bandwidth by the number of query heads, but it also changes the model architecture and removes a head axis that systems might otherwise shard.^[3][4]

Grouped-query attention uses more than one key-value head but fewer key-value heads than query heads. It creates an intermediate memory and quality tradeoff between multi-head and multi-query attention. The original GQA study converted T5 checkpoints and found quality close to multi-head attention with speed comparable to multi-query attention in its tested tasks. That result is evidence for the study protocol, not a guarantee for every model or workload.^[6]

Because cache payload is proportional to H_KV, reducing 32 key-value heads to 8 gives a nominal fourfold payload reduction when all other terms are equal. The realized serving gain can be smaller or larger because attention kernels, batch capacity, communication, and weight traffic also affect latency and throughput.^[4][6]

Latent and local attention

Multi-head latent attention stores a compressed latent representation from which attention keys and values are derived. DeepSeek-V2 reported a 93.3 percent KV-cache reduction relative to DeepSeek 67B and attributed part of its higher generation throughput to this architecture. Both figures are model-relative results from the DeepSeek-V2 evaluation, not general reduction factors for latent attention.^[8]

Sliding-window attention limits each layer to a fixed span of recent positions, which bounds that layer's cache once the window is full. Mistral 7B combined sliding-window attention with grouped-query attention. A bounded local cache does not preserve direct access to every old token, and support must be part of the model's attention design. Discarding old state from a model trained for full attention is not equivalent to running that model with its full context.^[7]

Hybrid models may alternate local and global attention layers, use different window sizes by layer, or maintain recurrent state instead of a KV cache in some layers. Their memory must be calculated per layer rather than by applying one head count and one window to the whole network.^[2][7]

Cross-layer key-value sharing

Head sharing shrinks the cache along the head axis and a bounded window shrinks it along the token axis. A third axis is depth. Cross-Layer Attention ties key and value heads across adjacent layers so that several layers read one stored copy instead of each keeping its own. Its authors trained 1B- and 3B-parameter models from scratch and reported roughly a further twofold cache reduction on top of multi-query attention at close to unmodified multi-query accuracy. They present this as a Pareto improvement on the memory and accuracy tradeoff available from head sharing alone, measured on their own models, not as a free saving.^[31]

Character.AI described a production version of the same combination in June 2024. Its engineering post says the company used multi-query attention in every attention layer, interleaved local and global attention so that only one layer in six was global and the local span was 1,024 tokens, and tied key-value state across neighbouring layers, with multiple global layers tied together across blocks because global layers dominate the cache at long context. Character.AI reported that the combination cut KV cache size by more than 20 times without a quality regression it could measure on its evaluations, that it also trained and served in int8 including the cache, and that its serving cost had fallen by a factor of 33 since late 2022. These are the company's own figures for an unpublished model on its own stack.^[32]

Trained layer sharing is a different operation from merging the layers of a finished checkpoint. The former fixes the sharing pattern before training and the model learns around it; the latter, discussed below, approximates state that the model was trained to keep separate.^[20][31]

Runtime allocation and scheduling

Dynamic, static, and paged storage

A dynamic contiguous cache grows with the sequence. It avoids reserving a full maximum length at request admission, but repeated growth or relocation can be costly. A static cache reserves a maximum shape in advance, which can work well with compiled graphs but may waste memory when requests end early or have varied lengths. Offloaded caches move some layer state to host memory, trading device capacity for transfer traffic and often lower throughput.^[2]

PagedAttention, introduced with vLLM, divides a request's logical cache into fixed-size blocks that can map to noncontiguous physical memory. Blocks are allocated as a sequence grows and reclaimed when it ends. Copy-on-write sharing can let parallel samples or common prefixes share blocks until their tokens diverge. In the SOSP 2023 evaluation, vLLM reported two to four times the throughput of its comparison systems at similar latency; that range belongs to the paper's models, hardware, and workloads.^[9]

Paged storage reduces internal waste and external fragmentation, but it requires block tables and attention kernels that can follow the page mapping. vAttention proposed a different design: reserve a contiguous virtual range and map physical GPU memory into it on demand with CUDA virtual-memory mechanisms. Its ASPLOS 2025 paper reported up to 1.23 times higher end-to-end serving throughput than tested PagedAttention-based kernels in long-context experiments. This is a comparison of particular implementations, not proof that either allocation design is always faster.^[11]

Iteration-level scheduling

Cache lifetime is tied to request scheduling. A fixed batch can leave finished requests occupying slots while shorter requests wait. Orca instead scheduled at iteration granularity and applied batching selectively, allowing the active set to change between generation iterations. Modern continuous batching follows the same broad goal: admit, advance, pause, or retire requests while promptly assigning and reclaiming their cache state.^[10]

Iteration-level scheduling introduces tradeoffs among batch efficiency, head-of-line blocking, preemption cost, and available cache memory. A preempted request may retain its blocks, move them to slower memory, or discard them and later recompute its prefix. The best choice depends on transfer bandwidth, expected reuse, request priority, and service-level objectives.^[9][13]

Prefix reuse and prompt caching

Two requests with an identical token prefix can reuse that prefix's KV state because causal attention for the prefix does not depend on later suffix tokens. Correct reuse requires more than matching visible text: token IDs, model weights, adapters, positional treatment, cache format, and relevant preprocessing must agree. The reusable range ends at the first incompatible block or token.^[12][13]

RadixAttention in SGLang organizes reusable prefixes in a radix tree, while block-based runtimes can hash or index completed prefix blocks. Reuse can reduce prefill computation and physical memory when multiple requests share system instructions, documents, or conversation history. It does not reduce the decode attention span over the reused prefix, because new queries still attend to that state.^[12][13]

Prompt caching is also the name of hosted API features. Such a product may expose cached-token accounting, lower latency, or different billing while keeping its internal representation private. An API-level cache hit should not be described as a user-portable KV tensor unless the provider documents that interface. Mutable price tables, model catalogs, thresholds, and retention periods are product documentation rather than stable properties of KV caching.^[14]

Cross-request reuse must respect isolation boundaries. TensorRT-LLM 1.1.0 documents a cache-salt mechanism that restricts reuse to requests with the same salt, specifically to reduce prompt-theft risk. A serving system should also prevent cache state from surviving longer than policy allows or being reused after a model, adapter, or authorization context changes.^[13]

Reuse has become a visible part of hosted pricing, which gives the cache a second economic role beyond capacity. DeepSeek's published rate card for its deepseek-flash endpoint bills cache-hit input at 0.003 dollars per million tokens off-peak and 0.006 at peak, against 0.15 and 0.30 for cache-miss input, a fifty-to-one ratio between a reused prefix and a freshly prefilled one.^[43] DeepSeek's own argument for compressing the cache is framed in those terms: "Cache-hit charges often account for a large share of agent costs. Compressing the cache cuts those costs significantly."^[40] Character.AI reported a 95 percent hit rate from an inter-turn host-memory cache on dialogues averaging 180 messages of history.^[32] Hit rates and price ratios of that kind are properties of a workload and a price list rather than of KV caching, and both can change without any change to the model.

Compression and bounded caches

Quantization

KV-cache quantization stores keys and values with fewer bits. Moving from a two-byte element to a four-bit payload has a nominal fourfold tensor reduction, but the actual ratio is lower when scales, zero points, alignment, outlier channels, and a higher-precision residual region are included. Quantization also adds conversion work and may change model output.^[15][16]

KIVI found different outlier structure in keys and values and used per-channel quantization for keys and per-token quantization for values. Its two-bit method reported 2.6 times lower peak memory including model weights and 2.35 to 3.47 times higher throughput on its tested Llama 2, Falcon, and Mistral workloads. Those results do not mean every two-bit cache preserves quality or achieves the same system gain.^[15]

KVQuant combined per-channel key quantization, pre-RoPE key handling, nonuniform data types, dense-and-sparse outlier separation, and a full-precision residual window. Its design illustrates why a bare bit width is not a complete cache specification. The model, calibration data, context length, residual policy, and dequantization kernel all affect accuracy and performance.^[16]

Eviction, selection, and cross-layer compression

Eviction methods retain only a subset of token positions. H2O keeps a balance of recent tokens and attention heavy hitters. StreamingLLM keeps a small set of initial attention-sink tokens together with a recent window. SnapKV uses an observation window near the end of the prompt to select clustered prompt positions for each attention head. These methods use different signals and were evaluated on different models and tasks.^[17][18][19]

StreamingLLM's "infinite" framing refers to stable streaming generation with bounded state in its tests. It does not give the model lossless retrieval of every discarded token. Likewise, SnapKV's reported long-context results do not establish that its selected positions preserve every fact, ordering relation, or future query target.^[18][19]

MiniCache compresses along model depth by merging selected key-value states from adjacent layers while retaining unusually dissimilar pairs. This can complement token selection or quantization, but it changes the stored representation and depends on cross-layer similarity in the evaluated model.^[20]

No compression ratio is a universal quality guarantee. An MLSys 2025 study found that some compression implementations did not translate memory savings into production-level throughput, could lengthen generated answers, and hid per-sample failures behind aggregate scores. A 2026 reasoning-workload study further found that token-wise eviction could become unstable in multi-batch settings and could provoke longer reasoning sequences. These findings support evaluating complete outputs and systems, not cache size alone.^[27][28]

Per-token cache size as a published specification

Divide the payload formula above by batch size and stored length, and what remains is bytes of cache per token. That number is no longer something a reader derives from a configuration file. Vendors publish it, chart it across their own releases, and design against it.

It is not one measurement, though, and the differences matter more than the headline. A model with several attention branches can report only the branch that dominates at long context instead of the total. A serving stack can report what it holds in accelerator memory separately from what it writes to storage for later reuse. A vendor can quote a ratio against its own previous system rather than against a named public model. Two per-token numbers are comparable only when both state the same scope, the same stored precision, and the same model configuration.

Two reference points from the 2023 generation set the scale. The PagedAttention paper calculated that a single cached token of the 13-billion-parameter OPT model needs 800 KB, as 2 x 5,120 x 40 x 2 bytes, and that on a 40 GB A100 roughly 65 percent of memory held weights while close to 30 percent held KV cache.^[9] The Llama 2 13B calculation earlier in this article arrives at the same 819,200 bytes for the same reasons: forty layers of full multi-head attention at two bytes per stored element.^[5]

DeepSeek's published series

DeepSeek has published the longest public series of the measurement. The DeepSeek-V4.1-Flash technical report and model card carry a figure titled "Global KV Cache Per Token (Bytes)", captioned as "global KV cache size per token (in bytes) across generations of DeepSeek models".^[33][34] It has four bars with date labels and three annotated step ratios, and the caption states that V4.1-Flash "achieves approximately 4-fold and 437-fold reductions in per-token global KV cache size relative to DeepSeek-V4-Flash and DeepSeek-V1, respectively".^[33]

Every value in that table is DeepSeek measuring DeepSeek's own models in DeepSeek's own serving stack. None of it has been independently reproduced. The ratios DeepSeek itself states are the three adjacent steps and the 437-fold total; a ratio between any other pair of bars is the reader's own arithmetic rather than a published figure.

The date labels track DeepSeek's release announcements. DeepSeek LLM, the company's first model family, was published on November 29, 2023.^[36] The "2025.12" label refers to the official DeepSeek-V3.2 release of December 1, 2025, which DeepSeek announced as the "Official successor to V3.2-Exp", and not to the experimental precursor DeepSeek-V3.2-Exp of September 29, 2025.^[37][38] DeepSeek V4 entered preview on April 24, 2026, and DeepSeek V4.1-Flash was announced on September 10, 2026.^[39][40]

Global cache and persistent cache are different quantities

The report defines the charted quantity narrowly. DeepSeek-V4 pairs a global attention branch spanning the whole context with local sliding-window attention in every layer. The global branch holds "global KV, comprising main KV and indexer K", while the local branch holds state whose size is bounded by the window and therefore stops growing with sequence length. DeepSeek's stated reason for charting the global branch alone is that "for sufficiently long sequences, global KV therefore dominates the runtime KV footprint, which is constrained by HBM capacity".^[33] The chart is a measure of accelerator-resident cache at long context. It is not total KV memory at every context length, and it is not the state a stack persists to storage.

That persisted state carries a separate ratio, and the two are easy to merge by accident. DeepSeek reports the global cache of V4.1-Flash at roughly one quarter of DeepSeek V4-Flash, and separately reports the persistent KV cache, the state kept on SSD or in host memory for prefix reuse, at roughly one eighth. Its release note states the pair as "1/4 the HBM" and "1/8 the SSD storage".^[40] The report factors the eighth explicitly: "the persistent KV cache no longer stores SWA KV, which almost halves its size, and the global KV retained in it is further compressed to 1/4 of V4's footprint".^[33] A claim that the model cut its KV cache fourfold, or eightfold, says nothing until it names which of the two caches it means.

Which mechanism produced which step

Only the last step of the chart has a published attribution to named mechanisms. The earlier steps each bundle several changes.

The first step spans three model generations and about two years. DeepSeek LLM 67B is documented as 95 layers, model dimension 8,192, 64 query heads and 8 key-value heads, using grouped-query attention, and its released checkpoint stores bfloat16.^[35][36] Applying the payload formula above to those numbers gives 2 x 95 x 8 x 128 x 2 = 389,120 bytes per token, which is the chart's first bar exactly; the report does not say which checkpoint it means by "DeepSeek-V1". Between that model and V3.2, DeepSeek introduced multi-head latent attention in DeepSeek-V2, rebuilt the model at V3, and added DeepSeek Sparse Attention at V3.2. The V2 paper credits latent attention with the cache reduction, reporting 93.3 percent against DeepSeek 67B.^[8] DeepSeek's announcement for sparse attention describes it instead as achieving "fine-grained sparse attention with minimal impact on output quality" while "boosting long-context performance & reducing compute cost".^[38] Assigning the whole 8.1x to any one of those releases would misstate what changed.

The second step, from V3.2 to V4-Flash, came with DeepSeek's ground-up V4 architecture. The V4 paper's headline attention change is "a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency", alongside a new residual scheme and a change of optimizer. The paper reports that at a one-million-token context DeepSeek-V4-Pro needs 27 percent of the per-token inference FLOPs and 10 percent of the KV cache of DeepSeek-V3.2; that published ratio is for the Pro model, while the chart's bar is for Flash.^[41]

The third step is the one DeepSeek attributes directly: "Together, CSA2 and FP4 KV caching reduce global KV cache storage to approximately 1/4 of that of DeepSeek-V4-Flash."^[33] Compressed Sparse Attention 2 assigns every attention layer one of three static modes, so that layers reuse a preceding layer's main KV and indexer keys instead of each storing a copy, which is cross-layer sharing applied to the global branch. The precision change stores that cache in an E2M1 FP4 format with one E4M3 scale per sixteen channels, and DeepSeek says it "nearly halves the storage footprint" against the FP8 main KV cache of V4. Read together, those two statements put about half of the fourfold step on precision and the rest on layer sharing. The same release also introduced the Causal Encoder-Decoder layout, in which decoder global KV is projected from the final encoder hidden state and the model activates 8 billion parameters per token during prefill against 16 billion during decode. DeepSeek presents that layout mainly as a cost-of-compute change, and the sentences that attribute the 890-byte figure name CSA2 and FP4 caching.^[33][34]

Comparable claims from other vendors

The design target is general, but each vendor reports it in its own units.

Google's Gemma 3 report says the architecture was changed "to reduce the KV-cache memory that tends to explode with long context" by raising the ratio of local to global attention layers and keeping the local span short. Its ablation on a 2B model with a 32K-token prefill reports that a global-only configuration spends about 60 percent of inference memory on KV cache, and that a 1:3 local-to-global ratio with a 1,024-token sliding window brings that under 15 percent. The shipped Gemma 3 models use a 5:1 ratio with the same window.^[42] That is a share of inference memory, not bytes per token, and converting between the two requires the model size and context length the share was measured at.

Character.AI's figure, above, is a third form again: a ratio against the company's own earlier serving stack rather than against any named public model.^[32] The Cross-Layer Attention result is a fourth: a controlled ablation on models the authors trained themselves for the comparison.^[31]

Three cautions apply to all of them. They measure different quantities. With the partial exception of the academic ablations, they are vendor self-measurements of vendor systems rather than independent evaluations. And a smaller cache is not on its own a better system: the MLSys results above found compression that did not convert into production throughput and eviction that lengthened generated output, so a per-token number belongs next to end-to-end serving and quality measurements rather than in place of them.^[27][28]

Distributed and hierarchical caches

Disaggregated serving can place prefill and decode on different workers. Prefill workers create KV state, which must then move to decode workers. DistServe chose separate GPU allocations and parallel plans for the two phases and placed them with network bandwidth in mind. Its final OSDI 2024 evaluation reported either 7.4 times more requests served or a 12.6 times tighter service-level objective than comparison systems while meeting latency constraints for more than 90 percent of requests.^[21]

Those figures correct earlier prepublication values and remain protocol-specific. Disaggregation is beneficial only when reduced phase interference and specialized resource allocation outweigh cache-transfer and coordination costs. The cache can be large enough that moving it over a slow interconnect is more expensive than recomputing part of the prompt.^[21][23]

Mooncake extended the idea into a distributed cache spanning GPU memory, CPU memory, SSD, and network resources. Its scheduler seeks reusable prefixes, streams newly created state from prefill to decode workers, replicates hot blocks, and moves colder blocks down the hierarchy. The FAST 2025 paper documents a production-derived design and evaluation, but its deployment scale and capacity gains are authors' reported results rather than independent measurements of every installation.^[22]

CacheGen addressed network movement by encoding KV tensors before transfer and choosing among loading, compression levels, and recomputation according to bandwidth. This is a different problem from shrinking the active decode cache on one GPU: a representation optimized for transport may be decoded into a larger active cache at the destination.^[23]

A persistent tier has its own retention policy and its own failure mode. DeepSeek's V4.1-Flash report describes a persistent KV cache on SSD sized so that entries stay resident for more than 72 hours under its typical workloads, under LRU eviction, and notes that in the previous generation the sliding-window state accounted for nearly half of that capacity while being reused only inside an active session. Its stated remedy, SWA Bounded Replay, stops persisting the local state and rebuilds it on a miss by replaying only the most recent window of tokens rather than the window multiplied by the layer count. DeepSeek says the rebuilt state is approximate by design and reports negligible quality loss in its own testing, while also naming cache-resumption boundaries as an area needing more stress testing.^[33] The general point survives the specific design: what a system persists, for how long, and whether it can cheaply recompute what it dropped are policy choices that change the storage bill without changing the model.

Full prefill-decode separation is not automatically best on commodity clusters. The OSDI 2026 EcoServe study identified interconnect dependence as a limitation and evaluated a partially disaggregated schedule on Ethernet-connected GPUs. Its result reinforces that topology, cache-transfer volume, and latency targets must be included when comparing serving architectures.^[29]

Offloading and prefetch

Offloading moves the KV cache out of GPU memory into host memory or storage while the model keeps running on the GPU, so each decode step must fetch whatever state it needs back across PCI Express, whose bandwidth is far below that of GPU memory. Systems built around KV cache offloading therefore either fetch less, by combining offloading with sparse attention so that only the entries a step will read are moved, or fetch earlier, so that the transfer overlaps computation. InfiniGen (OSDI 2024) took the second route for dense models: it speculates which entries the next attention layer will need by performing a minimal rehearsal with the current layer's inputs and part of the next layer's query weights and key cache, then prefetches only those entries. Its authors report up to 3.00 times better overall performance than prior KV cache management methods in an offloading-based system.^[44]

NOSA (2025) argued that trainable sparse attention is a natural basis for offloading because its block selections show strong locality from step to step, but that unconstrained selections can still force large CPU-to-GPU transfers. It splits selection into a query-aware part and a query-agnostic part and applies an eviction policy over the query-agnostic part to bound the number of blocks fetched per step. With its companion inference system, NOSI, the authors report decoding throughput on 1B, 3B and 8B models up to 5.04 times that of full attention, 1.92 times InfLLM-V2 and 1.83 times ShadowKV.^[45]

SparDA (NVIDIA, June 2026) addresses the same transfer bottleneck by making the selection predictable one layer early. A fourth per-layer projection, the Forecast, chooses the next layer's KV blocks, so the runtime can issue the CPU-to-GPU copy for layer l+1 while layer l is still executing, using a persistent Triton kernel based on Unified Virtual Addressing that keeps a fixed set of thread blocks streaming transfers alongside the compute kernels. On an H100 at 128K context the authors report up to 1.69 times the decode throughput of the sparse-attention offload baseline on MiniCPM4.1-8B (1.40 times on NOSA-8B), with the largest gains at batch sizes 8 to 16, where prefetch and layer execution are roughly balanced. Against the non-offload sparse baseline, which could not run beyond batch 16 at 32K or beyond batch 4 at 128K, offloading let SparDA run larger batches and reach up to 5.28 times higher decode throughput. These are the authors' measurements inside one serving engine, and the 5.28 figure compares runs at different batch sizes.^[46] Prefetching does not remove the storage bill: whatever is offloaded still has to be held somewhere, and the retention and recomputation questions above apply to it.

Relation to attention kernels

Flash Attention reduces reads and writes of attention intermediates by tiling exact attention around the GPU memory hierarchy. It avoids materializing the full attention-score matrix in high-bandwidth memory. This is distinct from the persistent KV cache: incremental decode still needs retained keys and values unless the model uses another state mechanism or recomputes them.^[24]

FlashAttention-2 improved work partitioning and parallelism over the first algorithm. Kernel speedups do not directly equal end-to-end serving speedups because model weights, KV allocation, cache layout, scheduling, communication, and sampling remain. A paged cache may also require a paged-aware kernel rather than the contiguous layout assumed by another implementation.^[11][25]

Implementation details

Cache layouts commonly arrange dimensions corresponding to batch or request, token position, key-value head, and head dimension, but the physical order varies by framework and kernel. Some block pools pack several layers together; others keep a tensor per layer. A serialized cache is therefore not portable merely because two models have the same number of heads.^[2][13]

Position handling must remain consistent. With rotary position embeddings, a runtime may store keys before or after applying the positional rotation, depending on how its kernels are fused. Reusing a prefix at a different absolute position or under a different RoPE-scaling configuration can be invalid even when token IDs match. The same caution applies to attention masks and model-specific cache transformations.^[8][16]

Branching generation can share a prefix cache until branches diverge. Beam search, parallel sampling, and speculative decoding may then allocate additional blocks, discard rejected speculative state, or copy-on-write shared blocks. Capacity planning should use the number of live branches and accepted tokens, not only the final answer length.^[9]

With tensor parallelism, each worker may store a shard of the cache, replicate key-value heads, or combine sharding and replication. A model with eight key-value heads is not inherently limited to eight GPUs: systems can partition other dimensions or replicate heads when the parallel degree exceeds the head count. The choice changes memory, collective communication, and kernel efficiency.^[4]

Evaluation

A useful cache evaluation reports the full workload and separates memory from service quality. At minimum it should specify model checkpoint, cache format, prompt and output-length distributions, batch or arrival process, hardware, parallel layout, attention kernel, prefix-hit rate, and whether reported memory is tensor payload, allocated memory, or peak process memory.^[9][27]

Shared-context workloads need more than a single-request benchmark. SCBench evaluates cache generation, compression, retrieval, and loading across repeated-context tasks and found that some sublinear-memory methods degraded in multi-turn settings. Such lifecycle tests are especially relevant to chat, retrieval-augmented generation, and agent workflows where the same prefix can be reused, extended, evicted, and loaded repeatedly.^[26]

Numerical results should remain attached to their protocol. A method that increases maximum batch size under a memory cap may not improve single-request latency. A method that reduces cache bytes may lose its advantage after dequantization, transfers, or longer generated outputs. Reproducible comparisons therefore report both cache-only measurements and end-to-end serving behavior.^[15][21][27]

Limitations and operational risks

The principal limitation of a full KV cache is linear growth with stored tokens, active sequences, cache-producing layers, key-value heads, and bytes per element. Long contexts can reduce feasible batch size or exhaust device memory even when model weights fit. Local attention, head sharing, compression, paging, and offload address different factors and can be combined, but their costs do not disappear.^[4][13]

Cache state is tied to exact model execution. Changing weights, adapters, quantization conventions, tokenization, attention masks, positional configuration, or relevant multimodal inputs can require invalidation. Reusing incompatible state can silently produce wrong outputs rather than a clean error.^[2][13]

Approximate caches create an accuracy and reliability tradeoff. Perplexity or average benchmark scores are insufficient for applications that depend on a particular earlier fact, exact quotation, code token, or reasoning step. Evaluation should include position-sensitive retrieval, long generation, multi-turn use, and per-sample error analysis under the actual cache budget.^[26][27][28]

KV tensors can encode information about private prompts even though they are not human-readable text. Cross-tenant prefix matching, cache timing, offloaded files, crash dumps, and stale blocks therefore require access control, isolation, deletion, and encryption policies appropriate to the deployment. Prefix reuse should be treated as a security-sensitive optimization, not only a performance feature.^[13][14]

See also

• Beam search
• Tensor Parallelism
• Speculative Decoding
• Inference optimization
• Serving
• Cross-model KV cache transfer
• Sparse attention
• KV cache offloading

References

1. ^1 ^2 ^3Vaswani, A., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. Source
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Hugging Face. "KV cache strategies." Transformers 4.50.0 documentation. Source
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Shazeer, N. "Fast Transformer Decoding: One Write-Head is All You Need." arXiv:1911.02150, 2019. Source
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Pope, R., et al. "Efficiently Scaling Transformer Inference." Proceedings of Machine Learning and Systems 5, 2023. Source
5. ^1 ^2 ^3Touvron, H., et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv:2307.09288, 2023. Source
6. ^1 ^2Ainslie, J., et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." Proceedings of EMNLP 2023, pp. 4895-4901. Source
7. ^1 ^2Jiang, A. Q., et al. "Mistral 7B." arXiv:2310.06825, 2023. Source
8. ^1 ^2 ^3DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv:2405.04434, 2024. Source
9. ^1 ^2 ^3 ^4 ^5 ^6Kwon, W., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." Proceedings of the 29th ACM Symposium on Operating Systems Principles, 2023. Source
10. ^Yu, G.-I., et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation, 2022. Source
11. ^1 ^2 ^3Prabhu, R., et al. "vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention." Proceedings of ASPLOS 2025, 2025. Source
12. ^1 ^2Zheng, L., et al. "SGLang: Efficient Execution of Structured Language Model Programs." Advances in Neural Information Processing Systems 37, 2024. Source
13. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9NVIDIA. "KV Cache System." TensorRT-LLM 1.1.0 documentation. Source
14. ^1 ^2OpenAI. "Prompt Caching in the API." October 1, 2024. Source
15. ^1 ^2 ^3 ^4 ^5 ^6Liu, Z., et al. "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache." Proceedings of the 41st International Conference on Machine Learning, 2024. Source
16. ^1 ^2 ^3Hooper, C., et al. "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization." Advances in Neural Information Processing Systems 37, 2024. Source
17. ^1 ^2Zhang, Z., et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." Advances in Neural Information Processing Systems 36, 2023. Source
18. ^1 ^2Xiao, G., et al. "Efficient Streaming Language Models with Attention Sinks." International Conference on Learning Representations, 2024. Source
19. ^1 ^2Li, Y., et al. "SnapKV: LLM Knows What You are Looking for Before Generation." Advances in Neural Information Processing Systems 37, 2024. Source
20. ^1 ^2 ^3Liu, A., et al. "MiniCache: KV Cache Compression in Depth Dimension for Large Language Models." Advances in Neural Information Processing Systems 37, 2024. Source
21. ^1 ^2 ^3 ^4Zhong, Y., et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. Source
22. ^Qin, R., et al. "Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot." 23rd USENIX Conference on File and Storage Technologies, 2025. Source
23. ^1 ^2Liu, Y., et al. "CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving." Proceedings of ACM SIGCOMM 2024, pp. 38-56. Source
24. ^Dao, T., et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." Advances in Neural Information Processing Systems 35, 2022. Source
25. ^Dao, T. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." International Conference on Learning Representations, 2024. Source
26. ^1 ^2Li, Y., et al. "SCBench: A KV Cache-Centric Analysis of Long-Context Methods." International Conference on Learning Representations, 2025. Source
27. ^1 ^2 ^3 ^4 ^5Gao, W., et al. "Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving." Proceedings of Machine Learning and Systems 7, 2025. Source
28. ^1 ^2 ^3Tian, J., et al. "SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models." Proceedings of Machine Learning and Systems 8, 2026. Source
29. ^Du, J., et al. "Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration." 20th USENIX Symposium on Operating Systems Design and Implementation, 2026. Source
30. ^Touvron, H., et al. "LLaMA: Open and Efficient Foundation Language Models." arXiv:2302.13971, 2023. Source
31. ^1 ^2 ^3Brandon, W., et al. "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention." arXiv:2405.12981, 2024. Source
32. ^1 ^2 ^3Character.AI. "Optimizing AI Inference at Character.AI." June 20, 2024. Source
33. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8DeepSeek-AI. "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression." Technical report, September 2026. Source
34. ^1 ^2DeepSeek-AI. "DeepSeek-V4.1-Flash." Hugging Face model card, September 10, 2026. Source
35. ^DeepSeek-AI. "DeepSeek LLM: Scaling Open-Source Language Models with Longtermism." arXiv:2401.02954, 2024. Source
36. ^1 ^2DeepSeek-AI. "deepseek-llm-67b-base." Hugging Face model repository, November 29, 2023. Source
37. ^DeepSeek. "DeepSeek-V3.2 Release." DeepSeek API Docs, December 1, 2025. Source
38. ^1 ^2DeepSeek. "Introducing DeepSeek-V3.2-Exp." DeepSeek API Docs, September 29, 2025. Source
39. ^DeepSeek. "DeepSeek V4 Preview Release." DeepSeek API Docs, April 24, 2026. Source
40. ^1 ^2 ^3DeepSeek. "DeepSeek-V4.1-Flash: Smarter, Faster, More Efficient." DeepSeek API Docs, September 10, 2026. Source
41. ^DeepSeek-AI. "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence." arXiv:2606.19348, 2026. Source
42. ^Gemma Team. "Gemma 3 Technical Report." arXiv:2503.19786, 2025. Source
43. ^DeepSeek. "Models & Pricing." DeepSeek API Docs, accessed September 11, 2026. Source
44. ^Lee, W., et al. "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management." 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. Source
45. ^Huang, Y., et al. "NOSA: Native and Offloadable Sparse Attention." arXiv:2510.13602, 2025. Source
46. ^Fu, Y., et al. "SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference." arXiv:2606.04511, 2026. Source