# KV Cache

> Source: https://aiwiki.ai/wiki/kv_cache
> Updated: 2026-07-31
> Fact-checked: 2026-07-31
> Categories: AI Inference, Deep Learning, Machine Learning, Transformer Models
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "KV Cache." aiwiki.ai, 31 Jul 2026. https://aiwiki.ai/wiki/kv_cache
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**KV cache**, short for key-value cache, is transient model state used during [Transformer](https://aiwiki.ai/wiki/transformers) generation. At each attention layer, it retains the key and value vectors already computed for prompt tokens and earlier output tokens. An [autoregressive model](https://aiwiki.ai/wiki/autoregressive_model) can then process only the newest token instead of running every earlier token through the full model again.[1][2]

The cache is central to Transformer-based [large language model](https://aiwiki.ai/wiki/large_language_model) [inference](https://aiwiki.ai/wiki/inference), but it is not a cache of text answers or database records. It contains model-specific tensors. Their shape, numerical format, positional treatment, and allocation layout depend on the model and runtime. A cache created for one set of weights, adapter parameters, or token positions is not generally valid for another.[2][3]

KV caching removes repeated computation for the old tokens' projections and feed-forward layers. It does not make dense attention independent of context length: each new query must still attend to the retained keys and values. Cache memory therefore grows linearly with the number of stored tokens, and the amount read during each dense decode step grows with the active context.[3][4]

## How it works

### Causal self-attention

In a causal [self-attention](https://aiwiki.ai/wiki/self_attention) layer, hidden state `x_t` at position `t` is projected into a query `q_t`, key `k_t`, and value `v_t`. The query is compared with keys at allowed positions, and the resulting attention weights combine their values. Causal masking prevents position `t` from using a later position. The original Transformer established this attention structure, although it did not define a standard runtime cache interface.[1]

During incremental generation, keys and values for earlier positions do not change when a later token is appended. A runtime can retain those tensors, compute only `q_t`, `k_t`, and `v_t` for the new position at each layer, append the new key and value, and evaluate attention over the accumulated state. Implementations often expose this state as `past_key_values`, a cache object, or an engine-managed block pool.[2][3]

This article primarily describes decoder-only causal self-attention. Encoder-decoder models can maintain separate decoder self-attention state and cross-attention state derived from the encoder output. Models without attention may use recurrent or state-space caches instead of key-value tensors.[2]

Caching is exact only when the retained tensors have the same numerical meaning as recomputation would produce. Lower-precision storage, token eviction, cross-layer merging, or approximate retrieval can save memory but may change attention results. Cache allocation and reuse are therefore separate questions from cache compression.[15][17][20]

### Prefill and decode

An inference request usually has two phases. During **prefill**, the model processes the prompt and creates key-value state for its tokens. Prompt tokens are available together, so their matrix operations can be parallelized, although full causal attention still has quadratic arithmetic in prompt length. During **decode**, the model produces one or more new tokens sequentially and extends the cache after each accepted token.[4][21]

| Phase | Input to a model step | Cache action | Common latency concern |
| --- | --- | --- | --- |
| Prefill | Many prompt tokens | Create the prompt cache | Time to first token |
| Decode | Usually one new token per sequence | Read prior state and append new state | Time per output token |

The distinction matters because the phases stress hardware differently. Prefill can expose substantial matrix-multiplication parallelism. Decode has a sequential dependency between output tokens, and at low or moderate batch sizes it often spends substantial time moving weights and KV state from high-bandwidth memory. Whether weights or cache traffic dominates depends on model size, batch size, sequence length, data type, parallel layout, and hardware.[3][4]

### What computation the cache saves

Without a cache, producing a new token by rerunning the entire growing prefix would repeat the old tokens' projections, attention layers, and feed-forward layers. With a cache, those old key and value projections are reused. Holding model dimensions fixed, dense attention for one new token still reads and scores a number of cached positions proportional to the current sequence length.[3][4]

For a prompt of `P` tokens followed by `N` generated tokens, cached dense decode attention has sequence-length work proportional to `N x P + N x (N - 1) / 2` at each layer. The non-attention work for old positions is not repeated. This is why describing cached decode as simply `O(1)` is misleading: the newly projected token is constant in count, but the dense attention span is not.[1][3]

## Memory accounting

For a decoder-only model with the same attention shape in every layer, the raw KV tensor payload can be estimated as:[3][4][11]

`payload bytes = 2 x sum(T_i) x L x H_KV x D_head x S`

Here, `T_i` is the stored length of active sequence `i`, `L` is the number of cache-producing layers, `H_KV` is the number of key-value heads, `D_head` is the dimension of each head, and `S` is bytes per stored element. The factor of two represents keys and values. For a uniform batch, `sum(T_i)` becomes batch size times stored tokens per sequence.[3][4]

This formula describes tensor payload, not necessarily reserved device memory. Real runtimes may also hold block tables, alignment padding, quantization scales and zero points, unquantized residual windows, beam or speculative branches, allocator slack, and duplicated state caused by sharding or replication. Conversely, exact-prefix sharing can let several requests reference the same physical blocks.[9][13][15]

### Corrected Llama 2 13B example

Llama 2 13B has a 4,096-token training context and does not use grouped-query attention. Its inherited LLaMA architecture has 40 layers, hidden dimension 5,120, and 40 attention heads, giving a head dimension of 128. Under the explicit assumptions of multi-head attention, one sequence, no prefix sharing, and two-byte key and value elements, the calculation is `2 x 40 x 40 x 128 x 2` bytes per token.[5][30]

| Quantity | Exact raw payload |
| --- | ---: |
| One cached token | 819,200 bytes = 800 KiB = 0.78125 MiB |
| 4,096 cached tokens | 3,355,443,200 bytes = 3.125 GiB |

These are binary units and exclude model weights, activations, allocator overhead, and runtime metadata. A quantized cache or a model with fewer key-value heads would use a different amount. Extending the arithmetic past 4,096 tokens would be a hypothetical memory calculation, not evidence that the unmodified model reliably supports a longer [context window](https://aiwiki.ai/wiki/context_window).[5][15]

## Architectural ways to reduce the cache

### Multi-query and grouped-query attention

In standard multi-head attention, every query head has its own key and value head, so `H_KV` equals the number of query heads. Multi-query attention uses one shared key head and one shared value head. This can reduce raw cache payload and key-value bandwidth by the number of query heads, but it also changes the model architecture and removes a head axis that systems might otherwise shard.[3][4]

[Grouped-query attention](https://aiwiki.ai/wiki/grouped_query_attention) uses more than one key-value head but fewer key-value heads than query heads. It creates an intermediate memory and quality tradeoff between multi-head and multi-query attention. The original GQA study converted T5 checkpoints and found quality close to multi-head attention with speed comparable to multi-query attention in its tested tasks. That result is evidence for the study protocol, not a guarantee for every model or workload.[6]

Because cache payload is proportional to `H_KV`, reducing 32 key-value heads to 8 gives a nominal fourfold payload reduction when all other terms are equal. The realized serving gain can be smaller or larger because attention kernels, batch capacity, communication, and weight traffic also affect latency and throughput.[4][6]

### Latent and local attention

Multi-head latent attention stores a compressed latent representation from which attention keys and values are derived. [DeepSeek-V2](https://aiwiki.ai/wiki/deepseek_v2) reported a 93.3 percent KV-cache reduction relative to DeepSeek 67B and attributed part of its higher generation throughput to this architecture. Both figures are model-relative results from the DeepSeek-V2 evaluation, not general reduction factors for latent attention.[8]

Sliding-window attention limits each layer to a fixed span of recent positions, which bounds that layer's cache once the window is full. Mistral 7B combined sliding-window attention with grouped-query attention. A bounded local cache does not preserve direct access to every old token, and support must be part of the model's attention design. Discarding old state from a model trained for full attention is not equivalent to running that model with its full context.[7]

Hybrid models may alternate local and global attention layers, use different window sizes by layer, or maintain recurrent state instead of a KV cache in some layers. Their memory must be calculated per layer rather than by applying one head count and one window to the whole network.[2][7]

## Runtime allocation and scheduling

### Dynamic, static, and paged storage

A dynamic contiguous cache grows with the sequence. It avoids reserving a full maximum length at request admission, but repeated growth or relocation can be costly. A static cache reserves a maximum shape in advance, which can work well with compiled graphs but may waste memory when requests end early or have varied lengths. Offloaded caches move some layer state to host memory, trading device capacity for transfer traffic and often lower throughput.[2]

[PagedAttention](https://aiwiki.ai/wiki/paged_attention), introduced with [vLLM](https://aiwiki.ai/wiki/vllm), divides a request's logical cache into fixed-size blocks that can map to noncontiguous physical memory. Blocks are allocated as a sequence grows and reclaimed when it ends. Copy-on-write sharing can let parallel samples or common prefixes share blocks until their tokens diverge. In the SOSP 2023 evaluation, vLLM reported two to four times the throughput of its comparison systems at similar latency; that range belongs to the paper's models, hardware, and workloads.[9]

Paged storage reduces internal waste and external fragmentation, but it requires block tables and attention kernels that can follow the page mapping. vAttention proposed a different design: reserve a contiguous virtual range and map physical GPU memory into it on demand with CUDA virtual-memory mechanisms. Its ASPLOS 2025 paper reported up to 1.23 times higher end-to-end serving throughput than tested PagedAttention-based kernels in long-context experiments. This is a comparison of particular implementations, not proof that either allocation design is always faster.[11]

### Iteration-level scheduling

Cache lifetime is tied to request scheduling. A fixed batch can leave finished requests occupying slots while shorter requests wait. Orca instead scheduled at iteration granularity and applied batching selectively, allowing the active set to change between generation iterations. Modern [continuous batching](https://aiwiki.ai/wiki/continuous_batching) follows the same broad goal: admit, advance, pause, or retire requests while promptly assigning and reclaiming their cache state.[10]

Iteration-level scheduling introduces tradeoffs among batch efficiency, head-of-line blocking, preemption cost, and available cache memory. A preempted request may retain its blocks, move them to slower memory, or discard them and later recompute its prefix. The best choice depends on transfer bandwidth, expected reuse, request priority, and service-level objectives.[9][13]

## Prefix reuse and prompt caching

Two requests with an identical token prefix can reuse that prefix's KV state because causal attention for the prefix does not depend on later suffix tokens. Correct reuse requires more than matching visible text: token IDs, model weights, adapters, positional treatment, cache format, and relevant preprocessing must agree. The reusable range ends at the first incompatible block or token.[12][13]

RadixAttention in SGLang organizes reusable prefixes in a radix tree, while block-based runtimes can hash or index completed prefix blocks. Reuse can reduce prefill computation and physical memory when multiple requests share system instructions, documents, or conversation history. It does not reduce the decode attention span over the reused prefix, because new queries still attend to that state.[12][13]

[Prompt caching](https://aiwiki.ai/wiki/prompt_caching) is also the name of hosted API features. Such a product may expose cached-token accounting, lower latency, or different billing while keeping its internal representation private. An API-level cache hit should not be described as a user-portable KV tensor unless the provider documents that interface. Mutable price tables, model catalogs, thresholds, and retention periods are product documentation rather than stable properties of KV caching.[14]

Cross-request reuse must respect isolation boundaries. TensorRT-LLM 1.1.0 documents a cache-salt mechanism that restricts reuse to requests with the same salt, specifically to reduce prompt-theft risk. A serving system should also prevent cache state from surviving longer than policy allows or being reused after a model, adapter, or authorization context changes.[13]

## Compression and bounded caches

### Quantization

[KV-cache quantization](https://aiwiki.ai/wiki/kv_cache_quantization) stores keys and values with fewer bits. Moving from a two-byte element to a four-bit payload has a nominal fourfold tensor reduction, but the actual ratio is lower when scales, zero points, alignment, outlier channels, and a higher-precision residual region are included. Quantization also adds conversion work and may change model output.[15][16]

KIVI found different outlier structure in keys and values and used per-channel quantization for keys and per-token quantization for values. Its two-bit method reported 2.6 times lower peak memory including model weights and 2.35 to 3.47 times higher throughput on its tested Llama 2, Falcon, and Mistral workloads. Those results do not mean every two-bit cache preserves quality or achieves the same system gain.[15]

KVQuant combined per-channel key quantization, pre-RoPE key handling, nonuniform data types, dense-and-sparse outlier separation, and a full-precision residual window. Its design illustrates why a bare bit width is not a complete cache specification. The model, calibration data, context length, residual policy, and dequantization kernel all affect accuracy and performance.[16]

### Eviction, selection, and cross-layer compression

Eviction methods retain only a subset of token positions. H2O keeps a balance of recent tokens and attention heavy hitters. StreamingLLM keeps a small set of initial attention-sink tokens together with a recent window. SnapKV uses an observation window near the end of the prompt to select clustered prompt positions for each attention head. These methods use different signals and were evaluated on different models and tasks.[17][18][19]

StreamingLLM's "infinite" framing refers to stable streaming generation with bounded state in its tests. It does not give the model lossless retrieval of every discarded token. Likewise, SnapKV's reported long-context results do not establish that its selected positions preserve every fact, ordering relation, or future query target.[18][19]

MiniCache compresses along model depth by merging selected key-value states from adjacent layers while retaining unusually dissimilar pairs. This can complement token selection or quantization, but it changes the stored representation and depends on cross-layer similarity in the evaluated model.[20]

| Strategy | Primary memory lever | Main risk or cost |
| --- | --- | --- |
| Fewer key-value heads | Reduce state per token | Requires a compatible model architecture |
| Quantization | Reduce bits per stored element | Numerical error and conversion overhead |
| Sliding window or eviction | Reduce stored token positions | Loss of older information |
| Cross-layer merging | Reduce duplicated state across depth | Approximation depends on layer similarity |
| Host or storage offload | Move state out of device memory | Transfer latency and bandwidth |

No compression ratio is a universal quality guarantee. An MLSys 2025 study found that some compression implementations did not translate memory savings into production-level throughput, could lengthen generated answers, and hid per-sample failures behind aggregate scores. A 2026 reasoning-workload study further found that token-wise eviction could become unstable in multi-batch settings and could provoke longer reasoning sequences. These findings support evaluating complete outputs and systems, not cache size alone.[27][28]

## Distributed and hierarchical caches

[Disaggregated serving](https://aiwiki.ai/wiki/disaggregated_serving) can place prefill and decode on different workers. Prefill workers create KV state, which must then move to decode workers. DistServe chose separate GPU allocations and parallel plans for the two phases and placed them with network bandwidth in mind. Its final OSDI 2024 evaluation reported either 7.4 times more requests served or a 12.6 times tighter service-level objective than comparison systems while meeting latency constraints for more than 90 percent of requests.[21]

Those figures correct earlier prepublication values and remain protocol-specific. Disaggregation is beneficial only when reduced phase interference and specialized resource allocation outweigh cache-transfer and coordination costs. The cache can be large enough that moving it over a slow interconnect is more expensive than recomputing part of the prompt.[21][23]

Mooncake extended the idea into a distributed cache spanning GPU memory, CPU memory, SSD, and network resources. Its scheduler seeks reusable prefixes, streams newly created state from prefill to decode workers, replicates hot blocks, and moves colder blocks down the hierarchy. The FAST 2025 paper documents a production-derived design and evaluation, but its deployment scale and capacity gains are authors' reported results rather than independent measurements of every installation.[22]

CacheGen addressed network movement by encoding KV tensors before transfer and choosing among loading, compression levels, and recomputation according to bandwidth. This is a different problem from shrinking the active decode cache on one GPU: a representation optimized for transport may be decoded into a larger active cache at the destination.[23]

Full prefill-decode separation is not automatically best on commodity clusters. The OSDI 2026 EcoServe study identified interconnect dependence as a limitation and evaluated a partially disaggregated schedule on Ethernet-connected GPUs. Its result reinforces that topology, cache-transfer volume, and latency targets must be included when comparing serving architectures.[29]

## Relation to attention kernels

[Flash Attention](https://aiwiki.ai/wiki/flash_attention) reduces reads and writes of attention intermediates by tiling exact attention around the GPU memory hierarchy. It avoids materializing the full attention-score matrix in high-bandwidth memory. This is distinct from the persistent KV cache: incremental decode still needs retained keys and values unless the model uses another state mechanism or recomputes them.[24]

FlashAttention-2 improved work partitioning and parallelism over the first algorithm. Kernel speedups do not directly equal end-to-end serving speedups because model weights, KV allocation, cache layout, scheduling, communication, and sampling remain. A paged cache may also require a paged-aware kernel rather than the contiguous layout assumed by another implementation.[11][25]

## Implementation details

Cache layouts commonly arrange dimensions corresponding to batch or request, token position, key-value head, and head dimension, but the physical order varies by framework and kernel. Some block pools pack several layers together; others keep a tensor per layer. A serialized cache is therefore not portable merely because two models have the same number of heads.[2][13]

Position handling must remain consistent. With rotary position embeddings, a runtime may store keys before or after applying the positional rotation, depending on how its kernels are fused. Reusing a prefix at a different absolute position or under a different RoPE-scaling configuration can be invalid even when token IDs match. The same caution applies to attention masks and model-specific cache transformations.[8][16]

Branching generation can share a prefix cache until branches diverge. Beam search, parallel sampling, and speculative decoding may then allocate additional blocks, discard rejected speculative state, or copy-on-write shared blocks. Capacity planning should use the number of live branches and accepted tokens, not only the final answer length.[9]

With tensor parallelism, each worker may store a shard of the cache, replicate key-value heads, or combine sharding and replication. A model with eight key-value heads is not inherently limited to eight GPUs: systems can partition other dimensions or replicate heads when the parallel degree exceeds the head count. The choice changes memory, collective communication, and kernel efficiency.[4]

## Evaluation

A useful cache evaluation reports the full workload and separates memory from service quality. At minimum it should specify model checkpoint, cache format, prompt and output-length distributions, batch or arrival process, hardware, parallel layout, attention kernel, prefix-hit rate, and whether reported memory is tensor payload, allocated memory, or peak process memory.[9][27]

| Metric | What it captures | Common confounder |
| --- | --- | --- |
| Time to first token | Prompt processing and queueing | Prefix hits can bypass part of prefill |
| Time per output token | Decode responsiveness | Grows with active context and batch composition |
| Throughput | Tokens or requests completed per time | Can hide tail latency or missed objectives |
| Goodput | Work completed within latency objectives | Depends on the stated objective and arrival trace |
| Peak memory | Maximum runtime footprint | May include weights, allocator reserve, and temporary buffers |
| Task quality | Effect of approximate caching | Aggregate scores can hide individual failures |

Shared-context workloads need more than a single-request benchmark. SCBench evaluates cache generation, compression, retrieval, and loading across repeated-context tasks and found that some sublinear-memory methods degraded in multi-turn settings. Such lifecycle tests are especially relevant to chat, retrieval-augmented generation, and agent workflows where the same prefix can be reused, extended, evicted, and loaded repeatedly.[26]

Numerical results should remain attached to their protocol. A method that increases maximum batch size under a memory cap may not improve single-request latency. A method that reduces cache bytes may lose its advantage after dequantization, transfers, or longer generated outputs. Reproducible comparisons therefore report both cache-only measurements and end-to-end serving behavior.[15][21][27]

## Limitations and operational risks

The principal limitation of a full KV cache is linear growth with stored tokens, active sequences, cache-producing layers, key-value heads, and bytes per element. Long contexts can reduce feasible batch size or exhaust device memory even when model weights fit. Local attention, head sharing, compression, paging, and offload address different factors and can be combined, but their costs do not disappear.[4][13]

Cache state is tied to exact model execution. Changing weights, adapters, quantization conventions, tokenization, attention masks, positional configuration, or relevant multimodal inputs can require invalidation. Reusing incompatible state can silently produce wrong outputs rather than a clean error.[2][13]

Approximate caches create an accuracy and reliability tradeoff. Perplexity or average benchmark scores are insufficient for applications that depend on a particular earlier fact, exact quotation, code token, or reasoning step. Evaluation should include position-sensitive retrieval, long generation, multi-turn use, and per-sample error analysis under the actual cache budget.[26][27][28]

KV tensors can encode information about private prompts even though they are not human-readable text. Cross-tenant prefix matching, cache timing, offloaded files, crash dumps, and stale blocks therefore require access control, isolation, deletion, and encryption policies appropriate to the deployment. Prefix reuse should be treated as a security-sensitive optimization, not only a performance feature.[13][14]

## See also

- [Beam search](https://aiwiki.ai/wiki/beam_search)
- [Tensor Parallelism](https://aiwiki.ai/wiki/tensor_parallelism)
- [Speculative Decoding](https://aiwiki.ai/wiki/speculative_decoding)
- [Inference optimization](https://aiwiki.ai/wiki/inference_optimization)
- [Serving](https://aiwiki.ai/wiki/serving)

## References

1. Vaswani, A., et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. [Source](https://arxiv.org/abs/1706.03762)
2. Hugging Face. "KV cache strategies." Transformers 4.50.0 documentation. [Source](https://huggingface.co/docs/transformers/v4.50.0/kv_cache)
3. Shazeer, N. "Fast Transformer Decoding: One Write-Head is All You Need." arXiv:1911.02150, 2019. [Source](https://arxiv.org/abs/1911.02150)
4. Pope, R., et al. "Efficiently Scaling Transformer Inference." Proceedings of Machine Learning and Systems 5, 2023. [Source](https://proceedings.mlsys.org/paper_files/paper/2023/hash/c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html)
5. Touvron, H., et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv:2307.09288, 2023. [Source](https://arxiv.org/abs/2307.09288)
6. Ainslie, J., et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." Proceedings of EMNLP 2023, pp. 4895-4901. [Source](https://aclanthology.org/2023.emnlp-main.298/)
7. Jiang, A. Q., et al. "Mistral 7B." arXiv:2310.06825, 2023. [Source](https://arxiv.org/abs/2310.06825)
8. DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv:2405.04434, 2024. [Source](https://arxiv.org/abs/2405.04434)
9. Kwon, W., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." Proceedings of the 29th ACM Symposium on Operating Systems Principles, 2023. [Source](https://arxiv.org/abs/2309.06180)
10. Yu, G.-I., et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." 16th USENIX Symposium on Operating Systems Design and Implementation, 2022. [Source](https://www.usenix.org/conference/osdi22/presentation/yu)
11. Prabhu, R., et al. "vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention." Proceedings of ASPLOS 2025, 2025. [Source](https://doi.org/10.1145/3669940.3707256)
12. Zheng, L., et al. "SGLang: Efficient Execution of Structured Language Model Programs." Advances in Neural Information Processing Systems 37, 2024. [Source](https://proceedings.neurips.cc/paper_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract-Conference.html)
13. NVIDIA. "KV Cache System." TensorRT-LLM 1.1.0 documentation. [Source](https://nvidia.github.io/TensorRT-LLM/1.1.0/features/kvcache.html)
14. OpenAI. "Prompt Caching in the API." October 1, 2024. [Source](https://openai.com/index/api-prompt-caching/)
15. Liu, Z., et al. "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache." Proceedings of the 41st International Conference on Machine Learning, 2024. [Source](https://proceedings.mlr.press/v235/liu24bz.html)
16. Hooper, C., et al. "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization." Advances in Neural Information Processing Systems 37, 2024. [Source](https://proceedings.neurips.cc/paper_files/paper/2024/hash/028fcbcf85435d39a40c4d61b42c99a4-Abstract-Conference.html)
17. Zhang, Z., et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." Advances in Neural Information Processing Systems 36, 2023. [Source](https://proceedings.neurips.cc/paper_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html)
18. Xiao, G., et al. "Efficient Streaming Language Models with Attention Sinks." International Conference on Learning Representations, 2024. [Source](https://proceedings.iclr.cc/paper_files/paper/2024/hash/5e5fd18f863cbe6d8ae392a93fd271c9-Abstract-Conference.html)
19. Li, Y., et al. "SnapKV: LLM Knows What You are Looking for Before Generation." Advances in Neural Information Processing Systems 37, 2024. [Source](https://proceedings.neurips.cc/paper_files/paper/2024/hash/28ab418242603e0f7323e54185d19bde-Abstract-Conference.html)
20. Liu, A., et al. "MiniCache: KV Cache Compression in Depth Dimension for Large Language Models." Advances in Neural Information Processing Systems 37, 2024. [Source](https://proceedings.neurips.cc/paper_files/paper/2024/hash/fd0705710bf01b88a60a3d479ea341d9-Abstract-Conference.html)
21. Zhong, Y., et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." 18th USENIX Symposium on Operating Systems Design and Implementation, 2024. [Source](https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin)
22. Qin, R., et al. "Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot." 23rd USENIX Conference on File and Storage Technologies, 2025. [Source](https://www.usenix.org/conference/fast25/presentation/qin)
23. Liu, Y., et al. "CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving." Proceedings of ACM SIGCOMM 2024, pp. 38-56. [Source](https://doi.org/10.1145/3651890.3672274)
24. Dao, T., et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." Advances in Neural Information Processing Systems 35, 2022. [Source](https://proceedings.neurips.cc/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html)
25. Dao, T. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." International Conference on Learning Representations, 2024. [Source](https://proceedings.iclr.cc/paper_files/paper/2024/hash/98ed250b203d1ac6b24bbcf263e3d4a7-Abstract-Conference.html)
26. Li, Y., et al. "SCBench: A KV Cache-Centric Analysis of Long-Context Methods." International Conference on Learning Representations, 2025. [Source](https://proceedings.iclr.cc/paper_files/paper/2025/hash/a540b17fb2295c736d5afd6c507acf66-Abstract-Conference.html)
27. Gao, W., et al. "Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving." Proceedings of Machine Learning and Systems 7, 2025. [Source](https://proceedings.mlsys.org/paper_files/paper/2025/hash/26289c647c6828e862e271ca3c490486-Abstract-Conference.html)
28. Tian, J., et al. "SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models." Proceedings of Machine Learning and Systems 8, 2026. [Source](https://proceedings.mlsys.org/paper_files/paper/2026/hash/45c1f6a8cbf2da59ebf2c802b4f742cd-Abstract-Conference.html)
29. Du, J., et al. "Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration." 20th USENIX Symposium on Operating Systems Design and Implementation, 2026. [Source](https://www.usenix.org/conference/osdi26/presentation/du)
30. Touvron, H., et al. "LLaMA: Open and Efficient Foundation Language Models." arXiv:2302.13971, 2023. [Source](https://arxiv.org/abs/2302.13971)