KV cache offloading

KV cache offloading is the practice of moving part or all of a Transformer model's KV cache out of accelerator memory (GPU HBM) into a larger, slower tier such as CPU DRAM, local NVMe storage, or remote storage, and then bringing back, or recomputing, what a later step needs. It exists because the cache grows with every stored token and every concurrent sequence while the memory attached to a GPU does not, so long contexts and large batches run out of HBM long before they run out of compute.^[1][2] The cost of the technique is a bandwidth constraint: the host link that connects a GPU to CPU memory is far slower than the GPU's own memory, so a design that moves too much data per decode step trades an out-of-memory failure for a transfer stall.^[3][4]

Offloading takes three broad forms. Serving engines such as vLLM, SGLang, and TensorRT-LLM copy completed blocks of a finished or preempted request to host memory or storage and reload them when a later request shares the same prefix, an extension of prefix caching.^[5][6][7] Throughput-oriented single-GPU systems such as FlexGen stream weights and cache through the GPU layer by layer.^[1] And a line of research beginning with InfiniGen keeps the full cache on the CPU during decoding and fetches only the entries that sparse attention will use for the current step, an idea that NOSA and SparDA have since built into models trained with the transfer budget in mind.^[2][3][4][8]

Why the cache leaves the GPU

The size of a full-attention KV cache is the product of stored tokens, cache-producing layers, key-value heads, head dimension, bytes per element, and a factor of two for keys and values; the KV cache article gives the formula and worked examples. A concrete case is MiniCPM4.1-8B, one of the two models used to evaluate SparDA. Its configuration file lists 32 layers, 32 attention heads over a hidden size of 4,096 (so a head dimension of 128), only 2 key-value heads because it uses grouped-query attention, and bfloat16 weights.^[9] At two bytes per element that is 2 x 32 x 2 x 128 x 2 = 32,768 bytes, or 32 KiB, per cached token. A single 131,072-token sequence therefore holds 4 GiB of cache, and a batch of 64 such sequences holds 256 GiB, more than three times the 80 GB of an NVIDIA H100.^[10] Models without such aggressive head sharing are far larger per token; the PagedAttention paper's example of the 13-billion-parameter OPT model, with full multi-head attention over 40 layers, needs 800 KB per token, twenty-five times as much.^[23]

The memory that can absorb that overflow sits at the far end of progressively slower links. NVIDIA's H100 specification lists 3.35 TB/s of HBM bandwidth for the SXM part against 128 GB/s for its PCIe Gen5 interface.^[10] NVIDIA describes the NVLink-C2C link of the GH200 Grace Hopper module as 900 GB/s bidirectional and "7x the bandwidth of x16 PCIe Gen links", which puts the PCIe figure at roughly 128 GB/s counted across both directions, or about 64 GB/s each way.^[11] Storage and network tiers are slower again and have less predictable latency, which is why SGLang's documentation treats host-to-GPU loading and storage-to-host prefetching as separate problems with separate policies.^[6]

The arithmetic of a dense decode step shows why the host link, not host capacity, is the binding constraint. Dense attention reads every stored key and value for every generated token. For the 4 GiB cache of one 131,072-token MiniCPM4.1-8B sequence, that read takes about 1.3 milliseconds from H100 HBM at 3.35 TB/s and about 67 milliseconds over one direction of PCIe Gen5 at 64 GB/s, before any computation.^[9][10][11] A system that offloads the whole cache and reads all of it back each step is therefore roughly fifty times slower per token than one that keeps it resident, which is why every practical offloading design either reloads the cache once per request, streams it layer by layer behind computation, or reads only a sparse subset per step.

Reloading a prefix once: serving-engine offload tiers

The most common production form of offloading is a spillover for prefix caching. When a request completes or is preempted, its KV blocks are copied to host memory instead of being freed; if a later request begins with the same tokens, the blocks are copied back and the engine skips that part of prefill. The transfer happens once per request rather than once per decode step, so the cost is a time-to-first-token penalty on a hit, weighed against recomputing the prefix.

vLLM

vLLM's OffloadingConnector extends the prefix cache "by offloading completed KV blocks to slower but larger tiers (CPU host memory, plus optional secondary tiers) as they are produced", promoting hits back to the GPU on demand; transfers use DMA through cudaMemcpyAsync and run asynchronously alongside model computation.^[5] Two configurations exist: a single-tier CPUOffloadingSpec that copies completed GPU blocks into pinned host memory, and a TieringOffloadingSpec with a CPU primary tier plus ordered secondary tiers of type fs (a filesystem directory), obj (an S3-compatible object store through the NIXL OBJ backend), or p2p (block exchange between vLLM instances over RDMA via NIXL). Only the CPU tier has direct GPU access; every GPU-to-secondary transfer is staged through it. The primary tier uses lru eviction by default with arc as an alternative, a request can cap how many tokens it loads with max_load_tokens (tokens beyond the cap are recomputed), and by default only prompt blocks are offloaded while decode blocks are skipped.^[5]

A January 2026 post by Or Ozeri and Danny Harnik of the vLLM team at IBM Research describes the feature's history and motivation. The offloading connector was introduced in vLLM 0.11.0 on top of an asynchronous connector API added in 0.9.0. The authors note that offloading helps even when requests share no prefix, because a preempted request's cache can be restored from DRAM instead of recomputed. In their single-request benchmark on Llama-3.1-8B-Instruct and an H100, loading a cached prompt from CPU cut time-to-first-token by 2x to 22x depending on prompt size, and in a 10,000-request throughput test the gain rose with the CPU hit rate up to 9x. A large part of the 0.12.0 improvement came from changing vLLM's GPU memory layout so that one logical block holds the KV data of all layers contiguously, raising the effective physical block size from 32 KB to 2 MB for Llama-3.1-8B-Instruct so that DMA copies run at full speed.^[16]

vLLM's separate --cpu-offload-gb option is a different mechanism. Its configuration class is documented as "Configuration for model weight offloading": it keeps part of the weights in pinned CPU memory and reads them over the interconnect on every forward pass, and it does not touch the KV cache.^[17]

SGLang HiCache

SGLang's HiCache organizes GPU memory as L1, host memory as L2, and distributed storage as L3, "inspired by the classic three-level cache design of modern CPUs". A HiRadixTree extends the RadixAttention prefix tree so that each node records where a span's KV cache lives: GPU, CPU, L3, or several at once. L1 and L2 are private to one inference instance (host memory of several machines cannot be pooled into one L2), and L3 is the only tier that can be shared across instances, through backends including Mooncake, DeepSeek 3FS, NIXL, AIBrix KVCache, and a plain file directory; LMCache is available as an alternative layer.^[6]

The workflow is a local match against L1 and L2, a prefetch from L3 into L2 that triggers when the L3 hit exceeds a threshold (256 tokens by default), and a write-back after prefill. Three prefetch termination policies exist (best_effort, wait_complete, and timeout) and three write policies (write_through, write_through_selective, and write_back). For the CPU-to-GPU path, HiCache overlaps loading layer N+1 while computing layer N, and it adds GPU-assisted I/O kernels that its documentation says reach up to 3x the transfer speed of plain cudaMemcpyAsync. For host and storage tiers HiCache supports "page first" and "page first direct" layouts that keep all of a page's KV data contiguous, alongside the layer-first layout the GPU kernels expect.^[6] In the September 2025 announcement, the SGLang team reported up to 6x throughput and up to 80 percent lower time-to-first-token on its own long-context and multi-turn benchmarks.^[18]

TensorRT-LLM

TensorRT-LLM's block-pool KV cache system supports reuse across requests and, in its documentation's words, "uses a suite of tools like offloading and prioritized eviction to increase reuse". Before a block is evicted from GPU memory it can be copied to host memory, where it remains in the reuse tree until evicted from there; a reused block is copied back first. Host offloading is enabled by host_cache_size, the number of bytes of host memory to reserve, which defaults to 0. Blocks carry a retention priority (default 35), and secondary_offload_min_priority prevents low-priority blocks from being offloaded at all so that they are dropped without generating host traffic.^[7]

LMCache

LMCache is a standalone KV cache layer that "extracts and stores KV caches generated by modern LLM engines (vLLM and SGLang) out of the GPU memory and shares them across engines and queries". Its October 2025 paper presents it as supporting both offloading for prefix reuse and prefill-decode disaggregation, built on batched data movement with compute and I/O pipelining and a modular connector component; the authors report up to 15x throughput gains combined with vLLM on multi-round question answering and document analysis.^[13] Its documentation lists CPU RAM, local disk, Redis and Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS as storage backends.^[19]

NVIDIA Dynamo

NVIDIA Dynamo's KV Block Manager (KVBM) tiers blocks "GPU (G1) -> host/CPU (G2) -> disk (G3) -> object storage (G4)", each tier enabled by giving it a size. Its reference warns that a CPU cache smaller than the engine's GPU cache causes churn, offloading blocks after every forward pass. Offload policies between tiers are lists of pass_all, presence, or presence_lfu rules, and transfers run over NIXL backends such as UCX, POSIX, and GDS.^[12] Dynamo's current offloading tutorial has users pick one of four backends per worker: KVBM (its native block manager, which the page calls the recommended starting point for vLLM and which also serves TensorRT-LLM), LMCache for vLLM, HiCache for SGLang, or FlexKV (an open-source runtime from the taco-project organisation with host, SSD and cloud-storage tiers), and states that they are alternatives rather than layers to stack.^[20]

Mooncake and DeepSeek's persistent cache

Mooncake, the serving platform behind Kimi, pushed the tier model across a whole cluster. Its FAST 2025 paper (the conference's Best Paper) describes "a KVCache-centric disaggregated architecture" that separates prefill and decode clusters and "efficiently utilizes the underexploited CPU, DRAM, SSD and NIC resources of the GPU cluster to establish a disaggregated KVCache"; the authors report 59 to 498 percent more effective request capacity on real traces within latency objectives.^[15]

DeepSeek turned storage-tier offloading into an API product on August 2, 2024, when it introduced "Context Caching on Disk": prefixes expected to be reused are cached on "a distributed disk array", cache hits are billed at a lower rate, the storage unit is 64 tokens, and the company credited the MLA architecture of DeepSeek-V2 with shrinking the cache enough that it was "enabling efficient storage on low-cost disks".^[14] The DeepSeek-V4.1-Flash technical report of September 2026 shows what a mature persistent tier looks like: global KV is kept on SSD under LRU eviction with a guaranteed lifetime of at least 72 hours, while sliding-window KV, which the report says accounted for nearly half of the persistent capacity in the V4 deployment yet is reused only within a minutes-long session window, was moved to a host-DRAM pool with a short time-to-live and, on a miss, rebuilt approximately by replaying only the most recent window of tokens rather than the window multiplied by the layer count. DeepSeek reports that the change cut its persistent cache footprint to one-eighth of V4's, and that its FP4 main KV format "nearly halves the storage footprint, both in HBM and when offloaded to SSD".^[21]

Library-level offloading in Transformers

The Hugging Face Transformers library exposes a simpler, single-request form. Setting cache_implementation="offloaded" (or "offloaded_static") moves the KV cache of every layer except the one being computed to the CPU, prefetches the next layer's cache asynchronously, and sends the current layer's cache back after attention. The documentation positions it as a remedy for out-of-memory errors on small GPUs at the cost of "a small degradation in generation throughput".^[22]

Streaming everything: FlexGen

FlexGen, published at ICML 2023 by Ying Sheng, Lianmin Zheng, and colleagues, is the reference design for throughput-oriented offloading of an entire model on one GPU. It treats weights, activations, and the KV cache as three tensor classes that can each be placed in GPU, CPU, or disk memory, uses a linear-programming cost model to search for a placement and a "zig-zag block schedule" that overlaps I/O with computation, and compresses both weights and KV cache to 4 bits to reduce I/O. On a single 16 GB NVIDIA T4 with 208 GB of CPU DRAM and a 1.5 TB SSD, the authors ran OPT-175B at 1 token per second with an effective batch size of 144, against baselines (DeepSpeed Zero-Inference and Hugging Face Accelerate) that could not exceed a batch size of 2 without running out of memory.^[1] FlexGen accepts latencies of thousands of seconds per batch; its design target is batched offline work, not interactive serving.

Fetching a sparse subset per step

The third family keeps the full cache in host memory during decoding and moves only the entries the current step will attend to. This only works if the model can decide, cheaply and early enough, which entries matter.

InfiniGen

InfiniGen (Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim of Seoul National University, OSDI 2024) introduced speculative prefetch. Its insight is that "a few important tokens that are essential for computing the subsequent attention layer" can be predicted "by performing a minimal rehearsal with the inputs of the current layer and part of the query weight and key cache of the subsequent layer". During prefill it skews the query and key matrices to concentrate importance in a few columns and generates partial weights; during decoding, at layer i-1 it rehearses layer i's attention with those partial weights and a partial key cache and prefetches only the KV entries the rehearsal ranks highly. Implemented on FlexGen, it reported up to 3.00x speedup over prior KV cache management methods with up to 32.6 percentage points higher accuracy on the authors' tasks.^[2] SparDA's later measurements place InfiniGen well below a plain sparse-offload baseline in decode throughput and attribute this to gathering the selected blocks on the CPU before transfer.^[3]

ShadowKV

ShadowKV (Hanshi Sun and colleagues, first posted October 2024) splits the cache by tensor. Observing that pre-RoPE keys are strongly low-rank while values are not, it keeps a low-rank projection of the key cache on the GPU together with per-chunk "landmarks" and a small set of outlier chunks (0.2 to 0.3 percent of chunks), and offloads the value cache to the CPU. At each decode step landmarks select the chunks to attend to under a sparse budget of 1.56 percent of tokens, the selected keys are reconstructed from the low-rank store, and the matching values are fetched from the CPU, with the two overlapped on separate CUDA streams. The authors report up to 6x larger batch sizes and up to 3.04x higher throughput on an A100 across Llama-3.1-8B, GLM-4-9B-1M, and other long-context models "without sacrificing accuracy".^[8]

Trainable and native offloading

Training-free sparse fetch has a known failure mode that the NOSA paper names directly: "it often degrades long-generation quality due to training-inference mismatch on sparse patterns", because the model was never trained to attend sparsely. Trainable sparse attention fixes the mismatch but, in NOSA's words, "is incompatible with efficient offloading, as unconstrained KV accesses may force large CPU-to-GPU transfers and erase throughput gains".^[4] Two 2025-2026 designs built on the InfLLM v2 sparse attention of MiniCPM4 address that gap by putting the transfer budget into the architecture.^[3][4]

NOSA

NOSA (Native and Offloadable Sparse Attention, Yuxiang Huang of Tsinghua University and colleagues including OpenBMB researchers, first posted October 2025 and expanded in January 2026) starts from InfLLM v2, the trainable block-sparse attention of MiniCPM4, and measures strong locality in its block selection across consecutive decode steps, so that blocks selected in one step can be kept on the GPU and reused in the next. It then decomposes selection into a query-aware component and a query-agnostic component and applies an eviction policy to the query-agnostic part "to bound the number of KV blocks fetched from the CPU". The companion inference system, NOSI, uses a Triton kernel that reads host memory directly through Unified Virtual Addressing, which the authors measured at up to 83 percent of peak PCIe bandwidth against under 2 GB/s for a plain PyTorch transfer path. They train 1B, 3B, and 8B models and, on the 8B model, report decode throughput up to 5.04x that of full attention, 1.92x InfLLM v2, and 1.83x ShadowKV.^[4]

SparDA

SparDA (Sparse Decoupled Attention), posted in June 2026 by Yaosheng Fu, Guangxuan Xiao, Xin Dong, Song Han, and Oreste Villa (affiliations NVIDIA, Thinking Machines Lab, ByteDance Seed, and MIT, with two authors' work done at NVIDIA), attacks the latency of the fetch itself. It adds a fourth per-layer projection, the Forecast, beside query, key, and value. The Forecast of layer l is trained, with an objective the authors say is inspired by DeepSeek Sparse Attention's indexer, to predict which KV blocks layer l+1 will select, so the runtime can begin fetching those blocks from pinned CPU memory on a dedicated CUDA stream while layer l is still executing. The paper contrasts this with InfiniGen, which "relies on the raw hidden state as a proxy for future attention and can be inaccurate when adjacent-layer similarity breaks down". A persistent Triton kernel using Unified Virtual Addressing keeps a fixed set of thread blocks servicing transfer tasks, with the count chosen per batch size because more copy threads approach the PCIe ceiling but take streaming multiprocessors from attention and feed-forward work. The Forecast projections add 33.5 million parameters, 0.41 percent of an 8B model, and are trained alone on top of MiniCPM4.1-8B and NOSA-8B without retraining the base model.^[3]

On an H100 at 128K context the authors report up to 1.25x prefill and 1.69x decode speedup over the sparse-attention offload baseline on MiniCPM4.1-8B (1.16x and 1.40x on NOSA-8B). The larger figure comes from capacity: the no-offload sparse baseline runs out of memory beyond a batch of 4 at 128K, while SparDA with offloading reaches a batch of 64 at 1,000.1 tokens per second, 5.28x the no-offload sparse baseline's 189.5 and 9.21x dense attention's 108.6. The paper also records that offloading itself has negligible effect on prefill throughput because the only prefill-time transfer is an asynchronous writeback of newly created KV to the CPU.^[3] The authors describe SparDA as an add-on whose accuracy is bounded by the underlying sparse method, and name token-level DSA in DeepSeek-V3.2 and GLM-5 as targets for future work.^[3]

Comparison of systems

Trade-offs

Transfer volume against link bandwidth. The worked example above sets the scale: a dense read of a 4 GiB cache over PCIe Gen5 takes tens of milliseconds, which is why prefix-reload systems pay the transfer once per request and sparse-fetch systems limit the fraction of the cache read per step. NOSA's analysis concludes that even with locality, "PCIe bandwidth limitations are difficult to overcome, making decoding become communication-bound", which motivated bounding the fetched volume in the architecture rather than in the runtime.^[4] SparDA's throughput tables show the other side of the same constraint: within the offload regime, its largest gains appear at intermediate batch sizes "where prefetch and layer execution are roughly balanced", and its own kernel takes streaming multiprocessors away from computation as it tries to approach the PCIe ceiling.^[3]

Granularity and layout. Small, fragmented copies waste the link. vLLM's measurement that DMA throughput collapses for small blocks led it to make each logical block's KV data for all layers contiguous, and HiCache decoupled its host layout from the GPU's layer-first layout for the same reason.^[6][16]

Prefetch accuracy. A sparse-fetch system is only as good as its prediction of which entries the next step needs. InfiniGen predicts from the previous layer's hidden state; SparDA argues that this proxy fails when adjacent layers disagree and replaces it with a trained projection.^[2][3] For storage tiers, HiCache's prefetch termination policies encode a different accuracy question: whether to wait for a slow prefetch to complete, which raises the hit rate, or to start computing with what has arrived, which protects latency.^[6]

Eviction and retention. What is kept in each tier is a policy choice. vLLM offers LRU and ARC on its host tier and offloads only prompt blocks by default; TensorRT-LLM gates offloading on block priority; Dynamo's KVBM filters on presence and frequency counts; DeepSeek keeps global KV for 72 hours but gives sliding-window KV a lifetime of minutes.^{[5][7][12][21]} Sizing matters too: vLLM's guide notes that a CPU tier smaller than the aggregate GPU cache "just mirrors what the GPU already holds and adds no hit rate".^[5]

Quality. Prefix reload is exact: the reloaded blocks are the same tensors that were computed. Sparse fetch is approximate, and NOSA documents the quality loss when a model trained densely is served sparsely.^[4] SparDA reports that its trained Forecast matched or slightly improved the sparse baseline's average scores on the models it tested, and that InfiniGen "suffers significant accuracy degradation" on the same tasks.^[3] DeepSeek's bounded replay is a deliberate third position: an approximate reconstruction of dropped state that the company judged cheap enough to justify not persisting it.^[21]

Recompute versus fetch. Every offloading design contains a point at which recomputation is cheaper than transfer. vLLM's max_load_tokens lets a request cap loads and recompute the rest; DeepSeek replaced an exact recovery that required replaying the window times the layer count with a bounded replay of one window; and the vLLM blog's numbers show the trade shifting with prompt length, from a 2x time-to-first-token saving on short prompts to 22x on long ones.^[5][16][21] The same question governs GPU-to-GPU cache movement in disaggregated serving.

Security. Offloaded blocks persist beyond the request that made them, on hosts and disks that may be shared. DeepSeek's API announcement states that each user's cache is isolated and that unused entries are cleared automatically; vLLM's filesystem tier names block files by content hash so that identical token content produces identical filenames across instances, which is what makes cross-instance sharing work and also what makes access control on the shared directory necessary.^[5][14]

See also

• KV cache
• Prefix caching
• KV-cache quantization
• H2O KV eviction
• Sparse attention
• Disaggregated serving
• PCI Express
• NVLink

References

1. ^1 ^2 ^3 ^4Sheng, Y., Zheng, L., Yuan, B., et al. "FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU." Proceedings of the 40th International Conference on Machine Learning, PMLR 202:31094-31116, 2023. Source
2. ^1 ^2 ^3 ^4 ^5Lee, W., Lee, J., Seo, J., and Sim, J. "InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management." 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024. Source
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Fu, Y., Xiao, G., Dong, X., Han, S., and Villa, O. "SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference." arXiv:2606.04511, June 2026. Source
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Huang, Y., Wang, P., Han, J., et al. "NOSA: Native and Offloadable Sparse Attention." arXiv:2510.13602, October 2025 (v2 January 2026). Source
5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11vLLM project. "KV Offloading Usage Guide." vLLM documentation, docs/features/kv_offloading_usage.md, accessed September 16, 2026. Source
6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11SGLang project. "HiCache System Design and Optimization." SGLang documentation, accessed September 16, 2026. Source
7. ^1 ^2 ^3 ^4 ^5NVIDIA. "KV Cache System." TensorRT LLM documentation (1.3.0rc26), accessed September 16, 2026. Source
8. ^1 ^2 ^3Sun, H., Chang, L.-W., Bao, W., et al. "ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference." arXiv:2410.21465, October 2024 (v3 April 2025). Source
9. ^1 ^2OpenBMB. "openbmb/MiniCPM4.1-8B config.json." Hugging Face model repository. Source
10. ^1 ^2 ^3 ^4 ^5NVIDIA. "NVIDIA H100 Tensor Core GPU" product specifications. Source
11. ^1 ^2 ^3NVIDIA. "NVIDIA Grace Hopper Superchip Architecture In Depth." NVIDIA Technical Blog. Source
12. ^1 ^2 ^3 ^4 ^5 ^6 ^7NVIDIA. "KVBM Configuration Reference." NVIDIA Dynamo documentation, accessed September 16, 2026. Source
13. ^1 ^2 ^3Liu, Y., Cheng, Y., Yao, J., et al. "LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference." arXiv:2510.09665, October 2025 (v2 December 2025). Source
14. ^1 ^2 ^3 ^4DeepSeek. "DeepSeek API introduces Context Caching on Disk, cutting prices by an order of magnitude." DeepSeek API Docs, August 2, 2024. Source
15. ^1 ^2 ^3Qin, R., Li, Z., He, W., et al. "Mooncake: Trading More Storage for Less Computation - A KVCache-centric Architecture for Serving LLM Chatbot." 23rd USENIX Conference on File and Storage Technologies (FAST 25), 2025. Source
16. ^1 ^2 ^3 ^4Ozeri, O., and Harnik, D. "Inside vLLM's New KV Offloading Connector: Smarter Memory Transfer for Maximizing Inference Throughput." vLLM Blog, January 8, 2026. Source
17. ^vLLM project. "vllm/config/offload.py" (configuration for model weight offloading). GitHub, accessed September 16, 2026. Source
18. ^1 ^2Xie, Z. "SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends." LMSYS Org blog, September 10, 2025. Source
19. ^1 ^2LMCache. "Welcome to LMCache!" LMCache documentation, accessed September 16, 2026. Source
20. ^NVIDIA. "Offload KV Cache Locally." NVIDIA Dynamo documentation, accessed September 16, 2026. Source
21. ^1 ^2 ^3 ^4 ^5DeepSeek-AI. "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression." Technical report, September 2026, Sections 2.4 and 3.2. Source
22. ^1 ^2Hugging Face. "KV cache strategies." Transformers documentation (main), accessed September 16, 2026. Source
23. ^Kwon, W., Li, Z., Zhuang, S., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." Proceedings of the 29th ACM Symposium on Operating Systems Principles, 2023, Section 1. Source