Cross-Model KV Cache Transfer

Cross-model KV cache transfer is an experimental technique for converting the KV cache produced by one large language model into the cache representation expected by another model. The method was introduced in the August 2026 preprint Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse by nine NVIDIA researchers. Its purpose is to let a receiving model begin decoding without repeating the prompt's prefill computation.^[1]

The paper studies differently sized models within the Qwen3, Llama 3.1, and Ministral 3 families. Every evaluated source-target pair has the same number of KV heads and the same dimension per KV head. The authors call this condition "matched KV." Their results do not establish that the method works between unrelated model families, with mismatched KV geometry, or in production systems.^[1]

Scroll sideways for more →

Problem and scope

During autoregressive inference, a model first processes the full prompt in a prefill pass. Each transformer layer stores the resulting key and value tensors so later tokens can attend to the existing context without processing it again. Ordinary prefix caching can reuse this work when the same model sees the same prefix. It does not directly solve a model switch because different large language models produce different internal representations.^[1]

The paper asks whether a cache from a source model can be transformed into a cache that the target model can use. If the transformation is accurate enough, a router could switch models without running the target over the full prompt. The mapping is directional: transferring from model A to model B and transferring from B to A require separate fitted mappers.^[1]

The experiments deliberately narrow the problem. Source and target share a tokenizer because they belong to the same family. All tested models use dense, full attention, and all six small-to-large pairs have 8 KV heads of dimension 128. Layer count, hidden size, and total parameter count may differ. Although the regression formula can accept unequal source and target feature dimensions, the paper does not test mismatched KV head counts or per-head dimensions. Matched KV is therefore an empirical boundary of the evidence, not a general guarantee of compatibility.^[1]

Mapper design

The mapper is fitted offline from paired traces. The source and target models process the same calibration tokens, and the method learns how source KV features predict each target layer's keys and values. The target model's weights are not fine-tuned.^[1]

For each target layer, the procedure has four main parts:

1. It ranks source layers by the coefficient of determination, or R-squared, of a single-source linear probe. The score is averaged over heads and over keys and values.
2. It chooses the top k source layers and concatenates all KV heads from those layers. Each target KV head can therefore use information from every source KV head in the selected layers.
3. It removes rotary position embedding rotations from source keys, fits in position-independent content space, and applies the target model's rotation after mapping. Values are mapped directly because they do not carry RoPE rotations.
4. It fits a separate affine ridge regression for every target layer, target head, and key-or-value output. With centered inputs and targets, the slope has the closed form W = (X^T X + lambda I)^-1 X^T Y, and the intercept is recovered from the means.^[1]

For Qwen3 14B to 32B, the authors report that the best single source layer explained, on average, 56% of the variance in target keys and 32% in target values. Using eight selected source layers raised those production-ridge figures to 79% and 65%. These measurements are specific to that pair and its calibration setup; they do not show that the same linear structure exists in every model family.^[1]

The production setting in the paper used lambda = 0.01. The number of selected source layers was swept over 1, 2, 4, 6, 8, 10, 12, 16, 20, 24, and all available layers. A single k was then selected for each model pair using ARC-Challenge, HellaSwag, WinoGrande, and MMLU. In practice, WinoGrande was unavailable for most values of k, so ARC-Challenge, HellaSwag, and MMLU determined the selection. GSM8K, CoQA, and latency were held out from this choice. A leave-one-benchmark-out analysis changed reported held-out accuracy by at most 2.49 percentage points and by 0.30 points on average, but this was not a fully separate model-selection dataset.^[1]

Calibration and fitting

The main calibration set contained 500 FineWeb-Edu sequences of 1,024 tokens. Stride-4 subsampling yielded about 128,000 token observations per target head. Forward passes used bfloat16, covariance accumulation used float32, and analysis used float64. Fitting one directional mapper took 47 to 87 minutes on one node with eight H100 GPUs, depending on the pair.^[1]

The resulting mappers were large despite being linear. At the selected k, they contained 1.01 billion to 3.36 billion coefficients for keys and values together, requiring 4 GB to 12 GB of storage. For example, Qwen3 14B to 32B used 1.07 billion coefficients and 4 GB, while Llama 3.1 8B to 70B used 3.36 billion coefficients and 12 GB. A fleet of P models may need as many as P(P-1) directional mappers if every ordered pair is supported.^[1]

Reported transfer quality

The authors evaluated six small-to-large transfers. All numbers below are author-reported results from the August 2026 preprint, not production measurements or independently established performance guarantees. "Retention" means transfer accuracy divided by the target model's standalone-prefill accuracy. Because benchmark chance levels differ, the paper also reports floor-normalized retention, which maps chance performance to 0% and the target's standalone score to 100%.^[1]

Scroll sideways for more →

Four of the six pairs retained approximately 73% to 98% of the target's standalone accuracy averaged across the five benchmarks. The two transfers into Ministral 3 14B retained only 41.6% and 44.2% on that average, or 11.1% and 14.7% after chance-floor normalization. The large spread shows why matching head count and head dimension should not be interpreted as sufficient for a reliable transfer.^[1]

The paper also examined what predicts success. Across six pairs evaluated in both directions on HellaSwag, the cosine similarity between mapped and ground-truth attention outputs correlated with retention at Pearson r = +0.57. Calibration-domain key R-squared had r = -0.20. This is a post-hoc association across 12 directional evaluations, not evidence that attention-output cosine can predict an untested pair before a mapper is fitted.^[1]

Nonlinear extension

The authors replaced ridge regression with a multilayer perceptron for four Qwen3 and Ministral 3 pairs. Each target layer, head, and key-or-value output received its own network with two 1,024-unit ReLU hidden layers. The networks used Adam for 20 epochs, a learning rate of 0.001, batches of 4,096, and mean squared error on the same calibration data.^[1]

This experiment measured only HellaSwag retention. It does not show that a nonlinear mapper repairs the failing pairs on MMLU, GSM8K, ARC-Challenge, WinoGrande, long-context workloads, or production traffic.^[1]

Scroll sideways for more →

The nonlinear networks slightly underperformed ridge on the two already successful cases and sharply improved the two HellaSwag failures. The largest gain, 36.8 percentage points, is the basis for the paper's rounded "up to 37 points" statement. It is specific to this four-pair HellaSwag comparison.^[1]

Latency and multi-turn tests

The paper reports that mapper application was 2.7 to 25.1 times faster than target re-prefill across its small-to-large latency sweep. These are measurements of two isolated compute paths on synthetic inputs, not complete user-request latency. The setup used one eight-H100 node with NVLink, bfloat16, 50 warmup trials, and 30 timed trials per cell. Source-cache transfers between GPUs were included. Re-prefill ran the target transformer body with flash_attention_2 and excluded the language-model head; the mapper ran eagerly without torch.compile or CUDA graphs. Shipping the mapped cache to a separate target process was not measured.^[1]

The detailed Qwen3 14B to 32B results illustrate how the reported speedup changed with sequence length:

Scroll sideways for more →

Multi-turn behavior was tested only on Qwen3 14B and 32B, in both directions. The evaluation used 100 CoQA conversations of about 15 turns across five domains and reported the first ten turns. For 14B to 32B, the gap from the target's standalone result widened by 1.7 percentage points from turn 1 to turn 10. For 32B to 14B, drift increased by 0.33 points per turn. The authors did not observe cascading failure within ten turns for this single model pair, but noted that the measured large-to-small drift would accumulate in longer sessions.^[1]

Relationship to other methods

Cross-model cache work covers different goals and compatibility assumptions:

Scroll sideways for more →

CacheBridge directly evaluated the two Ministral directions where the original linear mapper degraded. Its authors report recovering those cases, obtaining 99.83% mean target retention on the tested Qwen direction, reducing Qwen3 14B-to-32B mapper storage eightfold, and reducing 500-sequence construction time from 92.63 seconds to 8.63 seconds. Those figures are claims from a separate September 2026 preprint and should not be read as peer-reviewed or production validation of either approach.^[6]

Limitations

Scroll sideways for more →

The method is best understood as evidence that some matched, within-family KV representations admit a useful affine conversion. The paper does not establish universal cache interchangeability among grouped-query attention models or validate the technique in a production serving system.^[1]

References

1. ^Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, and Bita Darvish Rouhani. Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse. arXiv:2608.03893v1, August 4, 2026.
2. ^Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, Junchen Jiang, Shan Lu, Madan Musuvathi, and Esha Choukse. DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants. 23rd USENIX Symposium on Networked Systems Design and Implementation, May 2026, pp. 319-338.
3. ^Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, and Yu Wang. Cache-to-Cache: Direct Semantic Communication Between Large Language Models. International Conference on Learning Representations, 2026.
4. ^Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, and Arthur Szlam. Latent Space Communication via K-V Cache Alignment. arXiv:2601.06123v1, January 4, 2026.
5. ^Yi Zhao, Zuchao Li, and Hai Zhao. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025, pp. 19522-19533.
6. ^Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, and Tao Lin. CacheBridge: Efficient Cross-Model KV Cache Transfer. arXiv:2609.00891v1, September 1, 2026.