Citation and evidence

Prompt Caching

37 min full readUpdated 25 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

Prompt caching is a large language model (LLM) inference optimization that stores the computed key-value (KV) state of a repeated prompt prefix so it can be reused across API calls, cutting both cost and latency for the cached portion. The most-cited figures come from Anthropic, which said at launch that prompt caching can reduce costs by up to 90% and latency by up to 85% for long prompts [1]. When a prompt begins with the same tokens as a previous request (such as a system prompt, tool definitions, few-shot examples, or a shared document), the model skips recomputing the attention keys and values for those cached tokens and begins work only from the point where the prompts diverge. Every major LLM API provider now offers the feature. DeepSeek turned on disk-based context caching for all API users on August 2, 2024 [23], Anthropic launched prompt caching in public beta on August 14, 2024 [1], OpenAI added automatic prompt caching on October 1, 2024 at a 50% discount on cached input tokens [2], and Google says it introduced Gemini context caching in May 2024 [8]. As of 30 October 2026, cache reads cost between roughly 2% and 50% of the normal input price depending on provider and model, and both Anthropic and OpenAI's newest models charge a premium for writing to the cache [3][18][24].

Background: The KV Cache

To understand prompt caching, it is helpful to first understand the KV cache, which is the underlying mechanism that makes it possible.

In a transformer-based language model, each layer computes attention over the input sequence by projecting tokens into query (Q), key (K), and value (V) representations. During autoregressive generation, the model produces tokens one at a time. Without caching, generating each new token would require recomputing the keys and values for every previous token in the sequence, resulting in O(n^2) total computation across a sequence of length n.

The KV cache avoids this redundancy by storing the key and value tensors computed during previous forward passes. When generating the next token, the model only needs to compute the query, key, and value for the new token, then attend over the cached keys and values from all previous tokens. This reduces per-token computation from O(n) to O(1) (excluding the attention operation itself) and is essential for making autoregressive generation practical at scale [4].

The KV cache is a per-request resource: each inference request builds up its own KV cache as it processes the prompt and generates output tokens. Prompt caching extends this concept across requests, sharing cached KV representations between different API calls that happen to share the same prompt prefix. OpenAI's documentation describes the same idea: the prompt cache stores KV tensors, not the tokens themselves [18].

"What is Prompt Caching? Optimize LLM Latency with AI Transformers", video from IBM Technology, published February 7, 2026. Watch on YouTube.

How Does Prompt Caching Work?

Prompt caching operates on a simple principle: if two requests share the same initial tokens (the same prefix), the KV representations for those shared tokens are identical and do not need to be recomputed.

The workflow proceeds as follows:

  1. First request. The model processes the full prompt and generates the KV cache for all input tokens. The provider stores (or "writes") this KV cache, associating it with the exact token sequence.
  2. Subsequent requests. When a new request arrives whose prompt begins with the same tokens, the system detects the matching prefix and loads the stored KV cache. The model only needs to compute KV representations for the tokens that follow the cached prefix.
  3. Generation. The model generates output tokens using the combined KV cache (cached prefix plus newly computed tokens) as normal.

This means that the computational cost of processing the prompt is reduced in proportion to the fraction of tokens that are cached. For applications where a large system prompt or document is repeated across many requests, the savings can be very significant. Caching changes only how the input is processed: Anthropic states that the response is identical to what the model would return without prompt caching, and DeepSeek notes that output is still generated normally and remains subject to sampling settings such as temperature [3][22].

What Are the Requirements and Constraints?

Prompt caching works only for exact prefix matches. The cached tokens must be identical, in the same order, at the beginning of the prompt. If any token in the cached prefix changes, the cache is invalidated from that point forward. Anthropic's documentation says that by default the cache has a 5-minute lifetime, refreshed at no extra cost each time the cached content is used, and that cache hits require 100 percent identical prompt segments up to and including the block marked for caching [3].

Most providers enforce a minimum prefix length for caching to be effective, and caches have a limited time-to-live (TTL) before they expire. The figures below are from each provider's documentation as of 30 October 2026.

ProviderMinimum prefix lengthCache lifetimeCache control
Anthropic512 to 4,096 tokens depending on model5 minutes by default (refreshed on each hit) or 1 hourAutomatic (one top-level cache_control) or explicit breakpoints, up to 4 per request [3]
OpenAI (GPT-5.6 and later)1,024 visible input tokensMinimum 30 minutes by default; 12-hour minimum optional in the Responses APIImplicit or explicit breakpoints, up to 4 cache writes per request [18]
OpenAI (earlier models)Varies with request settingsin_memory (typically 5 to 10 minutes of inactivity, up to 1 hour) or 24hImplicit only [18]
Google (Gemini)2,048 tokens (Gemini 2.5) or 4,096 tokens (Gemini 3.1 Pro Preview, 3.6 Flash, 3.8 Flash, 4 Argon)Implicit: managed by Google; explicit: developer-set TTL, 1 hour by defaultImplicit on Gemini 2.5 and newer; explicit only on Gemini 3 and 2.5 models [7][21]
DeepSeekNot stated in the caching guideUnused cache is cleared, usually within a few hours to a few daysAutomatic, stored on disk [22]

Expanded article table

Provider Implementations

When Did Anthropic Launch Prompt Caching?

Anthropic announced prompt caching for the Claude API on August 14, 2024, making it one of the first major providers to offer the feature. It launched in public beta with support for Claude 3.5 Sonnet and Claude 3 Haiku, later expanding to Claude 3 Opus [1]. The feature became generally available on the Anthropic API on December 17, 2024 [1][16]. Anthropic's implementation gives developers explicit control over caching behavior [3].

At launch, Anthropic said that with prompt caching customers could "provide Claude with more background knowledge and example outputs" while "reducing costs by up to 90% and latency by up to 85% for long prompts" [1]. The launch post gave three example workloads:

Use case (Anthropic launch examples)Time to first token without cachingTime to first token with cachingCost reduction
Chat with a book (100,000-token cached prompt)11.5 s2.4 s (-79%)-90%
Many-shot prompting (10,000-token prompt)1.6 s1.1 s (-31%)-86%
Multi-turn conversation (10 turns with a long system prompt)~10 s~2.5 s (-75%)-53%

Expanded article table

These are Anthropic's own measurements from August 2024 [1].

Anthropic supports two approaches:

  • Automatic caching. Developers add a single cache_control field at the top level of the API request. The system applies the cache breakpoint to the last cacheable block and moves it forward as a conversation grows, so earlier turns are read from cache and only the newest content is written. Anthropic recommends it as the starting point for multi-turn conversations [3]. It launched on February 19, 2026 [16], a change one commentator described as fixing "the biggest hidden cost in AI agents" [5]. Automatic caching is available on every platform except the legacy Amazon Bedrock integration for Opus 4.6 and earlier models, which returns an error for a top-level cache_control field [3].
  • Explicit cache breakpoints. Developers place cache_control markers directly on individual content blocks (such as the system prompt or specific messages) to define exactly what gets cached. A maximum of 4 cache breakpoints is allowed per request, and when the two modes are combined, the automatic breakpoint uses one of the four slots [3].

How Anthropic matches cached prefixes

Anthropic's documentation describes three rules for explicit breakpoints. A cache write happens only at a breakpoint: marking a block writes one entry, a cumulative hash of the whole prefix ending at that block, so changing any earlier block produces a different hash. On a later request the system checks for an entry at the breakpoint and, if none exists, walks backward one block at a time looking for an entry that an earlier request already wrote. That lookback covers at most 20 positions per breakpoint; on the Claude API a run of consecutive tool_use blocks, or of consecutive tool_result blocks, counts as a single position [3].

Two practical consequences follow. A breakpoint placed on a block that changes every request (for example one containing a timestamp) writes a fresh entry each time and never produces a hit, because the lookback finds only earlier writes, not stable content. And a conversation that grows by 20 or more blocks between requests can push the previous write outside the lookback window, which is one reason to use a second breakpoint [3].

Other documented rules [3]:

  • The cache prefix is built in the order tools, then system, then messages.
  • The lifetime is measured from the start of the request that writes or reads the entry, so time spent streaming a long response counts against it.
  • A new entry becomes available only after the first response starts, so parallel requests that need a hit should wait for the first response.
  • When 1-hour and 5-minute breakpoints are mixed, the longer-TTL entries must come before the shorter ones.
  • Tool definitions, system blocks, text messages, images and documents in user turns, and tool use and tool results can be cached. Thinking blocks cannot be marked directly but are cached as part of earlier assistant turns, and empty text blocks cannot be cached.
  • Prompts shorter than the model's minimum are processed without caching and without an error; both cache_creation_input_tokens and cache_read_input_tokens are then 0.
  • There is no way to clear the cache manually.

What invalidates an Anthropic cache

Because the prefix is hierarchical, a change at one level invalidates that level and everything after it. According to Anthropic's documentation, changing tool definitions invalidates the entire cache; toggling web search or citations, or switching between fast and standard speed, invalidates the system and message caches; and changing tool_choice or adding or removing images invalidates the message cache. Changing the thinking configuration or the output_config.effort value always invalidates cached messages and, on some models, the tool and system caches as well [3].

Newer API features are designed to avoid these invalidations. On models that support it, a developer can append a role: "system" message partway through a conversation instead of editing the top-level system prompt, which leaves the cached prefix unchanged. A beta header (inline-tools-2026-09-15) lets a tool be added or redefined inside such a message rather than in the tools array [3][16].

Pricing multipliers and break-even

Cache writes incur a cost premium: writing to a 5-minute cache costs 1.25x the base input token price, while writing to a 1-hour cache costs 2x the base input token price. Cache reads (hits and refreshes) cost 0.1x (10%) of the base input token price on most models, 0.05x on Claude Opus 5.5 and Claude Sonnet 5.5, and 0.025x on Claude Fable 5.1 and Claude Mythos 5.1 [3]. Anthropic lowered Sonnet 5.5 cache reads from $0.20 to $0.10 per million tokens on October 7, 2026 [16]. The multipliers stack with other pricing modifiers such as the Batch API discount, long-context pricing, and data residency, and cache breakpoints themselves cost nothing [3].

Because of this pricing structure, a 5-minute cache pays for itself on its first cache hit: one write plus one read costs 1.25x + 0.1x = 1.35x the base price of the cached tokens, versus 2x to send the same content uncached twice. A 1-hour cache needs two reads (three requests in total) to come out ahead: 2x + 0.1x + 0.1x = 2.2x, versus 3x uncached (calculated from the multipliers in Anthropic's documentation [3]). Anthropic recommends the 1-hour cache for prompts reused less often than every 5 minutes but more often than every hour, and notes that cache hits are not deducted against rate limits [3]. The 1-hour duration stopped requiring a beta header on August 13, 2025 [16].

Anthropic Prompt Caching Pricing

ModelBase Input5-min Cache Write1-hour Cache WriteCache Read (Hit)Output
Claude Fable 5.1, Claude Mythos 5.1$10/MTok$12.50/MTok$20/MTok$0.25/MTok (0.025x)$50/MTok
Claude Opus 5.5$4/MTok$5/MTok$8/MTok$0.20/MTok (0.05x)$20/MTok
Claude Sonnet 5.5$2/MTok$2.50/MTok$4/MTok$0.10/MTok (0.05x)$10/MTok
Claude Haiku 5.5 (prompts up to 100,000 tokens)$0.10/MTok$0.125/MTok$0.20/MTok$0.01/MTok$0.50/MTok
Claude Haiku 5.5 (prompts over 100,000 tokens)$0.50/MTok$0.625/MTok$1/MTok$0.05/MTok$2.50/MTok
Claude Fable 5, Claude Mythos 5$10/MTok$12.50/MTok$20/MTok$1/MTok$50/MTok
Claude Opus 5, Opus 4.8, 4.7, 4.6, 4.5$5/MTok$6.25/MTok$10/MTok$0.50/MTok$25/MTok
Claude Sonnet 5$2/MTok$2.50/MTok$4/MTok$0.20/MTok$10/MTok
Claude Sonnet 4.6, Sonnet 4.5, Sonnet 4$3/MTok$3.75/MTok$6/MTok$0.30/MTok$15/MTok
Claude Haiku 4.5$1/MTok$1.25/MTok$2/MTok$0.10/MTok$5/MTok
Claude Opus 4.1, Opus 4$15/MTok$18.75/MTok$30/MTok$1.50/MTok$75/MTok
Claude Haiku 3.5$0.80/MTok$1/MTok$1.60/MTok$0.08/MTok$4/MTok

Expanded article table

MTok = million tokens. Pricing as listed in Anthropic's prompt caching documentation as of 30 October 2026 [3]. On Claude Haiku 5.5, which is priced by prompt length, cache reads and cache writes count toward a request's prompt length [3].

Minimum cacheable length

Anthropic's minimum prompt length for caching varies by model. The following thresholds apply on the Claude API, Claude Platform on AWS, Claude Models on Google Cloud (Google managed), and Microsoft Foundry; Amazon Bedrock publishes its own per-model minimums [3].

MinimumModels
512 tokensClaude Fable 5.1, Mythos 5.1, Opus 5.5, Opus 5, Sonnet 5.5, Fable 5, Mythos 5, Haiku 5.5
1,024 tokensClaude Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5, Opus 4.1, Opus 4, Sonnet 4
2,048 tokensClaude Mythos Preview, Opus 4.7, Haiku 3.5
4,096 tokensClaude Opus 4.6, Opus 4.5, Haiku 4.5

Expanded article table

Isolation and data retention

Prompt caches are never shared between organizations, even for identical prompts. On the Claude API, Claude Platform on AWS, and Microsoft Foundry, caches are also isolated by workspace within an organization; on Amazon Bedrock and Claude Models on Google Cloud (Google managed), isolation is at the organization level only. Cache keys are cryptographic hashes of the prompt up to the cache-control point [3]. Prompt caching is eligible for zero data retention: Anthropic says KV cache representations and hashes are held in memory only, not stored at rest, and are deleted promptly (though not immediately) after their 5-minute or 1-hour minimum lifetime [3]. In Claude Managed Agents, prompt caching is always on, with a 5-minute default lifetime [3].

When Did OpenAI Add Prompt Caching?

OpenAI introduced automatic prompt caching on October 1, 2024, for the latest versions of GPT-4o, GPT-4o mini, o1-preview, and o1-mini, plus fine-tuned versions of those models. Unlike Anthropic's approach at the time, there was no parameter to set. As OpenAI stated, "API calls to supported models will automatically benefit from Prompt Caching on prompts longer than 1,024 tokens," and developers could "get a 50% discount" on reused input tokens [2]. The cache matched the longest previously computed prefix, starting at 1,024 tokens and increasing in 128-token increments; caches were typically cleared after 5 to 10 minutes of inactivity and always removed within one hour of last use, and were not shared between organizations [2].

OpenAI later raised the discount for new model families. With the GPT-4.1 series in April 2025 it increased the prompt caching discount to 75% (up from 50%) [20], and GPT-5 was listed in September 2025 at $1.25 per million input tokens and $0.125 per million cached input tokens, a 90% discount [6].

Model familyCached input priceInput priceSource
GPT-4o (October 2024)$1.25/MTok (50% off)$2.50/MTok[2]
GPT-4o mini (October 2024)$0.075/MTok (50% off)$0.15/MTok[2]
GPT-4.1 family (April 2025)25% of input (75% off)Varies by model[20]
GPT-5 (September 2025)$0.125/MTok (90% off)$1.25/MTok[6]

Expanded article table

Current OpenAI behavior (GPT-5.6 and later)

OpenAI's current prompt caching guide splits behavior by model generation. For GPT-5.6 and later models, caching is no longer free to write: cache writes cost 1.25x the uncached input rate for the default 30-minute minimum lifetime, and the Responses API offers a 12-hour minimum lifetime at a 2.5x write rate. Reads cost 0.1x the input rate on most of these models and 0.05x on GPT-6.1 Sol and GPT-6.1 Terra, whichever lifetime was chosen. These are total rates rather than surcharges, and OpenAI gives the same arithmetic as Anthropic's 5-minute tier: one write and one full reuse cost 1.35x, versus 2x without caching [18]. The guide describes the overall saving as a cached-input discount of up to 95% [18].

These models also gained developer-controlled breakpoints. prompt_cache_options.mode selects implicit mode, where OpenAI places a breakpoint at the end of the latest eligible message, or explicit-only mode, where only blocks marked with prompt_cache_breakpoint are written and a request with no explicit breakpoints does no caching at all. Each request can create up to four cache writes, and an implicit breakpoint uses one of the four slots. Lookup checks the first 2 and latest 50 explicit breakpoints and, in implicit mode, up to 20 earlier eligible message endings. A prompt_cache_options.prewarm flag prepares the cache without generating output [18].

For models before GPT-5.6, only implicit caching is supported, there is no cache-write charge, and lifetime is set with prompt_cache_retention: in_memory entries typically last 5 to 10 minutes of inactivity and up to an hour, while 24h extended retention can keep entries for up to 24 hours. GPT-5.5 and GPT-5.5 Pro support only 24h, which typically keeps entries for around 30 minutes. Organizations without zero data retention default to 24h on models that support both options [18]. On those older models, a stable prompt_cache_key helps route related requests to the same machine; OpenAI says cached states live on individual machines, that traffic above 15 requests per minute can lead to overflow routing, and that each key should carry about 15 requests per minute in total. On GPT-5.6 and later, OpenAI handles routing itself and the key is used mainly for separate cache accounting, which OpenAI notes can also prevent users from probing whether another user's content was cached [18]. Caches are not shared across organizations and cannot be reused across regional processing boundaries [18].

Model (standard tier)Input30-min write12-hour writeCached input (read)Output
GPT-6 Astra$10.00/MTok$12.50/MTok$25.00/MTok$1.00/MTok$50.00/MTok
GPT-6.1 Sol$2.00/MTok$2.50/MTok$5.00/MTok$0.10/MTok$10.00/MTok
GPT-6.1 Terra$1.00/MTok$1.25/MTok$2.50/MTok$0.05/MTok$5.00/MTok

Expanded article table

Standard-tier rates from OpenAI's pricing page as of 30 October 2026; batch, flex, and fast tiers are priced differently [19].

How Does Google Context Caching Work?

Google calls the feature context caching. In a May 2025 post, Google's Logan Kilpatrick wrote that the company "pioneered context caching in May of 2024," offering a 75% saving on repeated context through explicit caching [8]. Google offers both explicit and implicit caching [7][21].

Explicit caching. Developers create a named cache object via the API, specifying the content to cache and a TTL; if no TTL is set, it defaults to 1 hour, and there are no minimum or maximum bounds. Billing combines the cached tokens (charged at a reduced rate when used in later prompts) with a storage charge based on how long the tokens are kept [21]. As of October 2026, explicit caching is still labelled beta (under v1beta), is available only through the generateContent API on Gemini 3 and 2.5 models, and is not supported in the Interactions API or on Gemini 4 Argon and later models [7][21].

Implicit caching. Rolled out on May 8, 2025, for the Gemini API, implicit caching automatically passes cache cost savings to developers without requiring them to create explicit cache objects. At launch it applied to Gemini 2.5 models with the same 75% token discount, and Google lowered the minimum request size to 1,024 tokens for 2.5 Flash and 2,048 tokens for 2.5 Pro [8]. Google's documentation now describes implicit caching as enabled by default for all Gemini 2.5 and newer models, with no cost-saving guarantee, while explicit caching guarantees the saving [7][21]. On Gemini 4 models, a multi-turn or agent request caches the full previous request only if it adds at least 2,050 tokens; for smaller additions, the last 2,050 tokens of the previous request are reprocessed at the standard input rate [7].

Gemini modelInput priceCached input priceStorage (explicit only)
Gemini 4 Argon$2.00/MTok through Nov 30, 2026; $4.00 from Dec 1, 2026$0.10/MTok; $0.20 from Dec 1, 2026None (implicit caching only)
Gemini 3.8 Flash$0.75/MTok through Dec 31, 2026; $1.50 from Jan 1, 2027$0.075/MTok; $0.15 from Jan 1, 2027$0.50/MTok/hour; $1.00 from Jan 1, 2027
Gemini 2.5 Pro (prompts up to 200k tokens)$1.25/MTok$0.125/MTok$4.50/MTok/hour
Gemini 2.5 Flash (text, image, video)$0.30/MTok$0.03/MTok$1.00/MTok/hour

Expanded article table

Standard paid-tier pricing on the Gemini Developer API as of 30 October 2026; Vertex AI pricing may differ [9].

The current minimum cacheable content for implicit caching is 2,048 tokens for Gemini 2.5 Pro and 2.5 Flash, and 4,096 tokens for Gemini 3.1 Pro Preview, Gemini 3.6 Flash, Gemini 3.8 Flash, and Gemini 4 Argon [7]. For explicit caching, the maximum cacheable size equals the model's maximum input, and cached tokens count toward normal token limits [21].

How Does DeepSeek Context Caching Work?

DeepSeek announced "Context Caching on Disk" on August 2, 2024. Instead of keeping caches only in GPU memory, it stores content expected to be reused on a distributed disk array, and the service was turned on for all users with no code changes. Cache hits were priced at $0.014 per million tokens at launch, which DeepSeek said cut API costs by up to 90%, and only identical prefixes starting from the first token count as hits [23].

DeepSeek's current guide says that, because of its sliding-window attention, each cached prefix is stored as an independent "cache prefix unit" and a request hits the cache only if it fully matches one. Units are persisted at the end of each request's user input and model output, when the system detects a prefix shared across several requests, and at fixed token intervals for long inputs or outputs. The cache is best-effort, takes seconds to build, and is cleared once unused, usually within a few hours to a few days. Responses report prompt_cache_hit_tokens and prompt_cache_miss_tokens [22].

DeepSeek modelCache hit (off-peak / peak)Cache miss (off-peak / peak)Output (off-peak / peak)
deepseek-flash (DeepSeek V4.1-Flash)$0.003 / $0.006 per MTok$0.15 / $0.30 per MTok$0.60 / $1.20 per MTok
deepseek-v4-pro (DeepSeek V4-Pro)$0.022 / $0.044 per MTok$0.66 / $1.32 per MTok$1.98 / $3.96 per MTok

Expanded article table

Prices as of 30 October 2026. Off-peak rates are half the peak rates; peak hours are 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays, excluding Chinese public holidays [24].

How Do Prompt Caching Implementations Differ Across Providers?

The following table summarizes the key differences between prompt caching implementations across providers, as documented on 30 October 2026:

FeatureAnthropic (Claude)OpenAI (GPT)Google (Gemini)DeepSeek
Launch dateAugust 2024October 2024May 2024 (explicit, per Google), May 2025 (implicit)August 2024
Cache typeAutomatic + explicit breakpointsImplicit only (pre-GPT-5.6); implicit + explicit (GPT-5.6+)Explicit + implicit (explicit not on Gemini 4)Automatic only
Minimum prefix512 to 4,096 tokens by model1,024 tokens (GPT-5.6+); varies on older models2,048 or 4,096 tokensNot stated
Cache write cost1.25x (5 min) or 2x (1 hour) base input1.25x (30 min) or 2.5x (12 hours) on GPT-5.6+; no premium on older modelsStandard input rate; explicit adds storageNo premium (miss price)
Cache read price0.1x base input (0.05x or 0.025x on some models)0.1x or 0.05x on GPT-5.6+; 50% to 90% off on older models0.05x to 0.1x on current modelsAbout 2% to 3.3% of the miss price
Storage costNoneNonePer MTok per hour (explicit only)None
Cache duration5 minutes or 1 hour30 minutes or 12 hours (GPT-5.6+); minutes to 24 hours (older)Configurable TTL (explicit), managed (implicit)Hours to days, best-effort
Break-evenFirst read (5-min), second read (1-hour)First read (30-min); immediate on older modelsDepends on storage cost vs. read savingsImmediate
Sources[3][18][19][7][9][21][22][24]

Expanded article table

What Is Prompt Caching Used For?

Prompt caching is most valuable in scenarios where a significant portion of the prompt remains constant across multiple API calls.

System Prompts

Most LLM applications include a system prompt that defines the model's role, behavior constraints, and output format. This system prompt is identical across every user interaction. Without caching, the model reprocesses these tokens on every request. With caching, the system prompt is processed once and reused, reducing both cost and latency for all subsequent requests [3].

For applications with large system prompts (thousands of tokens containing detailed instructions, personas, or tool definitions), the savings are proportionally larger. A 5,000-token system prompt that is repeated across 100 requests per minute would see its effective input cost reduced by roughly 90% for those tokens on a provider that bills reads at 0.1x.

Few-Shot Examples

Applications that use few-shot learning include multiple input-output examples in the prompt to guide the model's behavior. These examples are typically static and repeated across all requests. Prompt caching allows these examples to be processed once and reused, which is particularly valuable when using many examples or when examples include long text passages. Anthropic's documentation suggests that caching makes it practical to include 20 or more diverse examples of high-quality answers [3].

Document Analysis

When using an LLM to analyze a document (answering questions, extracting information, summarizing), the document content is included in the prompt. If multiple questions are asked about the same document across different API calls, prompt caching avoids reprocessing the document each time. This is directly relevant to retrieval-augmented generation (RAG) workflows where a retrieved context is queried multiple times [3][22].

Multi-Turn Conversations

In conversational applications, each new message in a conversation typically includes the full conversation history as context. As the conversation grows, the prompt becomes longer with each turn. Prompt caching helps because each new turn's prompt shares a long common prefix (the conversation history up to the previous turn) with the prior request. The model only needs to process the new user message and any new system content [3]. OpenAI's guide notes that reusing the growing history can save more input tokens than caching only the initial instructions, and warns that summarization, compaction, or truncation changes the prefix and resets reuse [18].

Agentic Workflows

AI agent systems that use tool calling and multi-step reasoning often make many sequential API calls with similar or overlapping prompts. The system prompt, tool definitions, and accumulated context tend to remain stable across calls within a single agent run. Prompt caching can significantly reduce the cost of these repetitive computations. Anthropic lists agentic tool use with multiple tool calls and iterative code changes among its target use cases, and its automatic caching mode was built for conversations that keep growing [3].

Implementation Strategies

How Should Prompts Be Structured for Caching?

To maximize cache hit rates, developers should structure their prompts so that static content appears at the beginning and variable content at the end. The render order on the Anthropic API is tools, then system, then messages, so the most stable content (frozen system prompt, deterministic tool list) belongs first [3]. A recommended ordering is:

  1. System prompt (static)
  2. Few-shot examples (static)
  3. Shared context or documents (semi-static)
  4. Conversation history (growing)
  5. Current user message (variable)

This structure ensures that the longest possible prefix remains cacheable across requests. Placing variable content before static content (e.g., putting the user message before the system prompt, or interpolating a timestamp into the system prompt) would prevent caching of everything that follows [3][18]. On Anthropic's API, the breakpoint belongs on the last block that stays identical across requests, not on a varying block after it [3]. OpenAI recommends keeping tool definitions and their order stable, disabling tools with tool_choice set to "none" rather than removing them, and restricting callable tools with allowed_tools while leaving the tools list unchanged [18].

Cache Warm-Up

Developers can "warm up" the cache by making an initial request that includes the content to be cached, so that later user-facing requests hit a warm cache. Anthropic supports this directly: a request with max_tokens: 0 reads the prompt, writes the cache at any cache_control breakpoint, and returns immediately with an empty content array and stop_reason: "max_tokens", billing the cache write but no output tokens. Such requests are rejected if they enable streaming, manual extended thinking, structured outputs, or forced tool use, and are not allowed inside Message Batches [3]. OpenAI offers the equivalent for GPT-5.6 and later through prompt_cache_options.prewarm in the Responses API [18]. For the default 5-minute Anthropic cache, a pre-warm must be repeated at least every 5 minutes to keep the cache alive [3].

Monitoring Cache Performance

All major providers return cache hit information in their API responses:

ProviderUsage fields
Anthropiccache_creation_input_tokens, cache_read_input_tokens, and input_tokens (only the tokens after the last breakpoint), plus a cache_creation object splitting writes into ephemeral_5m_input_tokens and ephemeral_1h_input_tokens [3]
OpenAI (Responses API)usage.input_tokens_details.cached_tokens and usage.input_tokens_details.cache_write_tokens [18]
Google (Gemini)usage.total_cached_tokens in the Interactions API (Python and JavaScript); the response's usage_metadata in the generateContent API, which Google's 2025 launch post said reports cached_content_token_count [7][8][21]
DeepSeekprompt_cache_hit_tokens and prompt_cache_miss_tokens [22]

Expanded article table

On Anthropic's API, total input equals cache_read_input_tokens + cache_creation_input_tokens + input_tokens, so input_tokens is typically much smaller than the real prompt when caching works [3]. Monitoring these fields allows developers to measure cache hit rates and optimize their prompt structures accordingly. If cache_read_input_tokens stays at zero across repeated identical-prefix requests, a silent invalidator (a timestamp, a random ID, or non-deterministic JSON serialization in the prefix) is usually at work; Anthropic's documentation specifically warns that some languages, such as Swift and Go, randomize key order when converting tool_use content to JSON [3].

Both Anthropic and OpenAI now provide diagnostic tools. Anthropic's cache diagnostics feature, launched in beta on May 13, 2026 and taken out of beta on September 23, 2026, compares a request with the previous response named in diagnostics.previous_message_id and reports where the prefix diverged (model, system prompt, tools, or message history). It stores only a fingerprint of hashes and token-count estimates, not prompt content [16][17]. OpenAI offers a Prompt Caching Dashboard and a Prompt Cache Diagnostics tool [18].

Prefix Caching in Open-Source Inference Engines

Prompt caching is not limited to proprietary APIs. Open-source LLM inference engines have implemented similar mechanisms, often called "prefix caching" or "automatic prefix caching" (APC).

vLLM

vLLM, the widely used high-throughput LLM serving engine, implements Automatic Prefix Caching (APC) that caches KV representations at the block level. vLLM's PagedAttention system manages the KV cache in fixed-size blocks, and APC identifies each block by a hash of the tokens in that block together with the hash of the prefix before it. When a new request arrives, vLLM checks if any prefix blocks match previously computed blocks and reuses them; only full blocks are cached, and unused blocks are evicted in least-recently-used order [10].

vLLM's APC requires exact token-level matches and works at block boundaries. The approach is deterministic: given the same token sequence, the same cache blocks will be matched. Since version 0.11, the default hashing algorithm is SHA-256, which addresses collision risks, and the hash can include extra values such as LoRA IDs, multimodal input hashes, and cache salts that isolate tenants [10].

SGLang

SGLang takes a different approach to prefix caching using a radix tree, a technique its authors call RadixAttention. SGLang keeps the KV cache of finished requests in a radix tree and manages it as an LRU cache, which allows efficient prefix search, reuse, insertion, and eviction at the token level rather than at fixed block boundaries [25]. This makes SGLang's caching more flexible, particularly for multi-turn conversations where the shared prefix length varies unpredictably [11].

SGLang's radix tree approach has been shown to be particularly effective for multi-turn conversation workloads, where the conversation history grows incrementally and the shared prefix length changes with each turn [11].

FeaturevLLM APCSGLang Radix Attention
Matching granularityBlock-level (full blocks only)Token-level
Data structureHash of block contents plus parent hashRadix tree
Best forBatch inference, templated promptsMulti-turn conversations, variable prefixes
Automatic discoveryYes (within block boundaries)Yes (any token boundary)
Memory managementPagedAttention blocks with LRU evictionRadix tree with LRU eviction

Expanded article table

LMCache

LMCache is an open-source KV caching layer that works with both vLLM and SGLang. It extracts and stores KV caches generated by the inference engine and moves them out of GPU memory to CPU memory, disk, or even a distributed cache. This allows KV caches to be shared across multiple GPU workers or even across different machines, enabling prefix caching at the cluster level rather than just the single-GPU level [12].

How Much Does Prompt Caching Save?

The performance benefits of prompt caching fall into two categories: cost reduction and latency reduction.

Cost Reduction

The cost savings are straightforward to calculate from the provider's pricing. The table below is an illustrative calculation, not a measured benchmark: a 50,000-token shared prefix plus 1,000 new tokens per request, sent 10 times within the cache lifetime, using list prices as of 30 October 2026. It assumes the first request writes the prefix and the next nine read it, with the breakpoint at the end of the shared prefix.

ScenarioWithout cachingWith cachingSavings on input
Claude Sonnet 5.5 (5-minute cache: $2 input, $2.50 write, $0.10 read)$1.020$0.127 first request + 9 x $0.007 = $0.190~81%
GPT-6.1 Sol (30-minute cache: $2 input, $2.50 write, $0.10 read)$1.020$0.190 (same rates as above)~81%
Gemini 2.5 Pro implicit caching ($1.25 input, $0.125 cached, no write premium), if all nine later requests hit$0.6375$0.06375 + 9 x $0.0075 = $0.13125~79%

Expanded article table

Prices from [3], [19], and [9]. Gemini implicit caching gives no guarantee of a hit [21].

Latency Reduction

Cached tokens do not need to go through the full forward pass of the model, which reduces the time-to-first-token (TTFT). Anthropic reported up to 85% latency reduction for long cached prompts at launch; in its own example, a 100,000-token cached prompt cut time to first token from 11.5 seconds to 2.4 seconds [1]. Anthropic's documentation says the 5-minute and 1-hour caches behave the same with respect to latency [3]. The actual latency improvement depends on the ratio of cached to uncached tokens and the model size. OpenAI cautions that the minimum lifetime governs eligibility for cache-read pricing and does not guarantee that every request reuses stored KV tensors or runs faster [18].

Semantic Caching Is a Different Technique

Despite the similar name, semantic caching does not reuse a model's KV tensors. A semantic cache stores earlier prompts together with their completed responses, represents prompts with embeddings, and may return a stored response when a new prompt is judged sufficiently similar. On a cache hit, the language model need not run at all.[13][14] Prompt or prefix caching instead requires an exact shared token prefix and still runs the model to generate a new continuation.[3][10]

That difference introduces a quality trade-off. Semantic similarity between two prompts does not guarantee that one response correctly answers both. Zhu, Zhu, and Jiao found that a general-purpose embedding model performed near chance on a deliberately difficult cache-reuse dataset, and improved the area under the receiver operating characteristic curve from 0.51 to 0.81 only after task-specific distillation.[14] Later work on vCache likewise found that one static similarity threshold could produce unpredictable error rates across prompts, motivating per-entry thresholds and explicit error bounds.[15] Semantic caching therefore requires application-specific evaluation, invalidation rules, and error monitoring. It should not be described as a transparent substitute for the exact-prefix mechanisms covered in this article.

Relationship to Other Optimization Techniques

Prompt caching is one of several techniques for reducing the cost and latency of LLM inference. It is complementary to most other approaches:

  • Quantization. Reduces model size and memory usage. Orthogonal to prompt caching; both can be applied simultaneously.
  • Speculative decoding. Accelerates output token generation using a smaller draft model. Complements prompt caching, which primarily accelerates input processing.
  • Batch processing. Groups multiple requests for efficiency. Anthropic's caching multipliers stack with its Batch API discount, and OpenAI's batch tier lists lower cached-input and cache-write prices than its standard tier [3][19]. Anthropic notes that cache hits inside a batch are best-effort because batch requests can run concurrently and in any order, and suggests writing a shared prefix to the 1-hour cache with a single request before submitting the rest [3].
  • Knowledge distillation. Trains a smaller model to mimic a larger one. An alternative to prompt caching for reducing costs, though with potential quality trade-offs.
  • Model routing. Directs simpler queries to smaller, cheaper models. Complementary to prompt caching, as both reduce costs through different mechanisms.

Current State (2025-2026)

As of 30 October 2026, prompt caching is a standard feature across all major LLM API providers. Several trends characterize the current landscape:

Automatic caching as the default. OpenAI, Google (for Gemini 2.5 and newer), and DeepSeek apply caching without configuration, and Anthropic added a single-field automatic mode in February 2026 while keeping explicit breakpoints [3][7][16][18][22]. At Google the direction is toward implicit caching only: explicit cache objects are not supported on Gemini 4 Argon and later [7].

Write premiums and explicit control spreading. Anthropic charged for cache writes from the start. OpenAI, which originally charged nothing for writes, now bills 1.25x or 2.5x for writes on GPT-5.6 and later models and lets developers place explicit breakpoints [3][18].

Deeper discounts for newer models. Read prices have kept falling relative to input prices: from 50% off for GPT-4o to 90% for GPT-5 and 95% for GPT-6.1 Sol and Terra at OpenAI, and from 0.1x to 0.05x (Opus 5.5, Sonnet 5.5) and 0.025x (Fable 5.1, Mythos 5.1) at Anthropic [2][3][6][18]. DeepSeek's cache-hit price for deepseek-flash is 2% of its cache-miss price [24].

Longer cache durations. Anthropic's 1-hour option (at a 2x write cost) and OpenAI's 12-hour option (at a 2.5x write cost) serve applications where requests are spaced further apart [3][18].

Cache-preserving API design. Providers are adding features that change a conversation without breaking its cached prefix, such as Anthropic's mid-conversation system messages and in-message tool definitions and OpenAI's configuration_update item for changing reasoning effort, along with diagnostics that explain cache misses [3][16][17][18].

Integration with agent frameworks. As AI agent architectures become more common, prompt caching has become essential for managing costs. Agent workflows that involve dozens of sequential LLM calls with overlapping context are among the biggest beneficiaries of caching [3].

Open-source convergence. vLLM, SGLang, and other open-source inference engines continue to improve their prefix caching implementations. LMCache and similar projects are extending caching across multi-GPU and multi-node deployments, bringing the benefits of prompt caching to self-hosted model serving [10][11][12].

For any application that repeatedly sends similar prompts to an LLM API, caching offers substantial cost savings with modest implementation effort, provided the prompt is structured so its stable prefix stays byte-for-byte identical.

Continue reading

More to read

Topics:AI InferenceDeveloper ToolsLarge Language ModelsMachine Learning

See also

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7Anthropic. (2024). "Prompt caching with Claude." anthropic.com/...prompt-caching (original August 14, 2024 version archived at web.archive.org/...prompt-caching)
  2. ^1 ^2 ^3 ^4 ^5 ^6OpenAI. (2024). "Prompt Caching in the API." openai.com/...api-prompt-caching
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48Anthropic. "Prompt caching." Claude API Documentation (accessed 30 October 2026). platform.claude.com/...prompt-caching
  4. ^Pope, R., Douglas, S., Chowdhery, A., et al. (2022). "Efficiently Scaling Transformer Inference." MLSys 2023. arxiv.org/...2211.05102
  5. ^Njenga, J. (2026). "Anthropic Just Fixed the Biggest Hidden Cost in AI Agents (Automatic Prompt Caching)." Medium. medium.com/...utomatic-prompt-caching-9d47c95903c5
  6. ^1 ^2 ^3OpenAI. "API Pricing." openai.com/...pricing (version of 2 September 2025 archived at web.archive.org/...pricing)
  7. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Google. "Context caching." Gemini API Documentation (last updated 23 October 2026). ai.google.dev/...caching
  8. ^1 ^2 ^3 ^4Kilpatrick, L. (2025). "Gemini 2.5 Models Now Support Implicit Caching." Google Developers Blog, May 8, 2025. developers.googleblog.com/...port-implicit-caching
  9. ^1 ^2 ^3Google. "Gemini Developer API Pricing" (accessed 30 October 2026). ai.google.dev/...pricing
  10. ^1 ^2 ^3 ^4vLLM Documentation. "Automatic Prefix Caching." docs.vllm.ai/...prefix_caching
  11. ^1 ^2 ^3Moon, D. (2025). "Prefix Caching: SGLang vs vLLM." Medium. medium.com/...-vs-block-level-hashing-b99ece9977a1
  12. ^1 ^2Liu, Y., Cheng, Y., Yao, J., et al. (2025). "LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference." arXiv:2510.09665. arxiv.org/...2510.09665
  13. ^Bang, F. (2023). "GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings." Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software, 212-218. aclanthology.org/2023.nlposs-1.24
  14. ^1 ^2Zhu, H., Zhu, B., and Jiao, J. (2024). "Efficient Prompt Caching via Embedding Similarity." arXiv:2402.01173. arxiv.org/...2402.01173
  15. ^Schroeder, L. G., Desai, A., Cuadron, A., et al. (2026). "vCache: Verified Semantic Prompt Caching." ICLR 2026. arxiv.org/...2502.03771
  16. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Anthropic. "Claude Platform release notes" (accessed 30 October 2026). platform.claude.com/...overview
  17. ^1 ^2Anthropic. "Cache diagnostics." Claude API Documentation (accessed 30 October 2026). platform.claude.com/...cache-diagnostics
  18. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23OpenAI. "Prompt caching." OpenAI API Documentation (accessed 30 October 2026). developers.openai.com/...prompt-caching
  19. ^1 ^2 ^3 ^4OpenAI. "Pricing." OpenAI API Documentation (accessed 30 October 2026). developers.openai.com/...pricing
  20. ^1 ^2OpenAI. (2025). "Introducing GPT-4.1 in the API." April 14, 2025. openai.com/...gpt-4-1
  21. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Google. "Context caching." Gemini Generate Content API Documentation (last updated 23 October 2026). ai.google.dev/...caching
  22. ^1 ^2 ^3 ^4 ^5 ^6 ^7DeepSeek. "Context Caching." DeepSeek API Docs (accessed 30 October 2026). api-docs.deepseek.com/...kv_cache
  23. ^1 ^2DeepSeek. (2024). "DeepSeek API introduces Context Caching on Disk, cutting prices by an order of magnitude." August 2, 2024. api-docs.deepseek.com/...news0802
  24. ^1 ^2 ^3 ^4DeepSeek. "Models & Pricing." DeepSeek API Docs (accessed 30 October 2026). api-docs.deepseek.com/...pricing
  25. ^Zheng, L., Yin, L., Xie, Z., et al. (2023). "SGLang: Efficient Execution of Structured Language Model Programs." arXiv:2312.07104. arxiv.org/...2312.07104

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

10 revisions · v10 · 7,382 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Pricing, cache lifetimes, minimum lengths and API fields checked against Anthropic, OpenAI, Google Gemini and DeepSeek documentation and release notes, plus Anthropic's archived August 2024 launch post, October 2026.

Cite this page: AI Wiki. "Prompt Caching." aiwiki.ai, updated 29 Oct 2026, fact-checked 29 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/prompt_caching

Edit article