Prompt Cache in LLMs: From Inference Optimization to Commercial Bottleneck (2026 Vendor Strategies Compared)

Background: From Inference Optimization to Commercial Bottleneck

The technical principle behind Prompt Cache is not complex. During Transformer autoregressive generation, every new token requires attention computation over all preceding tokens. Without caching, recomputing the full history of K and V tensors at every step yields O(n^2) complexity. With KV Cache enabled, previously computed K/V tensors are reused, reducing per-token complexity to O(n).

This technique has existed since the GPT-2 era, but between 2024 and 2026 it suddenly became an industry focal point because it graduated from a pure inference optimization into a commercial bottleneck.

Core insight: LLM prompts contain large amounts of repeated text segments — system messages, prompt templates, context documents, and few-shot examples. These fragments are sent to the model across many requests. Prompt Cache precomputes and stores the attention states (KV Cache) of these fragments. When the same fragment reappears, the cached state is reused directly, and only the new portion is computed.

From KV Cache to Prompt Cache

Traditional KV Cache only handles caching within a single request — K/V of preceding tokens is stored, and the next token reuses it directly. Prompt Cache extends this to the cross-request and cross-session level: when user A submits a system prompt, and user B submits the same one, that prompt's attention state is already computed and can be loaded directly.

The Yale paper *Prompt Cache: Modular Attention Reuse for Low-Latency Inference* provides a prototype implementation achieving 8x latency reduction on CPU and 60x latency reduction on GPU, without modifying model parameters.

Key challenge: Transformer positional encoding embeds position information into each token's attention state. If the same text segment appears at different positions, its attention state differs as well, preventing direct reuse. The paper's solution is Prompt Markup Language (PML) — a schema that defines reusable "prompt modules," each assigned a unique position ID independent of global position. Experiments show that LLMs can handle attention states with non-contiguous position IDs, provided that the relative positions of tokens remain consistent, without degrading output quality.

2026 Vendor Prompt Cache Strategies Compared

DeepSeek V4: Compression as Justice

• 90% KV cache volume compression via hybrid attention architecture combining CSA (Cache-Strided Attention) and HCA (Hybrid Cache Attention).
• 1M context as standard across the lineup, not a premium tier.
• Hit pricing at 10% of standard input cost for cached tokens.

DeepSeek's strategy combines architectural compression with price leverage. By shrinking KV cache volume to one-tenth of its original size, long context shifts from a "VRAM killer" to an "affordable option."

Anthropic Claude: Predictable Enterprise Contract

• cache_control: API-level explicit caching controls letting developers mark which parts are cacheable.
• 4 cache breakpoints: prompts can be divided into multiple cacheable segments rather than one monolithic prefix.
• Cross-session global sharing, 1-hour TTL: the same cache can be reused across different sessions.
• Hit pricing at 1/10 of standard input cost.

Anthropic's strategy is predictability. Caching is not a black box but a commitment written into the API contract. Boris Cherny, creator of Claude Code, has acknowledged: "With the 1M context window, the cost of a cache miss is very high. If you leave your computer for over an hour and then resume an old session, you usually miss the cache entirely."

OpenAI GPT-5: Automatic Caching, Default 50% Off

• Automatic caching: hidden behind the API; developers do nothing extra.
• Hit pricing at 50% of standard input cost.
• No explicit cache control markers; hit rates determined automatically by the system.

OpenAI prioritizes ease of use. Developers do not need to learn a new API; the system decides what to cache. The tradeoff is lower transparency and controllability.

Google Gemini: Object-Oriented Abstraction

• CachedContent API: an object-oriented abstraction for enterprise-grade long-content reuse.
• 2M context window: the largest nominal context currently advertised.
• Explicit cache management: developers must explicitly create, update, and delete cache content.

Google's strategy is enterprise-grade management. CachedContent is a standalone API object with lifecycle control — well-suited to knowledge-base and document reuse in enterprise scenarios.

Comparison Table

| Vendor | Caching Strategy | Breakpoints | Hit Discount | TTL | Developer Control | |---|---|---|---|---|---| | DeepSeek | Architectural compression + automatic caching | Not disclosed | 10% | Not disclosed | Low | | Anthropic | Explicit cache_control | 4 | 10% | 1 hour | High | | OpenAI | Automatic caching | None | 50% | Not disclosed | Low | | Google | Object-based API | Not disclosed | Not disclosed | Developer-controlled | Very High |

Key Points

• Prompt Cache moves KV reuse from within-request to cross-request/cross-session, driven by repetitive prompt fragments (system prompts, templates, few-shot examples).
• Yale's Prompt Cache paper (PML approach) reports 8x CPU and 60x GPU latency reduction without model changes; non-contiguous position IDs preserve output quality.
• DeepSeek bets on architectural compression (CSA/HCA, 90% volume reduction) plus aggressive 10% hit pricing.
• Anthropic prioritizes explicit, contract-level caching with 4 breakpoints, cross-session sharing, 1-hour TTL, and 10% hit pricing.
• OpenAI favors zero-config automatic caching at 50% hit pricing, sacrificing transparency.
• Google exposes CachedContent as an object API with explicit lifecycle control and a 2M context window.
• No unified standard exists: the four vendors pursue compression, contract clarity, ease of use, and enterprise object management respectively — an ongoing caching arms race.