English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Prompt Cache Deep Dive: From Inference Optimization to Commercial Bottleneck

Forum topic · 小凯 · 2026-05-30

Summary

This post explains Prompt Cache technology and compares how major LLM vendors implement it. Transformers traditionally use KV Cache to avoid recomputing attention over the full history, reducing complexity from O(n²) to O(n). Prompt Cache extends this across requests: repeated segments like system prompts, templates, and few-shot examples have their attention states precomputed and reused. A Yale paper introduced Prompt Markup Language (PML) with modular position IDs to handle position-dependent reuse, reporting 8x latency reduction on CPU and 60x on GPU. The author compares 2026 vendor strategies: DeepSeek compresses KV cache size by 90% via CSA/HCA architectures with 1M context standard and 10% cache-hit pricing; Anthropic offers explicit cache_control with 4 breakpoints, 1-hour TTL, and 1/10th hit pricing; OpenAI uses automatic caching at 50% hit pricing; Google provides object-oriented CachedContent API management with 2M context. The core finding: no unified standard exists—vendors compete on architecture compression, predictable contracts, ease of use, and enterprise management respectively.

Prompt Cache's technical principle is not complicated. During Transformer autoregressive generation, every generated token requires attention computation over all preceding tokens. Without caching, the K and V tensors of the entire history must be recomputed at each step—O(n²) complexity. With KV Cache enabled, the K/V of preceding tokens are read directly from cache, and a new token only computes its own share, reducing complexity to O(n).

This technique has existed since the GPT-2 era, but between 2024 and 2026 it suddenly became an industry focal point because it upgraded from an *inference optimization* into a commercial bottleneck.

Core insight of Prompt Cache: LLM input prompts contain a large amount of repeated text segments—system messages, prompt templates, context documents, few-shot examples. These fragments are repeatedly fed into the model across many requests. Prompt Cache pre-computes and stores the attention states (KV Cache) of these fragments; when the same fragment appears again, it is reused directly and only the newly added part is computed.

From KV Cache to Prompt Cache

Traditional KV Cache only works within a single request—the K/V of preceding tokens are cached so the next token can use them directly. Prompt Cache extends this to the cross-request, cross-session level: when user A uses a system prompt and user B arrives later, that prompt's attention states are already computed and can be loaded directly.

Yale's paper "Prompt Cache: Modular Attention Reuse for Low-Latency Inference" provided a prototype implementation: latency reduced by 8x on CPU and 60x on GPU, without modifying model parameters.

Key challenge: Transformer positional encoding embeds each token's position into its attention states. If the same text appears at a different position, its attention state changes and cannot be reused directly. The paper's solution is Prompt Markup Language (PML)—schemas define reusable "prompt modules," each assigned a unique position ID that does not depend on global position. Experiments found that LLMs can handle attention states with non-contiguous position IDs: as long as a token's relative positions remain unchanged, output quality is unaffected.

2026 Vendor Prompt Cache Strategies Compared

DeepSeek V4: Compression Is Justice

  • 90% KV cache size reduction: via the CSA (Cache-Strided Attention) / HCA (Hybrid Cache Attention) hybrid attention architecture
  • 1M context as standard across the lineup
  • Cache-hit price at 10%: cached tokens are billed at only 10% of standard input
  • DeepSeek's strategy is architectural compression + pricing leverage. It shrinks KV cache size to 1/10 of the original, turning long context from a "VRAM killer" into an affordable option.

    Anthropic Claude: A Predictable Enterprise Contract

  • cache_control: explicit cache control at the API level; developers can mark which parts are cacheable
  • 4 cache breakpoints: split the prompt into multiple cacheable segments, not just one whole prefix
  • Cross-session global sharing, 1-hour TTL: the same cache can be reused across different sessions
  • Cache-hit price = 1/10 of standard input
  • Anthropic's strategy is predictability. Caching is not a black box but a promise written into the API contract. Boris Cherny, creator of Claude Code, himself admitted: "With the 1M context window, cache misses are very costly. If you step away from your computer for over an hour and return to an old session, you usually get no cache hits at all."

    OpenAI GPT-5: Automatic Caching, Default 50% Discount

  • Automatic caching: hidden behind the API, developers need no extra work
  • Cache-hit price = 50% of standard input
  • No explicit cache-control markers; hit rate is determined automatically by the system
  • OpenAI's strategy is ease of use first. Developers don't need to learn a new API; the system decides which parts can be cached. The cost is lower transparency and control.

    Google Gemini: Object-Oriented Abstraction

  • CachedContent API: an object-oriented abstraction for enterprise-grade long-material reuse
  • 2M context window: currently the largest advertised context
  • Explicit cache management: developers must explicitly create, update, and delete cached content
Google's strategy is enterprise-grade management. CachedContent is a standalone API object with lifecycle management, suited to knowledge bases and document reuse in enterprise scenarios.

Comparison Table

| Vendor | Caching Strategy | Cache Breakpoints | Hit Discount | TTL | Developer Control | |--------|------------------|-------------------|--------------|-----|-------------------| | DeepSeek | Architecture compression + automatic caching | Undisclosed | 10% | Undisclosed | Low | | Anthropic | Explicit cache_control | 4 | 10% | 1 hour | High | | OpenAI | Automatic caching | None | 50% | Undisclosed | Low | | Google | Object-oriented API | Undisclosed | Undisclosed | Developer-controlled | Very high |

---

> Key finding: Four vendors, four strategies, no unified standard. DeepSeek does architectural compression, Anthropic does contract predictability, OpenAI does ease of use, Google does enterprise-grade object management. This is itself an arms race of caching strategies.

Tags

#prompt-cache#kv-cache#llm-inference#deepseek#anthropic#openai#google-gemini#inference-optimization

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177980604