English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

The Echo of Cache: How Prompt Cache Teaches LLMs to Build on What Came Before

Forum topic · 小凯 · 2026-05-18

Summary

This article from zhichai.net explains Prompt Caching, a technique that eliminates redundant computation in large language model (LLM) inference. Every time a user sends a message, LLMs normally reprocess the entire conversation history, system prompts, and tool definitions—the prompt prefix, which can account for over 90% of each request in long-context conversations. Prompt Cache stores the Key-Value (KV) states of repeated prefixes and reuses them for subsequent requests. Using Anthropic as the main example, the article details the economics: cache writes cost 1.25x the base price, reads cost only 10%, with a 5-minute TTL and a 1,024-token minimum. For a 100,000-token conversation over 20 turns, costs drop from $6.00 to $0.945—a 6x saving. The article covers strict byte-level prefix matching, three major cache-invalidation pitfalls (changing prompts, switching models, altering tool definitions), and Claude Code engineering practices: Sub-Agent architectures achieving 92% cache hit rates and 81% cost reduction, Plan Mode, lazy loading of tool definitions, context compaction, and cache-safe forking. It concludes that prompt caching is shifting from a cost optimization into an architectural paradigm for LLM applications.

The Echo of Cache: How Prompt Cache Teaches LLMs to "Learn" Continuity

The Problem: LLMs Re-read Everything from Scratch

Imagine a professor who, every time you ask a question about a 100,000-word paper, re-reads the entire document from the first word before answering. This is how most large language models actually work: on every request, they reprocess the full conversation history, system prompts, tool definitions, and reference material from the beginning.

This repeatedly processed content is called the prompt prefix. In deep conversations with ~100,000 tokens of context, the prefix can make up more than 90% of each request—yet the model treats it as brand-new input every time.

Prompt Caching exists to end this absurdity.

What Is Prompt Cache?

LLMs convert input text into internal representations called KV states (Key-Value states). This conversion is expensive, requiring heavy matrix computation and memory bandwidth. The core insight of Prompt Cache is simple: if a prefix is identical to a previous request, its KV states must be identical—so why compute it twice?

The system stores a prefix's KV states in a fast cache (GPU memory or dedicated cache servers) on first encounter. Subsequent requests with an exactly matching opening can reuse those states directly.

A useful analogy: a print shop customer who prints the same 100-page document daily, adding a handwritten note on the last page. Without caching, all 100 pages are reprinted each time. With caching, the unchanged 99 pages are archived once, and only the new page is printed.

The Economics (Anthropic Example)

  • Cache write: 1.25x base token price (archiving overhead)
  • Cache read: 10% of base price (90% off)
  • TTL: default 5 minutes, auto-expiry
  • Minimum: prefixes of at least 1,024 tokens qualify
  • A concrete example: a 100,000-token conversation

    | Scenario | First-turn cost | Each subsequent turn | |---|---|---| | No cache | $0.30 | $0.30 | | With cache | $0.375 | $0.03 |

    Over 20 turns:

  • Without cache: 20 × $0.30 = $6.00
  • With cache: $0.375 + 19 × $0.03 = $0.945
  • That is a greater than 6x saving. In Claude Code's real-world engineering, a 92% cache hit rate yielded an overall 81% cost reduction—cutting costs by nearly an order of magnitude in some scenarios.

    How the Cache Works

    Strict prefix matching

    Matching is byte-level and character-by-character. A single changed character, misplaced space, or extra newline invalidates the cache. KV states are extremely sensitive to input, so the system never risks a near-match—any mismatch means recomputation from scratch. Since matching starts from the beginning, prefix ordering is critical.

    TTL and eviction

    Cache entries expire after 5 minutes of inactivity (Anthropic's default), since GPU memory is precious. The 1,024-token minimum exists because caching short prefixes isn't worth the overhead.

    Three cache-invalidation pitfalls

    1. Don't touch the prompt. Any edit—even one character—breaks prefix matching. Teams must keep system prompts and tool definitions absolutely stable. 2. Don't switch models. Different models have entirely different KV representations; cached states are not portable across models. 3. Don't modify tool definitions. Tool descriptions participate in the prefix; adding a tool or changing a parameter description invalidates the cache.

    All three reflect the same rule: the cache only works for content that is completely unchanged.

    Claude Code Engineering Practices

  • Sub-Agent architecture: Split complex tasks among sub-agents, each caching only its own relevant context, rather than stuffing everything into one giant conversation. This enables up to a 92% cache hit rate.
  • Plan Mode: Before execution, the model generates a plan without loading any tool definitions—keeping the planning context short and avoiding wasted cache on premature tool definitions.
  • Lazy Loading: Tools and resources are loaded only when actually needed, maximizing limited cache space.
  • Compaction: Long conversations are periodically summarized; the summary replaces the raw history, preventing unbounded context growth.
  • Cache-Safe Forking: New task branches inherit the parent branch's prefix cache (system prompt, project context), recomputing only branch-specific content.
  • Beyond Cost: Latency and a New Paradigm

    Prompt caching doesn't just save money—it reduces latency. Skipping prefix KV computation for 100,000-token contexts can drop response times from seconds to hundreds of milliseconds, which is decisive for interactive tools like coding assistants.

    Anthropic's engineers frame caching not as a nice-to-have feature but as infrastructure-level architecture. Design implications:

  • Keep system prompts extremely stable, with version control
  • Modularize tool definitions to avoid unnecessary changes
  • Layer context: static content first, dynamic content last
  • Balance Sub-Agent granularity between task isolation and cache reuse
  • Looking forward, caching may evolve into a standalone cache-as-a-service model: register a large shared prefix once (e.g., a 500,000-token legal corpus), then reference it by cache ID. Finer-grained APIs—cache inspection, warming, cross-session sharing, and persistence beyond the 5-minute TTL—are plausible next steps.

    Closing Thought

    Heraclitus said no one steps into the same river twice. In the LLM world, the opposite happens daily: we step into the same river over and over, re-learning to swim each time. Prompt Cache is a gentle rebellion against that waste—if we already know the answer, why pretend it's the first encounter?

    ---

    References:

  • easy-learn-ai Prompt Cache teaching module (commit 515b759)
  • Anthropic: "Prompt Caching in Claude Code" — engineering practice sharing
  • Claude Code documentation: https://code.claude.com/docs

Tags

#prompt-caching#llm#anthropic#claude-code#kv-cache#inference-optimization#ai-engineering#cost-optimization

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620316