English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Prompt Cache: How LLMs Stop Recomputing the Same Prefixes

Forum topic · 小凯 · 2026-05-18

Summary

Large language models reprocess the entire conversation history, system prompts, tool definitions, and reference material on every request, with repeated prompt prefixes often accounting for over 90% of each call. This article from a Chinese tech forum explains prompt caching — a technique that stores the KV states of repeated prefixes so they can be reused instead of recomputed. Using Anthropic's pricing as an example, a 100,000-token conversation costs $0.30 per turn without caching but only $0.03 per turn after the first cache write ($0.375), roughly a 6x saving over 20 turns; Claude Code reportedly achieves a 92% cache hit rate and 81% overall cost reduction. The article details strict byte-level prefix matching, the 5-minute default TTL, the 1,024-token minimum threshold, and three common cache-busting mistakes: editing prompts, switching models, and changing tool definitions. It also covers architectural strategies such as sub-agents, Plan Mode, lazy loading, compaction, and cache-safe forking, arguing that caching should be treated as core infrastructure rather than an afterthought, and may evolve into a standalone "cache as a service" offering.

The Echo of Caching: How Prompt Cache Teaches LLMs to "Remember"

*Translated and adapted from a zhichai.net forum post.*

The Problem: Every Request Starts from Scratch

Imagine a professor re-reading an entire 100,000-word paper from the first character every time you ask a question — even when nothing has changed. That is how most LLMs work today: each time you press "send," the model reprocesses all conversation history, system prompts, tool definitions, and reference material. These repeated inputs are called the prompt prefix, and in long-context conversations they can make up over 90% of every request.

What Is Prompt Cache?

LLMs convert input text into internal KV (Key-Value) states through expensive matrix computations. But if a request's prefix is identical to a previously processed one, its KV states are identical too. Prompt Cache's core insight is simple: if the result is the same, why compute it twice?

On first encounter, a prefix's KV states are stored in high-speed memory (GPU memory or a dedicated cache server). Later requests with an exactly matching prefix reuse the stored states.

The Economics (Anthropic Example)

  • Cache write: 1.25x base input price
  • Cache read: 10% of base price (90% off)
  • Default TTL: 5 minutes
  • Minimum threshold: 1,024 tokens
  • For a 100,000-token conversation:

    | Scenario | First turn | Each subsequent turn | |----------|-----------|----------------------| | No cache | $0.30 | $0.30 | | With cache | $0.375 | $0.03 |

    Over 20 turns: $6.00 without caching vs. $0.945 with caching. In Claude Code's production engineering, a 92% cache hit rate delivered an 81% overall cost reduction.

    How Matching Works

    Matching is byte-level and exact. A single changed character, extra space, or added newline invalidates the cache from that point onward, because KV states are highly sensitive to input text. The prefix order matters: the system matches from the start, and one mismatch relegates everything after it to recomputation.

    Expiration and Thresholds

  • 5-minute default TTL: cached entries are evicted after 5 minutes of inactivity, since GPU memory is scarce.
  • 1,024-token minimum: shorter prefixes aren't worth caching.
  • Three Cache-Killing Pitfalls

    1. Don't edit the prompt — any textual change breaks prefix matching. Stable system prompts and tool definitions are critical. 2. Don't switch models — different models have incompatible KV representations. 3. Don't change tool definitions — tool descriptions are part of the prefix.

    All three reduce to the same rule: caching only works for completely unchanged content.

    Architectural Patterns from Claude Code

  • Sub-Agents: split large tasks so each agent caches only its own relevant context, dramatically raising cache utilization (the reported 92% hit rate).
  • Plan Mode: generate an execution plan *without* loading tool definitions first, keeping planning context short and cache-friendly.
  • Lazy Loading: only load tools and resources into context when actually needed.
  • Compaction: periodically distill long conversation histories into summaries to keep the context — and the cache — from growing without bound.
  • Cache-Safe Forking: new task branches inherit the parent branch's cached prefix, so only branch-specific content is recomputed.
  • Why It Matters

    Latency, Not Just Cost

    Skipping prefix KV computation for long contexts (100k-token codebases, legal documents, papers) can cut response times from seconds to hundreds of milliseconds — decisive for interactive applications.

    From Feature to Infrastructure

    Claude Code's engineers argue prompt caching should be a core architectural constraint from day one: version-control system prompts, modularize tool definitions, place static content before dynamic content, and size sub-agents to balance task isolation with cache reuse — analogous to how database indexes reshaped application design.

    Cache as a Service?

    Future directions include explicit cache registration (store a 500,000-token legal corpus once, reference it by cache ID), cache monitoring APIs, cache warming, cross-session sharing, and persistence beyond the 5-minute TTL.

    Closing Thought

    Heraclitus said you cannot step into the same river twice. In the LLM world, we step into the same river daily — and relearn swimming every time. Prompt Cache is a quiet rebellion against that waste: if we already know the answer, why pretend we're meeting it for the first time?

    References:

  • easy-learn-ai Prompt Cache teaching module (commit 515b759)
  • Anthropic: "Prompt Caching in Claude Code" engineering share
  • Claude Code documentation: https://code.claude.com/docs

Tags

#prompt-caching#llm#anthropic#claude-code#inference-optimization#kv-cache#ai-architecture#cost-optimization

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620316