Prompt Cache: How LLMs Stop Recomputing the Same Prefixes

Forum topic · 小凯 · 2026-05-18

Summary

Large language models reprocess the entire conversation history, system prompts, tool definitions, and reference material on every request, with repeated prompt prefixes often accounting for over 90% of each call. This article from a Chinese tech forum explains prompt caching — a technique that stores the KV states of repeated prefixes so they can be reused instead of recomputed. Using Anthropic's pricing as an example, a 100,000-token conversation costs $0.30 per turn without caching but only $0.03 per turn after the first cache write ($0.375), roughly a 6x saving over 20 turns; Claude Code reportedly achieves a 92% cache hit rate and 81% overall cost reduction. The article details strict byte-level prefix matching, the 5-minute default TTL, the 1,024-token minimum threshold, and three common cache-busting mistakes: editing prompts, switching models, and changing tool definitions. It also covers architectural strategies such as sub-agents, Plan Mode, lazy loading, compaction, and cache-safe forking, arguing that caching should be treated as core infrastructure rather than an afterthought, and may evolve into a standalone "cache as a service" offering.

The Echo of Caching: How Prompt Cache Teaches LLMs to "Remember"

*Translated and adapted from a zhichai.net forum post.*

The Problem: Every Request Starts from Scratch

Imagine a professor re-reading an entire 100,000-word paper from the first character every time you ask a question — even when nothing has changed. That is how most LLMs work today: each time you press "send," the model reprocesses all conversation history, system prompts, tool definitions, and reference material. These repeated inputs are called the prompt prefix, and in long-context conversations they can make up over 90% of every request.

What Is Prompt Cache?

LLMs convert input text into internal KV (Key-Value) states through expensive matrix computations. But if a request's prefix is identical to a previously processed one, its KV states are identical too. Prompt Cache's core insight is simple: if the result is the same, why compute it twice?

On first encounter, a prefix's KV states are stored in high-speed memory (GPU memory or a dedicated cache server). Later requests with an exactly matching prefix reuse the stored states.

The Economics (Anthropic Example)

Cache write: 1.25x base input price
Cache read: 10% of base price (90% off)
Default TTL: 5 minutes
Minimum threshold: 1,024 tokens

For a 100,000-token conversation:

| Scenario | First turn | Each subsequent turn | |----------|-----------|----------------------| | No cache | $0.30 | $0.30 | | With cache | $0.375 | $0.03 |

Over 20 turns: $6.00 without caching vs. $0.945 with caching. In Claude Code's production engineering, a 92% cache hit rate delivered an 81% overall cost reduction.

How Matching Works

Matching is byte-level and exact. A single changed character, extra space, or added newline invalidates the cache from that point onward, because KV states are highly sensitive to input text. The prefix order matters: the system matches from the start, and one mismatch relegates everything after it to recomputation.

Expiration and Thresholds

5-minute default TTL: cached entries are evicted after 5 minutes of inactivity, since GPU memory is scarce.
1,024-token minimum: shorter prefixes aren't worth caching.

Three Cache-Killing Pitfalls

1. Don't edit the prompt — any textual change breaks prefix matching. Stable system prompts and tool definitions are critical. 2. Don't switch models — different models have incompatible KV representations. 3. Don't change tool definitions — tool descriptions are part of the prefix.

All three reduce to the same rule: caching only works for completely unchanged content.

Architectural Patterns from Claude Code

Sub-Agents: split large tasks so each agent caches only its own relevant context, dramatically raising cache utilization (the reported 92% hit rate).
Plan Mode: generate an execution plan *without* loading tool definitions first, keeping planning context short and cache-friendly.
Lazy Loading: only load tools and resources into context when actually needed.
Compaction: periodically distill long conversation histories into summaries to keep the context — and the cache — from growing without bound.
Cache-Safe Forking: new task branches inherit the parent branch's cached prefix, so only branch-specific content is recomputed.

Why It Matters

Latency, Not Just Cost

Skipping prefix KV computation for long contexts (100k-token codebases, legal documents, papers) can cut response times from seconds to hundreds of milliseconds — decisive for interactive applications.

From Feature to Infrastructure

Claude Code's engineers argue prompt caching should be a core architectural constraint from day one: version-control system prompts, modularize tool definitions, place static content before dynamic content, and size sub-agents to balance task isolation with cache reuse — analogous to how database indexes reshaped application design.

Cache as a Service?

Future directions include explicit cache registration (store a 500,000-token legal corpus once, reference it by cache ID), cache monitoring APIs, cache warming, cross-session sharing, and persistence beyond the 5-minute TTL.

Closing Thought

Heraclitus said you cannot step into the same river twice. In the LLM world, we step into the same river daily — and relearn swimming every time. Prompt Cache is a quiet rebellion against that waste: if we already know the answer, why pretend we're meeting it for the first time?

References:

easy-learn-ai Prompt Cache teaching module (commit 515b759)
Anthropic: "Prompt Caching in Claude Code" engineering share
Claude Code documentation: https://code.claude.com/docs