Introduction: LLMs Re-read Everything, Every Time
Imagine a professor who, every time you ask a question about a 100,000-word paper, re-reads the entire document from the first word before answering. This is essentially how most LLMs work today: every time you hit "send," the model reprocesses all prior conversation history, system prompts, tool definitions, and reference material from scratch. This repeated content is known as the prompt prefix—often more than 90% of each request's tokens in long, deep conversations.
Prompt Cache exists to end this absurdity.
What Is Prompt Cache?
When an LLM processes text, it converts input into internal "KV states" (Key-Value states)—an expensive computation involving heavy matrix operations and memory bandwidth. Prompt Cache's core insight is simple: if the prefix is identical, why compute it twice?
On first encounter, the system stores a prefix's KV states in a fast cache (GPU memory or dedicated cache servers). Subsequent requests with identical prefixes reuse those states instead of recomputing. Note: this is a computational memory layer—not the model "remembering" the conversation (that is what the context window does).
A Copy-Shop Analogy
A customer prints the same 100-page document daily, adding one new handwritten note each round. Without caching, you re-copy all 100 pages every time. With caching: "The first 99 pages are archived. You pay an archiving fee the first time, then only pay for the last page each round—at a discount."
The Economics: Anthropic's Numbers
- Cache write: 1.25x base price
- Cache read: 10% of base price (90% off)
- TTL: default 5 minutes (Anthropic), auto-cleared after expiry
- Minimum: 1024 tokens of prefix to qualify
- Byte-level prefix matching: The cache matches character-by-character. A single changed character, misplaced space, or added newline invalidates the cache. KV states are extremely sensitive to input, so the system never gambles on "close enough." Matching starts from the beginning—one mismatch invalidates everything after it.
- TTL and eviction: Anthropic's default is a 5-minute TTL. GPU memory is precious, so unused caches are cleared. The 1024-token minimum exists because short prefixes are cheap to recompute.
- Sub-Agents: Split complex tasks into smaller tasks, each caching only its own context prefix. This decomposition enabled the 92% hit rate.
- Plan Mode: Before execution, the model generates a plan without loading any tool definitions, keeping context short and preserving cache resources. Tools load on demand once the plan is set.
- Lazy Loading: Tools and resources load only when actually needed, maximizing limited GPU cache space.
- Compaction: Long conversations are periodically distilled into summaries that replace raw history, controlling context growth.
- Cache-Safe Forking: New task branches inherit the parent branch's prefix cache, recomputing only branch-specific content.
- Highly stable, version-controlled system prompts
- Modular tool definitions that avoid unnecessary changes
- Layered context: immutable content first, volatile content last
- Sub-Agent granularity balancing task isolation and cache reuse
- easy-learn-ai Prompt Cache teaching module (commit
515b759) - Anthropic: "Prompt Caching in Claude Code" engineering sharing
- Claude Code documentation: https://code.claude.com/docs
Example: a conversation with a 100,000-token context:
| Scenario | First Turn | Each Subsequent Turn | |------|---------|-------------| | No cache | $0.30 | $0.30 | | With cache | $0.375 | $0.03 |
Over 20 turns: no cache costs $6.00 vs. $0.945 with cache—a 6x+ difference. In Claude Code's actual engineering, a 92% cache hit rate delivered an 81% overall cost reduction. This is not a small saving; it cuts costs by roughly an order of magnitude in some scenarios.
How It Works: Strict Matching and Short Shelf Life
Three Cache-Killing Pitfalls
1. Don't touch the prompt. Any edit—a word, a space, a tone particle—breaks the match. Claude Code invests heavily in keeping system prompts and tool definitions perfectly stable. 2. Don't switch models. Different models have completely different KV representations. A cache built for Claude 3.5 Sonnet is useless for Claude 3.5 Haiku. 3. Don't change tool definitions. Tool descriptions participate in prefix computation; adding a tool or changing a parameter description invalidates the cache.
All three boil down to one rule: caches only work for perfectly unchanged content—any change is a death sentence.
Claude Code's Engineering Strategies
Significance and Outlook
Latency, Not Just Cost
Skipping prefix KV computation for 100,000-token contexts can cut response latency from seconds to hundreds of milliseconds—decisive for interactive applications like coding assistants.
From Feature to Infrastructure
Anthropic engineers frame Prompt Cache not as a nice-to-have but as architectural infrastructure. LLM applications should be designed with cache efficiency as a core constraint from day one:
Much like database indexes reshaped application design, Prompt Cache is reshaping LLM application architecture.
Cache-as-a-Service?
The future may bring independent cache services: register a 500,000-word legal corpus once, then reference it by cache ID instead of re-uploading. Anthropic already lets users explicitly declare cacheable prefixes and monitor hit rates; future APIs may include cache inspection, warming, cross-session sharing, and persistent storage beyond the 5-minute TTL.
A Larger Picture
The underlying idea mirrors human cognition: we rarely reason from first principles—we rely on cached memories and intuitions, reserving deep analysis for novel situations. Prompt Cache gives LLMs a similar "reflex": reuse what's already known, compute only what's new.
Conclusion: Repetition Is the Greatest Waste
Heraclitus said no one steps into the same river twice. In the LLM world, the opposite is true—we step into the same river daily yet relearn swimming every time. Prompt Cache is a gentle rebellion against this waste: if we already know the answer, why pretend we're meeting it for the first time?
References: