The Echo of Cache: How Prompt Cache Teaches LLMs to "Learn" Continuity
The Problem: LLMs Re-read Everything from Scratch
Imagine a professor who, every time you ask a question about a 100,000-word paper, re-reads the entire document from the first word before answering. This is how most large language models actually work: on every request, they reprocess the full conversation history, system prompts, tool definitions, and reference material from the beginning.
This repeatedly processed content is called the prompt prefix. In deep conversations with ~100,000 tokens of context, the prefix can make up more than 90% of each request—yet the model treats it as brand-new input every time.
Prompt Caching exists to end this absurdity.
What Is Prompt Cache?
LLMs convert input text into internal representations called KV states (Key-Value states). This conversion is expensive, requiring heavy matrix computation and memory bandwidth. The core insight of Prompt Cache is simple: if a prefix is identical to a previous request, its KV states must be identical—so why compute it twice?
The system stores a prefix's KV states in a fast cache (GPU memory or dedicated cache servers) on first encounter. Subsequent requests with an exactly matching opening can reuse those states directly.
A useful analogy: a print shop customer who prints the same 100-page document daily, adding a handwritten note on the last page. Without caching, all 100 pages are reprinted each time. With caching, the unchanged 99 pages are archived once, and only the new page is printed.
The Economics (Anthropic Example)
- Cache write: 1.25x base token price (archiving overhead)
- Cache read: 10% of base price (90% off)
- TTL: default 5 minutes, auto-expiry
- Minimum: prefixes of at least 1,024 tokens qualify
- Without cache: 20 × $0.30 = $6.00
- With cache: $0.375 + 19 × $0.03 = $0.945
- Sub-Agent architecture: Split complex tasks among sub-agents, each caching only its own relevant context, rather than stuffing everything into one giant conversation. This enables up to a 92% cache hit rate.
- Plan Mode: Before execution, the model generates a plan without loading any tool definitions—keeping the planning context short and avoiding wasted cache on premature tool definitions.
- Lazy Loading: Tools and resources are loaded only when actually needed, maximizing limited cache space.
- Compaction: Long conversations are periodically summarized; the summary replaces the raw history, preventing unbounded context growth.
- Cache-Safe Forking: New task branches inherit the parent branch's prefix cache (system prompt, project context), recomputing only branch-specific content.
- Keep system prompts extremely stable, with version control
- Modularize tool definitions to avoid unnecessary changes
- Layer context: static content first, dynamic content last
- Balance Sub-Agent granularity between task isolation and cache reuse
- easy-learn-ai Prompt Cache teaching module (commit
515b759) - Anthropic: "Prompt Caching in Claude Code" — engineering practice sharing
- Claude Code documentation: https://code.claude.com/docs
A concrete example: a 100,000-token conversation
| Scenario | First-turn cost | Each subsequent turn | |---|---|---| | No cache | $0.30 | $0.30 | | With cache | $0.375 | $0.03 |
Over 20 turns:
That is a greater than 6x saving. In Claude Code's real-world engineering, a 92% cache hit rate yielded an overall 81% cost reduction—cutting costs by nearly an order of magnitude in some scenarios.
How the Cache Works
Strict prefix matching
Matching is byte-level and character-by-character. A single changed character, misplaced space, or extra newline invalidates the cache. KV states are extremely sensitive to input, so the system never risks a near-match—any mismatch means recomputation from scratch. Since matching starts from the beginning, prefix ordering is critical.
TTL and eviction
Cache entries expire after 5 minutes of inactivity (Anthropic's default), since GPU memory is precious. The 1,024-token minimum exists because caching short prefixes isn't worth the overhead.
Three cache-invalidation pitfalls
1. Don't touch the prompt. Any edit—even one character—breaks prefix matching. Teams must keep system prompts and tool definitions absolutely stable. 2. Don't switch models. Different models have entirely different KV representations; cached states are not portable across models. 3. Don't modify tool definitions. Tool descriptions participate in the prefix; adding a tool or changing a parameter description invalidates the cache.
All three reflect the same rule: the cache only works for content that is completely unchanged.
Claude Code Engineering Practices
Beyond Cost: Latency and a New Paradigm
Prompt caching doesn't just save money—it reduces latency. Skipping prefix KV computation for 100,000-token contexts can drop response times from seconds to hundreds of milliseconds, which is decisive for interactive tools like coding assistants.
Anthropic's engineers frame caching not as a nice-to-have feature but as infrastructure-level architecture. Design implications:
Looking forward, caching may evolve into a standalone cache-as-a-service model: register a large shared prefix once (e.g., a 500,000-token legal corpus), then reference it by cache ID. Finer-grained APIs—cache inspection, warming, cross-session sharing, and persistence beyond the 5-minute TTL—are plausible next steps.
Closing Thought
Heraclitus said no one steps into the same river twice. In the LLM world, the opposite happens daily: we step into the same river over and over, re-learning to swim each time. Prompt Cache is a gentle rebellion against that waste—if we already know the answer, why pretend it's the first encounter?
---
References: