Source commit: 515b759 -- feat(prompt-cache): add prompt cache module
When AI Learns Perfect Memory: How Prompt Cache Cuts Costs by 90%
Have you ever wondered why the longer you chat with Claude, the slower it responds and the bigger your bill grows?
The answer lies in an old engineering saying: "caching rules everything." In the AI Agent era, this saying hasn't aged—it has become the line between a system that runs and one that doesn't.
Prompt Cache is not the kind of cache you clear in browser settings. It's a precision mechanism inside LLM inference systems, built to tackle the most expensive, most time-wasting step in human-AI conversation.
---
1. Repeated Labor: An AI That Starts Over Every Turn
Imagine you've chatted with a coding assistant for 20 turns—fixing bugs, refactoring code, writing tests. On turn 20, you send a new request.
What happens in the background? The model re-encodes all 19 previous turns—system instructions, tool definitions, all history messages—from scratch. It computes attention weights and KV values token by token before it can even start answering.
Those 19 turns are identical to the last round, yet the model re-inventories the whole shelf like an obsessive librarian. This is the most expensive, most time-consuming part of LLM inference: the Prefill stage—converting input text into internal representations.
20 turns of chat, 19 rounds of redundant computation. Wasteful, right?
---
2. How Prompt Cache Works: Fast-Forward What's Already Computed
The solution in one sentence: mark a breakpoint in your request; the backend stores the encoding results from the start up to that breakpoint. Next time the prefix matches exactly, reuse it and skip the recomputation.
Think of writing a paper: with a cache, the already-written table of contents and earlier chapters fast-forward, and you only write the new section.
The keyword is prefix matching. Matching is not semantic similarity—it's byte-level prefix identity. If the beginning of your new request is identical to the last one, those computation results are reused directly.
---
3. The Numbers: Absurdly Good Economics
A few hard numbers explain why Anthropic treats cache hit rate as an infrastructure-level metric—on par with server uptime. When the hit rate drops, on-call alerts fire and a full incident response kicks in.
- Cached tokens cost one-tenth the price—a 90% saving
- First write costs 25% extra—1.25x to store the result
- Default TTL is 5 minutes, auto-renewed on each request at no extra charge; a paid 1-hour tier is also available
- Minimum cacheable length: generally 1024 tokens; 4096 for newer models
- Without cache: $0.30 per turn
- With cache: $0.375 first, then only $0.03 per turn
A concrete example: a long conversation of 100,000 characters on Claude Sonnet:
---
4. Prompt Ordering: The More Stable, the Earlier
Since cache relies on prefix matching, the ordering of prompt content is critical. Anthropic's best practice, top to bottom:
1. System instructions + tool definitions — fixed, shared across all sessions 2. Project documentation (e.g., CLAUDE.md) — shared within a project 3. Current session context — valid only within this conversation 4. Chat messages — grows turn by turn; only the last item is new each round
One rule: the less likely something is to change, the earlier it goes.
It's like organizing a desk: rarely-touched reference books on the bottom, this week's materials in the middle, today's draft on top—so you never have to re-sort the whole desk each day.
---
5. Three Pitfalls: One Small Detail Breaks the Whole Chain
Pitfall 1: Embedding the current time in fixed instructions. It changes every second, the prefix never matches, and the cache is dead. Correct approach: put updated time into conversation messages, never into system instructions.
Pitfall 2: Storing tool definitions in unordered containers. Python dicts or sets serialize in different orders each request, breaking the prefix. Fix: use ordered containers (list), or sort explicitly before sending.
Pitfall 3: Changing even one field of a tool parameter. A single field change invalidates the entire prefix chain. That's why in Claude Code's design, the tool set is essentially never modified mid-session.
---
6. Advanced Techniques: Plan Mode, Lazy Loading, Subtasks
Plan Mode: Switch State via Messages, Not Instructions
Claude Code has a "Plan Mode" where the model only thinks, doesn't execute. The intuitive approach—removing execution tools on entry and re-adding them on exit—would break the cache.
Instead, Anthropic keeps all tools in place and adds two special tools—"enter planning" and "exit planning." The constraint that planning mode forbids execution is conveyed by inserting a system message into the conversation, telling the model it is now planning.
The key distinction: system instructions are fixed and live inside the cached prefix; conversation messages are fluid and don't affect the prefix. The tool set never changes, so the cache stays valid.
Lazy Loading: Placeholders First, Load on Use
Claude Code may integrate dozens of external tools. Fully defining all of them wastes space; adding/removing on demand breaks the cache.
The compromise is lazy loading: start with lightweight placeholders—the model sees only tool names, not full parameter schemas. When the model actually needs a tool, it fetches the full definition via "tool search."
The prefix contains only the lightweight placeholders, so it never changes when tools load. Like a library catalog: browse the index first, then fetch the book—no need to pile every book on your desk.
Subtasks: Send the Intern to a Separate Desk
When a smaller model should do the work? Delegate a subtask. Subtasks have their own independent context and cache, so they don't pollute the main conversation's cache chain.
Concretely: the main model writes a task handoff, condensing the context and passing it to the subtask. When done, only the result returns to the main conversation. You don't let an intern sit at your desk using your computer—you assign them a separate machine, a clear task brief, and collect the results.
---
7. Cache as Infrastructure: No Cache, No Claude Code
Inside Anthropic there's a claim: without caching, there would be no Claude Code.
Why? AI coding assistants have long conversations—dozens of turns per session. Each turn re-sends the full context; recomputing from scratch every time would explode latency and cost.
The higher the cache hit rate, the more you can use within the same price. So Anthropic treats it not as a performance optimization but as a cornerstone of user experience. High hit rates don't just save money—they directly determine how generous usage quotas can be for paying users.
---
8. Looking Back: Every Lesson Says the Same Thing
All seven lessons boil down to one idea—the cache is prefix matching.
Any change at any point in the prefix invalidates everything after it. So: don't touch instructions, don't touch tools—one change, and the whole chain breaks. Switching models, switching accounts, starting fresh—the same logic applies.
This looks like cache optimization, but it's really a design philosophy: first accept the constraint you can't avoid, then build the entire system around it.
---
> This project comes from easy-learn-ai, which demonstrates the complete Prompt Cache knowledge system through 15 chapters of interactive animations, each with narration and visuals, playable directly in the browser. Reference: Anthropic engineers' sharing on prompt caching best practices in Claude Code.