Cache Rules Everything: When AI Starts "Remembering" Every Word You Say
> Source: easy-learn-ai commit 515b759
> Inspiration: Anthropic engineers' sharing of best practices on Claude Code prompt caching design
1. A Question: Why Does AI Get "Slower" the Longer You Chat?
Imagine this scenario. You chat all afternoon with a brilliant friend across twenty-plus turns. Every time you finish speaking, he falls into deep thought—and then you realize he isn't thinking about your latest sentence. He is re-reading your entire afternoon conversation, from the very first line, word for word, in his head.
You'd think he'd lost his mind.
But that's how almost all large language models work today.
Every time you send a new message, the backend must re-encode all the text you sent—system instructions, conversation history, tool definitions, the new message—from start to finish. This process is called Prefill, and it's the biggest driver of both latency and cost.
After 20 turns, the first 19 turns in the 20th request are identical to last time. Yet the model still recomputes from the first character—19 turns of wasted work.
This isn't an optimization problem. It's structural waste.
---
2. Prompt Caching: A "Notepad" for AI
Prompt caching solves this.
The principle in one sentence: you mark a breakpoint in the request. The backend stores the encoding of everything up to that point. Next time, if the prefix is exactly the same, it reuses the stored result and skips the recomputation.
Analogy: rewriting the table of contents and earlier chapters of your paper every time before adding new content. With caching, the already-copied parts fast-forward, and you only write the new section.
But this isn't a minor time-saver. Look at the real numbers.
---
3. The Numbers Don't Lie: 90% of Costs Can Be Saved
Anthropic has published the key ratios:
- Cache hits cost one-tenth the price (cache read = 10% of base price)
- First write costs 25% extra (cache write = 125% of base price)
- Every subsequent use saves 90%
There is a threshold—content that's too short can't be cached. Generally at least 1024 tokens, 4096 for newer models. Short prompts get no caching.
A concrete example: a 100,000-character long conversation.
Without caching, each turn on Claude Sonnet costs $0.30. With caching, the first turn is $0.375 (25% extra), then each turn after is only $0.03. Over 10 turns, that's roughly 90% saved on input costs.
It's not just money. Latency drops too—the more that's skipped, the faster the first token arrives (TTFT, Time To First Token).
Anthropic internally treats cache hit rate as an infrastructure-grade metric, on par with server uptime. When hit rate drops, on-call alerts fire and engineers handle it like a production incident—the original text uses the phrase "declare SEVs."
More importantly, high hit rates directly affect user experience—they let Anthropic offer paying users more generous usage limits. The higher the cache hit rate, the more you can use at the same price.
So caching isn't a nice-to-have for Claude Code. It's the precondition for the entire system to function. No cache, no Claude Code.
---
4. Core Principle: Prefix Matching
Understanding prompt caching takes four words: prefix matching.
If the next request's prefix matches the last one, previous computation is reused. Which means—any change at any position in the prefix invalidates the cache for everything after it.
Like a chain: change any one link, and every link after it must be redone.
Since caching depends on prefix matching, the ordering of prompt components is critical. Anthropic's best practice:
| Layer | Content | Sharing scope | |-------|---------|---------------| | 1 | System instructions + tool definitions | Shared across all sessions | | 2 | Project docs (CLAUDE.md) | Shared within a project | | 3 | Current session context | Valid only for this conversation | | 4 | Chat messages | Grows each turn; only the last message is new |
In short: the less likely something is to change, the earlier it goes.
It's like organizing a desk: reference books you never move go on the bottom shelf; this week's materials in the middle; today's draft on top. That way you don't re-sort the whole desk every time you sit down.
---
5. Seven Pitfalls: Anthropic's Landmines, So You Don't Step on Them
Pitfall 1: Embedding the current time in fixed instructions
It changes every second, instantly killing the cache.
Fix: don't modify system instructions; push updates into the next message. Attach a "system reminder" inside the next user message carrying the updated info. System instructions are the foundation—nailed down and immovable; messages are flowing water—change them however you like.
Pitfall 2: Tool definitions in unordered containers
Request order changes each time, and prefixes no longer match.
Fix: tool definitions must have a fixed, deterministic order.
Pitfall 3: Modifying tool parameters
Change even one field, and the entire prefix's cache is invalidated.
Counterintuitive trap: the current task only needs 3 tools—wouldn't removing the other 30 be cleaner?
Truth: tool definitions are part of the cached prefix. Adding or removing one breaks the cache. Once broken, the whole conversation's cache gets rebuilt—at a cost far exceeding the tiny space a few extra tool definitions occupy. It looks like an optimization but ends up making things worse.
Pitfall 4: Switching models
You might think: use a small model for easy questions to save money, switch back to the big one for hard ones. Reasonable, right?
Reality: caches are model-bound. Switching models invalidates all accumulated cache, forcing a rebuild from scratch. The rebuild cost often exceeds just letting the big model answer the easy question directly.
So Claude Code's strategy is—the main conversation uses one model from start to finish. Need a small model for something? Spawn a subtask. Subtasks have their own independent context and cache, so they don't pollute the main conversation's cache chain.
The concrete approach: have the main model write a task handoff note distilling the context. Pass it to the subtask for execution; only the result comes back to the main conversation. You don't let an intern sit at your desk using your computer—you assign them a separate machine, send clear instructions, and get the result back.
Account warning: caches are isolated per account. Some people try account pools for relaying. Mix the pool, hit rates plummet, and you lose money and accounts.
Pitfall 5: Tool switching in plan mode
Claude Code has a "plan mode": the model only thinks and plans, no execution.
The intuitive approach: remove execution tools on entering plan mode, add them back on exit.
But Anthropic doesn't do that. They keep all tools in place and instead add two special tools—"enter planning" and "exit planning." Calling "enter planning" switches the model to thinking mode; "exit planning" returns it.
How is the "no execution in plan mode" constraint communicated? By inserting a system message into the conversation telling the model it's now planning.
Note—inserted as a message in the conversation flow, not by changing system instructions. Keep these distinct: system instructions are fixed and live in the cached prefix; conversation messages are fluid and don't affect the prefix. The tool set never changes, so the cache stays valid.
A bonus: the model can decide for itself when to enter plan mode—on complex tasks it thinks first and then acts, no manual switching required.
Pitfall 6: Dozens of tools all crammed in
Claude Code may connect to dozens of external tools. Fully define all of them? Too much space. Add/remove on demand? Breaks the cache.
Anthropic's compromise: Lazy Loading.
Start with lightweight placeholders only. The model sees tool names without full parameter definitions. When it actually needs a tool, it pulls the full definition via a "tool search" feature.
Benefit: the prefix always contains just the lightweight placeholders, never changing because some tool was loaded. Cache stays rock-solid.
It's like a library's catalog index—you look up the book in the catalog, then fetch it from the shelf, instead of hauling every book onto your desk.
Notably, this tool search capability is now available via API, so developers can simplify their own tool management.
Pitfall 7: Context compaction starting from scratch
Long conversations eventually fill the context window. You compress earlier dialogue into a summary to free up space.
The problem: if the compaction request uses different system instructions and omits tool definitions, nothing matches the main conversation's cache from the first character. Two cache chains, zero reuse.
Worse, you're sending the whole conversation for summarization—at full, non-discounted cost. The longer the conversation, the more expensive.
Anthropic's solution: "Cache-Safe Forking."
The compaction request must use exactly the same system instructions, user context, and tool definitions as the main conversation, carrying the main conversation's messages as history. Then append one compaction instruction as a new user message at the end.
From the backend's perspective, this request is nearly identical to the previous one—same prefix, same tools, same history—so the prefix cache is directly reused. The new cost is only the compaction instruction itself.
Also reserve a compaction buffer so summary output has room—don't wait until the window is completely full; leave margin early.
One compaction operation reuses the entire cache the main conversation accumulated. Almost no extra cost.
---
6. Looking Back: Every Lesson Says the Same Thing
All seven lessons boil down to one sentence—the cache is prefix matching.
1. Prefix matching determines everything. Any change at any point in the prefix invalidates everything after it. 2. Use messages instead of instruction edits. For mode switches, time updates—put them in conversation messages; never touch system instructions. 3. Don't swap tools or models mid-conversation. Express state transitions with tools; replace tool add/remove with lazy loading.
4. Monitor cache hit rate like you monitor uptime. Anthropic pages engineers on cache disruptions and treats them as production incidents. 5. Forked operations must share the main conversation's prefix. Compaction, summaries, subtasks—all use identical parameters.
Every design decision flows from this one constraint. Don't edit instructions, don't touch tools—one touch, and the whole cache chain breaks. Switching models, switching accounts, starting from scratch—same principle.
This looks like cache optimization, but it's also a way of thinking—first accept the unavoidable constraint, then build the entire system around it.
---
7. Why This Matters
Prompt caching isn't a niche optimization. It's infrastructure for the AI Agent era.
Without it, every AI coding assistant, every multi-turn dialogue system, every tool-call-heavy agent would suffocate under the combined pressure of cost and latency. A 100,000-character conversation at $0.30 per turn means $30 over 100 turns—and that's excluding output tokens.
With it, the same conversation drops to one-tenth the cost, with response speeds users can't perceive waiting on.
Engineers have an old saying: "Cache rules everything around you." In the AI Agent era it still holds—except this time the cache doesn't store web pages or images. It stores the half-finished state of AI's thinking.
---
> 📌 This article is based on the easy-learn-ai project's "Prompt Cache, Made Easy" interactive tutorial—an immersive learning experience with 15 chapters, audio narration, and stage animations. If you want the technical details, go experience it yourself.