Source commit: 515b759
Have you ever wondered what actually happens on the server every time you send a message to ChatGPT or Claude?
It's not simply "thinking." In reality, every new message you send forces the server to re-encode everything you've said before—system instructions, tool definitions, conversation history, plus your new message—from the very beginning. This process is called prefill, and it's the main driver of both latency and cost.
What's worse: after 20 rounds of conversation, the first 19 rounds in request #20 are byte-for-byte identical to last time. Yet the model still recomputes from the first token. It's like copying the table of contents and the first chapters of a paper every time you write a new paragraph—19 rounds of wasted, repeated work.
Prompt caching exists to solve exactly this.
1. How It Works: Prefix Matching, Like Fast-Forwarding a Copy Job
The mechanism in one sentence: you mark a breakpoint in your request, and the backend stores the encoding results from the start up to that breakpoint. Next time the prefix is exactly the same, it's reused directly, skipping redundant computation.
Analogy: normally you'd re-copy the table of contents and earlier chapters before adding new content. With caching, the copied parts fast-forward, and you only write the new section.
This is the technology Anthropic uses extensively in Claude Code, its AI coding assistant where a single session can span dozens of rounds. Every round re-sends the full context. Without caching, latency and cost would explode.
Internally, Anthropic treats cache hit rate as an infrastructure-grade metric—on par with server uptime. If hit rate drops, an on-call alert fires and engineers handle it as a production incident. The original phrasing was "declaring a severity incident," which is far more serious than a routine alert.
More importantly: a high hit rate doesn't just save money—it directly improves user experience, letting Anthropic offer paid users more generous usage limits. The higher the hit rate, the more you can use at the same price.
So for Claude Code, caching isn't a nice-to-have optimization. It's the precondition for the entire system to function. No cache, no Claude Code.
2. The Numbers: A 90% Discount
Key figures:
- Cache hits cost 10% of the base input price
- First write costs 1.25x—a 25% premium
- Every subsequent read saves 90%
- Without caching: $0.30 per round
- With caching: $0.375 the first time, then only $0.03 per round
- Pitfall 1: Embedding the current time in fixed instructions. It changes every second, killing the cache instantly.
- Pitfall 2: Storing tool definitions in unordered containers. Python
dictorsetproduces a different order per request, so prefixes never match. - Pitfall 3: Changing a tool parameter. Even one modified field invalidates the cache for the entire prefix.
- Anthropic engineering blog: "Lessons from building Claude Code: Prompt caching is everything"
- easy-learn-ai project: interactive prompt-cache presentation site
The default cache TTL is 5 minutes, auto-renewed on each request within the window at no extra charge. A paid 1-hour option is also available.
There's a threshold: content that's too short can't be cached. Generally you need at least 1024 tokens (4096 for newer models). Short prompts get no cache.
A concrete example: a 100k-character long conversation on Claude Sonnet:
Over 10 rounds, that's roughly 90% saved on input cost. And it's not just money—latency drops too. The more that doesn't need recomputation, the faster the first token appears.
3. Order Matters: The Most Stable Content Goes First
Since caching relies on prefix matching, the arrangement of prompt content is critical.
Anthropic's recommended ordering: 1. Front: system instructions and tool definitions—fixed, shared across all sessions. 2. Second layer: project docs (CLAUDE.md)—shared within a project. 3. Third layer: current session context—valid only for this conversation. 4. Last: chat messages—growing each round, with only the newest message added.
One-line summary: the less something changes, the earlier it should appear.
It's like organizing a desk: rarely-touched reference books on the bottom shelf, this week's reading in the middle, today's draft on top. That way you don't have to翻 through the whole desk each time you sit down.
4. Three Pitfalls: One Small Detail Breaks the Whole Cache Chain
One overlooked detail, and the whole cache chain snaps.
5. Updating Information? Send a Message, Don't Touch the Prompt
During task execution, information easily goes stale—built-in timestamps become wrong, or the user edits a file. The instinctive fix is to update the system prompt, but that instantly invalidates the cache, which is expensive for users.
Claude Code's approach is to deliver updates as messages in the next turn instead of modifying the original prompt prefix. They quietly insert a <system-reminder> tag into the next user message or tool result to convey the latest information to the model.
This small trick perfectly preserves the precious cache.
6. Don't Switch Models, Don't Touch Tools
You might think: switch to Haiku for easy questions to save money, then back to Opus for hard ones—how sensible!
In reality, the cache is bound to the model. Switching models voids all accumulated cache and forces a rebuild from scratch. The rebuild cost often exceeds what it would cost to just let Opus answer the easy question.
So Claude Code's policy: the main conversation uses the same model from start to finish.
What about using a smaller model for work? Delegate to sub-tasks. Sub-tasks have their own independent context and cache, so they don't pollute the main conversation's cache chain. Concretely: the main model writes a handoff note condensing the context, passes it to the sub-task for execution, and only the result comes back to the main conversation.
Analogy: you wouldn't let an intern sit at your desk using your computer. You'd assign them a separate machine, send clear instructions, and have them return the results.
A warning for people running proxy/relay services: the cache is isolated per account. Mixing an account pool tanks the hit rate—you lose money and the accounts. Same for tools that constantly rotate accounts: don't switch back and forth every couple of messages.
7. Compressing Context, the Cache-Safe Way
Long conversations eventually fill the context window, requiring compression of prior dialogue into a summary to free space.
But here's the problem: if you spin up a separate API call for compression—different system prompt, no tool definitions—it mismatches the main conversation's cache from the very first token. Two cache chains, zero reuse, double the cost.
Anthropic's solution is Cache-Safe Forking:
The compression request must use exactly the same system instructions, user context, and tool definitions as the main conversation, with the main conversation's messages included as history. Then a single compression instruction is appended as a new user message at the end.
From the backend's perspective, this request looks nearly identical to the previous one: same prefix, same tools, same history—so prefix cache is directly reused. The only new cost is the compression instruction itself.
You should also reserve a compression buffer with enough room for the summary output. Don't wait until the window is completely full—leave headroom in advance.
One compression operation, reusing the entire accumulated cache of the main conversation. Almost no extra cost.
8. Plan Mode and Lazy Loading
Claude Code has another clever design called Plan mode.
When the user requests something complex, the model doesn't start working immediately. Instead it enters a "planning state"—analyzing the task, breaking it into steps, listing tools to call. Crucially, this state change isn't implemented by modifying the system prompt (which would break the cache), but simulated via a tool call.
The model invokes a special plan tool signaling "entering planning mode." The system recognizes the call and displays the planning UI in its reply. Throughout, the system prompt doesn't change by a single character—the cache stays intact.
Similarly, Lazy Loading is cache-friendly: rather than loading all tool definitions upfront, tools are loaded progressively as needed. This keeps the static prefix minimal, raising cache hit rates.
9. Takeaways for You
Whether you use Claude Code or build your own agent from scratch, the same rules apply: 1. Build prompts in order: system instructions → tool definitions → reference docs → conversation history 2. Update information via messages, never by editing the system prompt 3. Never switch models or tools mid-conversation 4. Use sub-agents for tasks that need a different model 5. Use cache-safe forking when compressing context 6. Monitor cache hit rate the way you monitor uptime
Claude Code was designed around prompt caching from day one. If you're building the next great agent, start with this rule too.
References: