Introduction: "Cache Rules Everything"
An old engineering saying goes: "Cache rules everything." It dates back to the early web, when engineers realized that pre-building common page components beat reassembling pages from scratch. In the AI Agent era, the same principle applies — except now we cache what the model has already computed, not web pages.
The Problem: Redundant Computation in Long Conversations
When you use Claude Code for 20 conversation turns, each turn forces the model to re-encode all previous turns from scratch. This process is called Prefill, and it dominates both latency and cost. Turn 20 re-computes turns 1–19 exactly as before — pure wasted work, like recopying earlier chapters of a paper every time you add a paragraph.
The Solution: Prompt Caching
Prompt caching works in one sentence: mark a breakpoint in your request; the backend stores the encoding results up to that point; if the next request shares the same prefix, it reuses them and skips the computation. The stored encoding is the KV Cache — intermediate vector representations that can be reused across requests.
The Real Numbers
- Cached tokens cost ~10% of normal input price; the first write costs 1.25x (25% extra), then saves 90% per reuse.
- Default TTL is 5 minutes, auto-renewed on each hit for free; a 1-hour paid option exists.
- Minimum cacheable length: typically 1024 tokens (4096 for newer models). Short prompts can't be cached.
- Example: a 100k-token conversation on Claude Sonnet costs $0.30/turn without caching, $0.375 for the first cached turn, then $0.03 per turn — about 90% savings over 10 turns.
- Latency drops too: fewer re-computed tokens means faster Time To First Token (TTFT).
- Embedding the current time in fixed instructions — it changes every second and kills the cache. Pass time dynamically at the message layer.
- Storing tool definitions in unordered containers (Python
set, JSObject) — ordering differs per request, breaking the prefix. Use ordered arrays. - Modifying a tool's parameters — even one field invalidates the whole cache chain.
Infrastructure-Grade Priority
Inside Anthropic, cache hit rate is monitored like server uptime. A drop in hit rate triggers on-call alerts and full incident response ("declare SEVs"). Higher hit rates also let Anthropic offer paid users more generous usage limits. Without caching, there is no Claude Code — long, multi-turn coding sessions would be economically and latency-wise impossible.
The Core Principle: Prefix Matching
Caching relies on prefix matching. Any change at any position in the prefix invalidates everything after it — like dominoes. This single constraint drives every engineering decision below.
Best Practices
1. Order your prompt by stability. Front-load: (1) system instructions and tool definitions (shared across sessions), (2) project docs, (3) session context, (4) chat messages last. Least-changeable content goes first — like keeping reference books at the bottom of your desk stack.
2. Three common pitfalls:
4. Don't switch models mid-conversation. Caches are model-bound — switching invalidates everything, and rebuilding costs more than just letting the big model answer the easy question. Claude Code keeps one model for the main conversation; small models handle sub-tasks with independent contexts and caches. The main model writes a hand-off message, the sub-task executes, and only the result returns. (Like giving an intern a separate machine, not your workstation.) For proxy/relay users: caches are per-account — rotating accounts destroys hit rates.
5. Don't touch the tool set. Removing 30 of 33 tools to "clean up" breaks the cache and forces a full rebuild — far costlier than the space a few extra definitions take.
6. Express state transitions via tools, not tool changes. Claude Code's plan mode keeps all tools in place and adds two special tools — "enter planning" and "exit planning." The no-execution constraint is conveyed by inserting a system message into the conversation flow (not the system prompt). Bonus: the model can decide on its own when to enter plan mode.
7. Lazy loading for many tools. Instead of full definitions, start with lightweight stubs (tool names only). When the model needs a tool, it fetches the full definition via the tool search API. The prefix stays stable — like a library index card: browse the catalog, then fetch only the book you need.
8. Cache-safe forking for compaction. When compressing a full context window, don't send a differently-configured request — you'd pay full, undiscounted price and build a separate cache chain. Instead, reuse the main conversation's exact system instructions, context, tool definitions, and message history, appending the compaction instruction as a new final user message. The prefix cache applies fully; the only new cost is the instruction itself. Reserve a buffer for the summary output before the window fills.
Recap: Everything Points to Prefix Matching
1. Prefix matching decides everything — any change invalidates everything after it. 2. Replace instruction edits with messages. 3. Never switch tools or models mid-conversation; use tool-based state transitions and lazy loading. 4. Monitor cache hit rate like uptime — Anthropic treats cache breaks as incidents. 5. Forks (compaction, summaries, sub-tasks) must share the main conversation's prefix.
Conclusion: Constraint as Framework
This looks like cache optimization, but it's really a design philosophy: identify the unbreakable constraint first, then build the entire system around it. In the AI Agent era, where every token is billed and conversations grow to fill context windows, caching is no longer an optional optimization — it's the precondition for the system existing at all.
Cache rules everything.