Cache Rules Everything: How the 90% of Money Wasted in AI Conversations Gets Saved
> Source commit: 515b759 (easy-learn-ai)
The Old Engineer's Mantra
"Cache rules everything" has been an engineering truism since the early web: instead of assembling a page from scratch each time, reuse the parts that rarely change. In the AI agent era, the same principle applies—only now we're caching what the model has already computed.
The Problem: AI's Redundant Labor
In a 20-turn conversation with Claude Code, every request forces the model to re-encode the entire prior context from scratch. This stage is called Prefill, and it dominates both latency and cost. Turn 20 contains turns 1–19 verbatim—yet the model recomputes everything, like rewriting the table of contents and earlier chapters every time you add a paragraph to a paper.
The Fix: Prompt Caching
Mark a breakpoint in your request; the platform stores the encoding of everything before it (KV Cache) and reuses it whenever the prefix matches exactly.
The real numbers:
- Cache hits cost ~10% of normal input price. First write costs 1.25x (a 25% premium), then every subsequent hit saves 90%.
- Default TTL is 5 minutes, auto-renewed on each request; a paid 1-hour option exists.
- Minimum cacheable length: typically 1,024 tokens (4,096 on newer models).
Infrastructure-Grade Priority
Anthropic treats cache hit rate like server uptime. When it drops, engineers declare SEVs—full incident response, not just a warning. Higher hit rates also let Anthropic offer paid users more generous usage limits. No cache, no Claude Code: long multi-turn sessions would explode in cost and latency.
The Core Principle: Prefix Matching
Any change at any position in the prefix invalidates everything after it—like dominoes. Every best practice follows from this constraint.
Order Your Prompt
1. First: system prompt and tool definitions (fixed, shared across sessions). 2. Second: project documents (shared within a project). 3. Third: current session context. 4. Last: chat messages (grows one message per turn).
The more stable, the earlier it goes.
Three Pitfalls
1. Embedding the current time in fixed instructions — changes every second; pass time dynamically in messages instead.
2. Tool definitions in unordered containers (Python set, JS Object) — request order varies each time; use ordered arrays.
3. Changing a tool's parameters — even one field breaks the entire prefix.
Don't Touch the Instructions
Need to update time or file state? Don't edit the system prompt—inject a "system reminder" into the next user message. System prompt is the foundation: nailed down. Messages are the flow: change freely.
Don't Switch Models
Caches are bound to a model. Switching invalidates everything and rebuilding costs more than letting the big model answer the easy question. Claude Code keeps one model for the main conversation; small models get delegated sub-tasks.
Sub-Tasks: Independent Caches
The main model writes a hand-off message condensing context; the sub-task runs in its own context and returns only results (like Claude Code's exploration mode). You don't let the intern use your workstation—you give them a separate machine and clear instructions. Note for API resellers/proxies: caches are isolated per account—rotating accounts mid-conversation tanks hit rates.
Don't Touch the Tools
Trimming 33 tools down to 3 for the current task looks like an optimization but severs the cache chain—rebuilding everything costs far more than the token cost of the unused definitions.
Plan Mode: State via Tools, Not Edits
Instead of removing execution tools in plan mode, Claude Code keeps all tools and adds two special ones—"enter plan" and "exit plan". The constraint is conveyed via an in-conversation system message, not a system prompt change. Distinction: system prompt = fixed, inside the cache prefix; system message = flowing, doesn't affect the prefix. Bonus: the model can decide on its own when to plan.
Lazy Loading: Index Cards
Dozens of external tools? Ship only lightweight stubs (name only). When the model actually needs one, it fetches the full definition via tool search (now available via API). The prefix stays stable—like a library catalog: browse the index, then fetch the book.
Compaction and Cache-Safe Forking
When the context window fills, the conversation must be summarized. Naively running summarization with a different system prompt (no tool definitions) misses the cache entirely and pays full price. Anthropic's cache-safe forking: the compaction request reuses the exact system prompt, context, tool definitions, and full message history, then appends the compaction instruction as a new user message. The incremental cost is only that final instruction. Keep a compaction buffer so you compress before the window is completely full.
The Seven Rules, Recapped
1. Prefix matching decides everything—any change invalidates all content after it. 2. Use messages, not instruction edits, for mode switches and updates. 3. Never swap tools or models mid-conversation—use tools for state, lazy loading for scaling tool sets. 4. Monitor cache hit rate like uptime. 5. Forks must share the main conversation's prefix—compaction, summaries, sub-tasks all reuse identical parameters.
Closing: Constraint as Framework
This looks like caching optimization, but it's a philosophy: accept the unchangeable constraint first, then design the system around it. In the agent era, when every token is billed and conversations grow toward the context limit, caching is no longer an optional optimization—it is the precondition for the system to exist at all.
Cache rules everything.