Introduction
In a twenty-turn conversation where only the last message is new, most LLMs still re-read everything from turn one—every system instruction, every tool definition, every prior message. That invisible rework is billed to you in full. Anthropic engineers crunched the numbers: a 100,000-token conversation on Claude Sonnet costs ~$0.30 per turn without caching ($6 over 20 turns), but with prompt caching enabled it's a one-time $0.375 write, then ~$0.03 per turn afterward. Roughly 90% savings.
How Prompt Caching Works
Every request triggers prefill—encoding the entire input into a KV Cache—which accounts for most inference cost and latency. Prompt caching is disarmingly simple: you mark a breakpoint in the request; the backend stores the encoding of everything up to that breakpoint; when the prefix is identical next time, it's reused, skipping redundant computation. No architecture changes, no new algorithms—just engineering-level reuse.
The Economics
- Cache hits cost ~10% of normal input price.
- First write costs 1.25x (25% premium), then saves 90% on every subsequent hit—profitable as long as the conversation lasts more than one turn.
- Default TTL is 5 minutes, auto-renewed on each request; a paid 1-hour tier exists for long-context scenarios.
- Minimum length: 1024 tokens (~4096 for newer models), so short prompts don't benefit. This is a long-conversation / agent-oriented feature.
- Latency also improves: the more prefix you reuse, the faster the TTFT (Time To First Token), which is what users actually feel in multi-turn chat.
- Lazy Loading: don't register all tools up front—tool definitions sit in the prefix, so fewer, later-loaded tools mean a shorter, more stable prefix.
- Plan Mode: let the model output a plan, get user confirmation, then execute—but switch modes via messages, not by editing system instructions, or the cache breaks.
- Cache-Safe Forking: when compressing, summarizing, or forking subtasks, the fork must share the main conversation's stable prefix (instructions, tools, project docs); only the message layer may diverge.
- Anthropic engineers' sharing on Claude Code prompt cache best practices
- easy-learn-ai project's Prompt Cache interactive tutorial (15 chapters)
Cache-Friendly Prompt Structure
Since caching is exact prefix matching, ordering in your prompt determines your hit rate. Anthropic's best practice layers content front-to-back:
1. System instructions + tool definitions — fixed across all sessions; never change them.
2. Project documentation (e.g., CLAUDE.md) — shared within a project, relatively stable.
3. Current session context — open files, today's goals.
4. Chat messages — appended turn by turn.
Rule of thumb: the more stable the content, the earlier it goes. A per-second timestamp at the front, or unordered containers for tool definitions, will kill the cache every time.
Three Cache-Breaking Pitfalls
1. Embedding datetime.now() in system instructions. The prefix changes every second. Fix: don't mutate instructions—inject updates as a <system-reminder> block inside the next user message. Keep the foundation fixed; let the messages flow.
2. Dict/set tool definitions. Unordered serialization shuffles the prefix each request. Use ordered lists.
3. Changing even one tool parameter field. Like a broken chain link—everything after it is invalidated. Design tools for stability; prefer state fields over adding/removing parameters.
Don't Switch Models Mid-Conversation
Counter-intuitive but true: cache is model-bound. Routing a trivial request to a cheaper model (e.g., Haiku instead of Opus) invalidates the entire accumulated cache, and rebuilding it often costs more than the model-price difference you saved. Claude Code's strategy: one model for the main conversation throughout. For light tasks, spawn subtasks—subagents carry their own independent context and cache, without polluting the main chain. The main model writes a condensed handoff; the subagent returns only results. Claude Code's Explore agents (Haiku-powered search/code-browsing/summarization) work exactly this way.
Warning for relay/proxy providers: caches are isolated per account. Account-pool rotation tanks your hit rate.
Advanced Techniques
Conclusion
Everything reduces to one constraint: the cache is prefix matching. Any change at any position invalidates everything after it. Design around it: use messages instead of instruction edits, don't swap tools or models mid-conversation, monitor cache hit rate like uptime, and always share prefixes when forking. With agent contexts routinely exceeding 100k tokens, prompt caching has shifted from optional optimization to a requirement—otherwise you're silently paying ~90% more on every API call.
References