Cache Rules Everything: How to Save the 90% You're Overpaying in AI Conversations

Forum topic · 小凯 · 2026-05-15

Summary

This article explains how prompt caching dramatically reduces the cost and latency of long AI conversations, using Claude Code as a case study. Every conversational turn re-encodes the full context (Prefill), which is the main driver of cost and time-to-first-token. Prompt caching stores the KV Cache of a request prefix and reuses it when the next request shares the same prefix: cache hits cost roughly 10% of normal input pricing, while the first write costs 1.25x, with a default 5-minute TTL (extendable). The core constraint is prefix matching — any change in the prefix invalidates everything after it. Derived best practices include: ordering prompts from static to dynamic (system instructions, tool definitions, project docs, then messages), never embedding timestamps or reordering tool containers in the prefix, keeping one model per conversation, delegating to subtasks with independent contexts, expressing mode changes via messages rather than tool-set edits, lazy-loading tool definitions, and using cache-safe forking for context compaction. Anthropic treats cache hit rate as an infrastructure-level metric. Without caching, long-context AI coding assistants like Claude Code would be economically unviable.

*Translated and adapted from a zhichai.net forum post (source commit: 515b759, easy-learn-ai).*

Introduction: "Cache Rules Everything"

There's an old saying in engineering: "Cache rules everything." It dates back to the early web, when engineers realized it was cheaper to reuse pre-built page components than to assemble everything from scratch. In the AI Agent era, the saying still holds — except what's being cached is no longer web pages, but the computation a large model has already done.

The Problem: AI's Repeated Labor

Imagine using Claude Code on a complex project. Over 20 turns, every new request re-encodes the entire prior context from the first token. This encoding process is called Prefill, and it's the biggest source of both latency and cost. Your 20th request is 95% identical to your 19th — but without caching, the model recomputes everything anyway.

The Fix: Prompt Caching

Prompt caching works like this: you mark a breakpoint in the request; everything up to that point gets its encoded results stored. If a future request has the identical prefix, the computation is reused.

At the technical level, these encoded results are the KV Cache — the intermediate vectors and attention computations the model produces while processing text.

The Real Numbers

Cache hits cost 10% of normal input pricing. The first write costs 1.25x — but every subsequent use saves 90%.
Default TTL is 5 minutes, auto-renewed by requests; a paid 1-hour option exists.
Minimum cacheable length: generally 1,024 tokens (4,096 for newer models). Short prompts don't qualify.

Example: a 100k-character conversation on Claude Sonnet —

Without caching: $0.30 per turn
With caching: $0.375 first write, then $0.03 per turn

Over 10 turns, that's ~90% savings on input costs. Latency improves too — the more you skip recomputing, the faster the first token arrives (TTFT, Time To First Token).

Infrastructure-Grade Priority

Inside Anthropic, cache hit rate is monitored like server uptime. A drop in hit rate triggers on-call alerts and engineers declare SEVs — full incident-response process. High hit rates don't just save money; they let Anthropic offer paying users more generous usage limits. For Claude Code, caching isn't an optimization — it's the precondition for the product to exist at all. Long multi-turn coding sessions would explode in cost and latency otherwise.

The Core Principle: Prefix Matching

The cache works on prefix matching: if the next request's prefix matches, prior computation is reused. Any change anywhere in the prefix invalidates everything after it — like dominoes. Every best practice below follows from this single constraint.

Best Practice 1: Order Your Prompt by Volatility

1. Front: system instructions and tool definitions (fixed, shared across sessions) 2. Second: project documentation (shared within a project) 3. Third: session context (per-conversation) 4. Last: chat messages (append-only per turn)

The less something changes, the further forward it goes.

The Three Common Pitfalls

1. Embedding current time in fixed instructions. It changes every second, killing the cache. Pass timestamps dynamically in the message layer. 2. Storing tool definitions in unordered containers (Python set, JS Object). Serialization order varies per request, breaking the prefix. Use ordered arrays. 3. Modifying tool parameters. Even one field change invalidates the entire prefix cache.

Best Practice 2: Don't Edit Instructions — Append Messages

When information goes stale (time, file state), don't modify the system prompt. Instead, attach a "system reminder" inside the next user message. System instructions are the foundation — nailed down. Messages are the flowing water — change freely.

Best Practice 3: Don't Switch Models Mid-Conversation

Cache is bound to the model. Switching models invalidates all accumulated cache and forces a rebuild that often costs more than just letting the big model answer the easy question. Claude Code keeps one model for the main conversation and delegates simpler work to subtasks.

Subtasks: Independent Caches

Subtasks get their own context and cache, so they don't pollute the main chain. The main model writes a hand-off message summarizing context, the subtask executes independently, and only results return. (Claude Code's exploration mode works this way.) It's like giving an intern a separate machine with clear instructions — not your own workstation.

A note for API-proxy operators: caches are isolated per account. Rotating accounts mid-conversation tanks your hit rate.

Best Practice 4: Don't Touch the Tool Set

Trimming unused tools mid-conversation looks like optimization but breaks the cache and forces a full rebuild — far costlier than the tokens saved.

Plan Mode via Tools, Not Tool Removal

Claude Code's plan mode keeps all tools in place and adds two special tools — "enter planning" and "exit planning." The constraint "no execution while planning" is conveyed by inserting a system message into the conversation flow, not by editing the system prompt:

System Prompt: fixed, part of the cached prefix.
System Message: fluid, doesn't affect the prefix.

Bonus: the model can decide on its own when to enter plan mode.

Lazy Loading: Library Index Cards

For dozens of external tools, Anthropic uses lazy loading: initially only lightweight stubs (tool names without full parameter definitions) sit in the prefix. When the model needs a tool, a "tool search" fetches the full definition. The prefix stays stable. This tool-search capability is now available via the public API.

Cache-Safe Forking for Compaction

When the context window fills, the conversation must be compacted into a summary. Naively sending it to a separate request with different instructions pays full, un-cached prices for the whole history. Anthropic's fix is cache-safe forking: the compaction request reuses the exact same system instructions, user context, tool definitions, and message history as the main conversation, appending only the compaction instruction as a new user message. The incremental cost is just that final instruction. A compaction buffer is reserved in advance so the summary has room.

Recap: Everything Points Back to Prefix Matching

1. Prefix matching decides everything. Any change invalidates all subsequent content. 2. Replace instruction edits with messages. 3. Never switch tools or models mid-conversation. Use tools to express state transitions; use lazy loading instead of add/remove. 4. Monitor cache hit rate like uptime. 5. Forks must share the main conversation's prefix — compaction, summaries, subtasks alike.

Conclusion: Constraints as Framework

This looks like cache optimization, but it's really a design philosophy: identify the non-negotiable constraint first, then build the entire system around it. Prefix matching is that constraint, and Anthropic built Claude Code's architecture around it. In the AI Agent era, when every token is billed and every conversation can fill the context window, caching is no longer an optional optimization — it's the precondition for the system to exist.

The old engineering saying holds: cache rules everything.