When AI Learns to Take Notes: The Secrets Behind Prompt Caching

Forum topic · 小凯 · 2026-05-13

Summary

This forum post explains prompt caching for large language models through a library metaphor: an LLM normally re-encodes the entire conversation history on every turn, like a librarian re-reading a book from page one. Prompt caching stores the encoded prefix up to a fixed breakpoint and reuses it when the next input shares an identical prefix. The author quantifies the savings on a 100k-token document: without caching, 10 turns cost about $3; with caching (25% write overhead, then roughly 10% of full price per turn), total cost drops to about $0.645 — nearly 80% saved — while time-to-first-token also falls. Caches default to a 5-minute TTL with auto-renewal and require a 1,024–4,096 token minimum prefix. Drawing on Anthropic's Claude Code practices, the post lists key lessons: keep system prompts static and switch modes via injected messages; use lazy-loaded tool placeholders so tool lists never change the prefix; isolate subtasks with their own caches and avoid shared API account pools; and compress long histories via cache-safe forking, reusing the exact main-conversation prefix. The unifying principle is prefix matching: any change, even one character, invalidates everything after it. The post argues prompt caching is infrastructure-level (Anthropic monitors hit rate with on-call alerts) because it makes affordable long multi-turn agent workflows possible.

This article is based on technical content from the easy-learn-ai project, commit 515b759, a 15-chapter interactive animated lecture on Prompt Caching, inspired by Anthropic engineers' best practices on Claude Code prompt cache design.

An Absurd Scenario

Imagine asking a librarian what the first chapter of *One Hundred Years of Solitude* is about. The librarian reads the book from the beginning to the end of chapter one. The next day you ask a follow-up question — and the librarian again re-reads from page one before answering. Absurd, right? Yet this is exactly how LLMs have long worked: in a 20-turn conversation, turn 20 re-encodes the identical history of turns 1–19 from scratch.

Until someone realized: if the earlier content hasn't changed, why not let the model "take notes"?

The Secret: Prompt Caching

Prompt caching marks a "breakpoint" on the input — like a bookmark in a book. The encoded prefix up to that point is stored. If the next input shares an identical prefix, the model reuses the cache instead of recomputing.

Consider a 100k-token document analysis:

Without cache: ~$0.30 per turn × 10 turns = $3
With cache: first pass costs 25% extra for the cache write ($0.375), then each subsequent turn is ~10% of full price ($0.03). Total: 0.375 + 9 × 0.03 = $0.645 — nearly 80% saved. Savings compound over 20–30 turns.

Speed improves too: time-to-first-token (TTFT) drops significantly because the model no longer waits for a full re-read of the history. Caches default to a 5-minute lifetime (auto-renewed while you keep chatting), and activation requires a minimum of 1,024–4,096 tokens — well within reach for 100k-token documents.

Why Anthropic Treats It as Infrastructure

Anthropic monitors prompt cache hit rate as an infrastructure-grade metric: if it drops, an SEV alert pages engineers. Without caching, Claude Code — which routinely runs dozens of deep turns over a codebase — would face exploding latency and cost.

The core principle behind everything is prefix matching: the cache only recognizes an exactly identical beginning. One character of divergence — even a space — forces recomputation of everything after it. Hence the governing constraint: don't change instructions, don't touch tools.

Claude Code's Lessons

1. Don't modify system prompts

Switching modes (e.g., "execute" → "planning") by editing system prompts changes the prefix and kills the cache. Instead, Claude Code uses "enter planning" / "exit planning" tools and injects mode constraints as conversation messages. Static instructions stay in the cached prefix; flowing messages don't affect it. The model can even decide on its own when to enter planning mode.

2. Don't swap tools mid-conversation

Loading and unloading tool definitions changes the prefix and invalidates the cache. The solution is lazy loading: the system exposes lightweight placeholder definitions (like a library index card listing titles only). When needed, a special "tool search" tool fetches the full definition. The prefix stays rock-solid.

3. Isolate subtasks

Delegating subtasks (e.g., code search) to a smaller model can pollute the main conversation with intermediate outputs, lengthening the prefix and lowering hit rate. Better: the main model writes a task handoff; the subtask runs with its own independent cache and returns only results.

A subtle risk: shared account pools. If many users share one API account, each has different prefixes, diluting hit rates to near zero — in extreme cases, the platform bans the account. Anthropic engineers have genuinely learned this the hard way.

4. Compress history with cache-safe forking

When the context window (e.g., 200k tokens) fills up, history must be compressed. The naive approach — a fresh request with different instructions and no tool definitions — fails to match the main conversation's cache at all and pays full price. Anthropic's cache-safe forking keeps the compression request's prefix identical to the main conversation (same system prompt, user context, tool definitions), appends the history, and adds one final user message: "compress the above into a summary." From the backend's view, the request is nearly identical to the last one, so the cache is reused; only the final compression instruction costs extra. A compression buffer prevents the request itself from overflowing the window.

Back to First Principles

Everything follows from one rule — prefix matching:

1. Use messages instead of instruction changes — messages flow, the prefix doesn't. 2. Never switch tools or models mid-conversation — both are part of the prefix. 3. Monitor cache hit rate like uptime — it's infrastructure, not an option. 4. Forked operations (compression, subtasks, rollbacks) must share the main conversation's prefix.

You don't need to memorize seven rules — just understand: the cache only recognizes an identical beginning.

A Revolution in "Memory"

Prompt caching looks like a cost trick, but its real significance is enabling long conversations at all. Without it, AI coding assistants couldn't work over large codebases for dozens of turns; "agents" and "deep research" would remain toy demos. It solves an ancient computing problem: how to remember the past while efficiently processing new input. Prompt caching installs a cache layer for the model's only form of memory — the context window.

When AI learns to "take notes," it's no longer the librarian who starts from page one every time. It bookmarks pages, writes summaries, checks the table of contents. What looks "smart" isn't complexity — it's the most basic economic principle: don't spend effort on problems already solved.

---

Source commit: 515b759 · Project: easy-learn-ai