Introduction
A large chunk of your LLM API bill is spent on computation you shouldn't have to pay for twice. This post traces "caching" through the entire LLM stack — from the attention mechanism, to API provider pricing, to real-world application engineering — and explains why DeepSeek is called a "cyber philanthropist," how token resellers profit from information asymmetry, and what the cache-heavy source code of Claude Code reveals about cost optimization.
Key points
- KV Cache stores the Key/Value matrices of past tokens during autoregressive generation, cutting per-step attention complexity from O(n²) to O(n). Without it, generating 1,000 tokens requires ~1 million attention computations vs. ~1,000 with caching.
- Only K and V are cached — past Q vectors are useless because each step only needs the current token's Q to query historical K/V.
- Prompt Cache operates at the API layer: providers cache the KV state of request prefixes. When a later request's prefix matches (from token 0), prefill is skipped. Anthropic and DeepSeek price cache hits at 0.1x input price (90% off); OpenAI at 0.25–0.5x. Cache blocks require exact prefix match with a minimum of ~1,024 tokens; TTLs range from 5 minutes (Anthropic, extendable on hit) to disk-persistent storage (DeepSeek, 10–30 minutes).
- DeepSeek's disk cache: its MLA (Multi-head Latent Attention) architecture compresses KV cache to ~7–10% of a standard Transformer's size, small enough to store on cheap distributed disks rather than expensive GPU memory. Reported figures: 56.3% of input tokens hit disk cache within 24 hours; V4-Flash hit price is 1/100 of uncached input; a 128K-token prompt sees first-token latency drop from ~13s (miss) to ~0.5s (hit) — a 26x improvement.
- Counter-intuitive implication: since cache-hit tokens are nearly free, on DeepSeek adding more context (few-shot examples, project docs, full conversation history with a stable prefix) costs nothing extra and improves quality. The old advice "keep prompts short" inverts into "what's most valuable to put in nearly-free stable context?"
- Static/dynamic prompt boundary: system prompt and tool definitions live in the cacheable static prefix; dynamic sections are explicitly marked (a
DANGEROUS_prefix warns that moving content into the static zone would invalidate the cache, raising costs ~12.5x from $0.30 to $3.75 per MTok). - Cache sharing: the YOLO intent classifier and the main query engine share one cached prefix, paying the write cost once.
- Auto compact: when conversation history balloons, it is folded into summaries to keep the dynamic tail from crowding out the cached prefix. Advice: compact early per sub-task, and avoid resuming sessions (serialization differences can invalidate the cache).
- CLAUDE.md best practice: keep it under ~50 lines as an index pointing to where information lives, with no dynamic content — every edit invalidates the prefix cache.
- Wilson Wu: Understanding KV-Cache — https://wilsonwu.me/blog/2025/kv-cache-in-llm/
- KV Cache inference optimization — https://zhuanlan.zhihu.com/p/700197845
- Prompt Caching — https://www.51cto.com/aigc/11620.html
- Sam Rose (Ngrok) on Prompt Caching — https://developer.volcengine.com/articles/7587308087636459574
- DeepSeek API docs: Context Caching on Disk — https://api-docs.deepseek.com/zh-cn/news/news0802
- Claude Guide: Prompt Caching — https://yeasy.gitbook.io/claude_guide/di-si-bu-fen-shi-zhan-pian/10_optimization/10.2_caching
- Claude Code Prompt Cache deep dive — https://developer.aliyun.com/article/1732421
- 21 Finance: The token reseller business — https://www.21jingji.com/article/20260506/herald/5b6fb5497a3a0bb307e1f01363780b88.html
The token reseller problem
The post describes three tiers of API resellers: legitimate bulk resellers (thin margins), account-pool operators (huge margins, high risk), and bait-and-switch resellers (advertising Claude but serving cheaper domestic models). The most hidden markup is cache arbitrage: resellers charge users full "cache miss" prices while settling at upstream "cache hit" rates (1/10 to 1/120), hiding prompt_cache_hit_tokens/prompt_cache_miss_tokens fields, or deliberately injecting per-request prefixes to poison the cache so hits never occur. One cited case: a developer burning 89M tokens in a day should have paid ~$2.50 with 98% DeepSeek cache hits, but a full-price reseller could charge ~$25.
Claude Code: caching as infrastructure
Claude Code's source code treats caching as a foundational design constraint:
Measured results: 1.8M input tokens at full price on Opus cost $9.00; with default caching (70% hit) ~$2.70; optimized to 92% hit rate, ~$1.40 — 15.6% of the uncached cost, an 84% saving.
The full stack of caching
1. Model layer: attention (Q/K/V), MLA compression (DeepSeek), FFN 2. Inference layer: KV Cache, PagedAttention (vLLM/TensorRT-LLM), quantization (FP16/INT8/INT4) 3. API layer: prompt cache prefix matching, TTL management, 0.1x hit pricing 4. Application layer: static-prefix/dynamic-suffix prompt structure, auto compact, cache sharing across components
Three rules for developers: keep your prefix stable (the precondition for any caching); compact early rather than hoarding history; and remember cache-hit tokens are nearly free — on DeepSeek, adding context beats trimming it.