Who Built the Road but Never Collected the Toll: A Feynman-Lens Look at PD Disaggregation and the Billing Black Hole
> Note: This post applies a Feynman-style lens based on public statements and verifiable facts. It is not Feynman's own view — he passed away in 1988, long before large language models.
Start with a picture.
A toll booth. A truck pulls up, and the attendant glances at the manifest: "This cargo was stored in our warehouse last week — you get the 'already stocked' discount." Swipe, let it through.
But the cargo never entered the warehouse. A branch factory in another city copied a "shadow" of the whole shipment overnight and trucked it to this toll booth. The real labor, the real cost, happened at the factory — and nobody paid for it. The booth only sees "the goods are here", so the entire shipment crosses at the cheapest rate.
That toll booth is the billing module of an LLM inference service. The "cargo shadow" is the KV Cache. The overnight branch factory is the Prefill node.
What It Actually Does (Beyond the Jargon)
Strip away the names — Prefill, Decode, PD disaggregation, KV Connector — and LLM text generation is just two steps:
Step 1: Prefill. The user's entire prompt is fed into the model at once, computing the KV Cache for every token in parallel. This is large matrix multiplication — compute-bound, bottlenecked by GPU FLOPS. It happens once per request, like reading a whole reference book in one sitting and marking the key points.
Step 2: Decode. From those notes, tokens are generated one by one. Each new token requires revisiting everything before it. This is memory-bandwidth-bound, bottlenecked by HBM bandwidth, and dominates overall request latency — like re-reading your marked-up book for every word you write.
One sentence: Prefill is "read everything at once, take full notes" (compute work); Decode is "write word by word, flipping through notes each time" (bandwidth work). Two different jobs with two different bottlenecks — naturally they should be separated.
Why Co-location Is Awkward
Traditional deployments put both steps on the same machine. Problems: Prefill's large matrix multiplications stall running Decode streams, making ITL (inter-token latency) erratic. Worse, resources idle — memory sits unused during Prefill; compute sits unused during Decode.
PD disaggregation: Prefill nodes run the compute-bound job and produce the KV Cache; Decode nodes run the bandwidth-bound job and consume it to generate tokens. Connectors like NIXL, LMCache, and Mooncake move the KV Cache from P nodes to D nodes. Each side scales independently, uses different hardware, and tunes its own batch size.
Sounds clean. But underneath lies a billing black hole.
The Shape of the Black Hole
Token billing has three tiers: input price, output price, and cache-hit price. Cache-hit price is far below input price — a "prefix cache hit" means compute was saved, so it deserves a discount.
Whether a hit occurred is determined by the field prompt_tokens_details.cached_tokens — the number of prompt tokens that hit an existing KV Cache.
Here's the trap: under PD disaggregation, the real Prefill computation happens on the Prefill node, which then ships the entire KV Cache via a connector to the Decode node. From the Decode node's view, the whole prompt's KV Cache "is already here" — so it reports everything as a cache hit.
If the billing module trusts the Decode-side cached_tokens, the entire prompt gets billed at the cheap cache-hit rate, while the actual Prefill compute cost never lands in the "input tokens" tier. Result: revenue quietly leaks every day. The Prefill node did the road-building; the toll booth waved the truck through at the "road was already there" rate.
This isn't speculation. The vLLM community has debated it across PRs and Issues:
- vLLM RFC #24256 (*Add a cache hit threshold to handle Preemptions in PD-Disaggregation*) explicitly states that cache hit rate "includes cache from external and previously offloaded KV-Cache, obtained via the KVConnector, and not just the local APC". In other words: cross-instance migrated KV counts as a "hit" — the very source of the black hole.
- The companion PR #24520 implements a "cache hit threshold", aiming to distinguish local APC hits from connector-migrated cache so the routing layer can do admission control — in plain terms, bringing cross-instance cache behavior back into the measurable, billable realm.
A Fresh Corroboration
Checking the widely cited DeepSeek V4-Pro pricing example — "hit 0.025 CNY/M, miss 3 CNY/M, 120x gap" — confirms it was the real pre-August 17, 2026 price. But on 2026-08-17, DeepSeek made a major repricing with peak/off-peak tiers:
| Tier | Cache hit (CNY/M) | Cache miss (CNY/M) | Gap | |---|---|---|---| | Before 8-17 | 0.025 | 3 | 120x | | After 8-17, off-peak | 0.15 | 4.5 | 30x | | After 8-17, peak | 0.3 | 9 | 30x |
Two things stand out:
1. The cache-hit price rose 1100% (12x) — the steepest increase of any tier. Why would an inference vendor suddenly raise its "cheap tier" tenfold? A reasonable explanation: cache-hit billing was severely underpriced, and it sits at the intersection of profit and risk. Vendors themselves are scrambling to get the "hit" accounting right — commercial proof that cache-hit billing is the lifeblood.
2. The hole's absolute leakage got bigger. Under old prices, mis-billing 1M "should-be-miss" tokens as hits lost 3 − 0.025 = 2.975 CNY. At new off-peak prices: 4.5 − 0.15 = 4.35 CNY; at peak: 9 − 0.3 = 8.7 CNY. The ratio shrank from 120x to 30x, but every leaked token now costs 1.5x to 3x more real money. The black hole didn't shrink — it changed shape and swallowed harder.
(Remark: from 8-23, weekends bill entirely at off-peak rates; this doesn't change the conclusion.)
One-Sentence Wrap-Up
PD disaggregation smartly splits "reading and note-taking" from "writing the essay" across two machines. But it creates an illusion: the Decode node mistakes the entire shipped-over notes for "cache I already had", and billing waves the whole prompt through at the cheapest hit rate — the factory that did the work never got paid, and the toll booth thinks it scored a bargain.
Nature cannot be fooled, but ledgers can — until month-end reconciliation, or until a vendor raises the hit price twelvefold, and you discover the crack was leaking an ocean.
So here's the Feynman question: "That cached_tokens in your report — is it a genuine local hit, or something someone else computed and shipped over? Until you can tell them apart, don't claim you've got your accounting figured out."
That's all there is to it.
---
*Generated 2026-08-25 00:45 · Source SHA256: 30bcbbce40688933 · Feynman Lens series*