PD Disaggregation's Billing Blind Spot: When Migrated KV Cache Gets Charged as a Cache Hit

Forum topic · QianXun · 2026-08-24

Summary

This forum post uses a Feynman-style analogy to explain a billing flaw in PD (Prefill-Decode) disaggregated LLM inference architectures. In PD disaggregation, Prefill nodes compute the KV Cache for the entire prompt, then transfer it to Decode nodes via connectors such as NIXL, LMCache, or Mooncake. From the Decode node's perspective, the whole prompt appears as 'already cached', so telemetry reports prompt_tokens_details.cached_tokens as a full hit. Billing systems that trust this field charge the cheapest cache-hit rate for tokens whose full prefill compute was actually performed and never billed at input rates. The author cites vLLM RFC #24256 and PR #24520, which acknowledge that cache-hit metrics include KV cache migrated via KVConnector, not just local APC hits, motivating a cache-hit threshold for admission control. The post also discusses DeepSeek's pricing (dated August 2026 in the source), where the cache-hit price reportedly rose ~12x, narrowing the hit/miss ratio from 120x to 30x but increasing absolute revenue leakage per mis-billed token. Key takeaway: operators must distinguish genuinely local cache hits from KV cache transferred from Prefill nodes, or prefill compute goes uncompensated.

PD Disaggregation's Billing Blind Spot: Who Built the Road but Never Collected the Toll

> Note: This uses a "Feynman lens" to reason from public statements and verifiable facts — not the views of Feynman himself.

The picture: a toll booth waves a truck through at the cheapest rate because "the goods were already in our warehouse." In reality, the goods never entered that warehouse — a factory in another city copied a "shadow" of the entire shipment and trucked it over a dedicated line overnight. The toll booth only sees the goods sitting there, so the whole shipment passes at the discount rate, and the factory that did the real work never gets paid.

The toll booth is the billing module of an LLM inference service.
The shadow of the goods is the KV Cache.
The overnight factory is the Prefill node.

What Prefill and Decode Actually Do

LLM text generation has two steps:

1. Prefill — the whole prompt is processed in parallel, computing the KV Cache for every token. This is large matrix multiplication: compute-bound, limited by GPU FLOPS. It happens once per request. 2. Decode — tokens are generated one at a time, each step attending over all previous KV entries. This is memory-bandwidth-bound, limited by HBM bandwidth, and dominates latency.

One is a compute job; the other is a bandwidth job. Mixing them on the same machine causes Prefill's big matmuls to stall ongoing Decode streams (jittery ITL), and leaves resources idle (memory free during prefill, compute free during decode).

PD Disaggregation

PD disaggregation splits the workload: Prefill nodes run the compute job and produce the KV Cache; Decode nodes run the bandwidth job and consume it. Connectors such as NIXL, LMCache, Mooncake transfer the KV Cache from P nodes to D nodes. Each side scales independently, uses different hardware, and tunes its own batch size.

The Billing Black Hole

Token billing has three tiers: input price, output price, and cache-hit price (the cheapest, since a prefix-cache hit supposedly means compute was saved). The hit/miss decision relies on prompt_tokens_details.cached_tokens.

The trap: under PD disaggregation, the prompt's Prefill actually ran on the Prefill node, which then shipped the entire KV Cache to the Decode node. From the Decode node's view, the whole prompt's KV Cache "is already here" — so it reports the entire prompt as cache hits. A billing system that only reads the Decode-side cached_tokens charges everything at the cheapest hit rate, while the actual Prefill compute is never billed at the input tier.

Result: significant revenue leakage every day.

This is not speculation — it is visible in the vLLM community:

vLLM RFC #24256 (*Add a cache hit threshold to handle Preemptions in PD-Disaggregation*) explicitly states that cache-hit rate "includes cache from external and previously offloaded KV-Cache, obtained via the KVConnector, and not just the local APC" — i.e., KV migrated across instances counts as a "hit." This is the origin of the black hole.
PR #24520 implements a cache-hit threshold to distinguish local APC hits from connector-migrated cache, letting the router do admission control — effectively trying to bring cross-instance cache migration back into the "meterable, billable" category.

A Fresh Corroboration from Pricing

The post examines DeepSeek pricing figures cited in the source (dated August 2026):

| Tier | Cache hit (CNY/M tokens) | Cache miss (CNY/M tokens) | Ratio | |---|---|---|---| | Before Aug 17 | 0.025 | 3 | 120x | | After Aug 17, off-peak | 0.15 | 4.5 | 30x | | After Aug 17, peak | 0.3 | 9 | 30x |

Two observations:

1. The cache-hit price rose ~1100% (12x) — the steepest increase of any tier. A plausible reading: cache-hit billing was historically underpriced and sits at the intersection of profit and risk; the vendor itself is trying to get the "hit" accounting right. 2. The absolute leakage grew. Under old prices, misclassifying 1M should-be-miss tokens as hits lost 3 − 0.025 = 2.975 CNY; off-peak now loses 4.5 − 0.15 = 4.35 CNY; peak loses 9 − 0.3 = 8.7 CNY. The ratio shrank from 120x to 30x, but each leaked token costs 1.5–3x more real money. The black hole didn't shrink — it changed shape and swallowed harder.

One-Sentence Summary

PD disaggregation wisely splits "taking reading notes" and "writing the essay" across two machines — but creates the illusion that the Decode node treats the entire transferred KV Cache as its own pre-existing cache. Billing then waives the whole prompt at the cheapest rate: the factory that did the work never got paid, while the toll booth thinks it found a bargain.

As Feynman would ask: "Is that cached_tokens in your report a genuine local hit, or something someone else computed and shipped over? Until you can tell them apart, don't claim your numbers are right."

That's all there is to it.