> TL;DR: In multi-turn agentic inference, KV-Cache read speed has become the system bottleneck. DeepSeek's DualPath recruits the idle storage bandwidth of decode engines, letting KV-Cache reach prefill via two routes — direct from storage, or relayed through decode engines — with the system choosing whichever is faster. Offline inference speeds up 1.87x, online serving 1.96x.
1. The Problem: Compute Isn't the Bottleneck, I/O Is
Agent workloads are characterized by long context, short appends, and high KV-Cache reuse.
In DeepSeek's production trace, a typical agent session spans 157 turns with an average context of 32,721 tokens, appending only 429 new tokens per turn. The KV-Cache hit rate is 98.7%. This means each turn re-uses 98.7% of attention computation as "old work" — no recomputation needed, just reading stored KV-Cache back.
Compute pressure is actually small. The real pressure is on storage bandwidth.
Modern inference clusters use PD disaggregation (prefill-decode separation):
- Prefill engine: processes input prompts, computes KV-Cache
- Decode engine: generates output token by token
- Long prefill storage NIC queues → use DE path
- High decode compute load → use PE path
- Both idle → load balance across both
- All GPU traffic uses GPUDirect RDMA (no CPU copies)
- QoS isolation: model communication is high priority, reserved 99% bandwidth; KV-Cache is low priority, using only idle bandwidth
- InfiniBand uses Virtual Lanes; RoCE uses Traffic Class + DSCP marking
- Storage NIC queue lengths
- GPU compute load
- Request workload characteristics (context length, append size)
- DS 660B: up to 1.87x (64K context, 1024 agents)
- DS 27B: up to 1.78x, average 1.64x, 2.46x in extreme configurations
- Qwen 32B: same trend
- DS 27B: 1.67x throughput gain
- DS 660B: 2.25x throughput gain
- Average: 1.96x
- 98.7% of tokens are "old" — no computing needed, only reading
- Prefill is no longer "compute" but "read"
- System performance is decided by GB/s, not FLOPS
- Paper: DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
- arXiv: 2602.21548
- Authors: Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang
- Institutions: Peking University, Tsinghua University, DeepSeek-AI
- Date: 2026-02-25
- Core code: ~5,000 lines on an in-house inference framework, built on FlashMLA, DeepGEMM, DeepEP, 3FS
- Models tested: DeepSeek 660B (MoE + sparse attention), DeepSeek 27B, Qwen 32B
- Configurations: 2P4D (DS660B), 1P2D (Qwen32B), 1P1D (DS27B)
- SLOs: TTFT ≤ 4s, TPOT ≤ 50ms
- Key numbers: 1.87x offline, 1.96x online, 98.7% KV-Cache hit rate, bottleneck-free P/D ratio range 1/7–7/2
Each agent turn, the prefill engine must load the previous turn's KV-Cache from external storage (SSD/3FS). Meanwhile, the decode engine's storage NIC sits mostly idle — it only stores new KV-Cache back, with far less read pressure than prefill.
Result: prefill nodes' storage NICs are saturated and GPUs wait idle for data, while decode nodes' storage NICs and GPUs both idle.
A simple experiment proves this is the bottleneck: no matter how the P/D ratio is adjusted (1P1D, 2P1D, 1P2D), performance stays the same as long as total storage bandwidth is constant. Performance is locked by storage bandwidth, not GPU compute.
2. DualPath: Two Routes Into Prefill
Core insight: KV-Cache doesn't have to go from storage directly into prefill. It can first travel through the decode engine's storage NIC, then reach prefill via RDMA over the compute network.
Two paths:
| Path | Flow | Bandwidth consumed | |------|------|--------------------| | PE Read Path | Storage → prefill engine → compute | prefill storage NIC | | DE Read Path | Storage → decode engine → RDMA → prefill engine | decode storage NIC + compute network |
A global scheduler dynamically selects routes based on real-time load:
This pools the cluster's storage NIC bandwidth — prefill and decode storage bandwidth combined serve KV-Cache loading.
3. Three Engineering Challenges
1. Fine-grained transfer
Layerwise prefill is standard in modern inference (load only one layer's KV-Cache into GPU HBM at a time, swap when done). This shatters KV-Cache into countless small chunks. Transferring these fragments must have minimal overhead and overlap with compute — no waiting for transfers to finish.
DualPath's approach: while computing each layer, the next layer's KV-Cache is already being DMA'd into a DRAM buffer in the background. When a layer finishes, the next is read straight from the buffer — no waiting on storage.
2. Traffic isolation
The DE Read Path injects KV-Cache traffic into the compute network (RDMA). But that network already carries latency-sensitive collectives — AllToAll (MoE expert parallelism), ReduceScatter/AllGather (tensor/context parallelism). KV-Cache traffic must not steal their bandwidth.
DualPath's CNIC-centric traffic management:
3. Dynamic load balancing
Route selection can't be blind. If all requests take the DE path, decode engines get crushed by KV-Cache transfer, squeezing out model computation.
The scheduler watches three signals:
The paper proves mathematically that with typical configurations (8 GPUs/node, 1 storage NIC/node), no new bottleneck appears as long as the P/D ratio stays between 1/7 and 7/2 — covering most deployment scenarios.
4. Ablation: Where Each Cut Lands
| Component | JCT reduction | Note | |-----------|---------------|------| | Layerwise Prefill | 17.21% | Relieves HBM bottleneck, enables larger batches | | + Dual-Path Loading | 38.19% | Core gain, unlocks decode storage bandwidth | | + Scheduling | 45.62% | Dynamic scheduling fully exploits both paths |
The layered ablation is clear: layerwise is the base optimization, dual-path is the main breakthrough, and scheduling cashes it in.
5. Performance Numbers
Offline inference (RL rollout scenarios):
Online serving (Poisson arrivals, TTFT ≤ 4s, TPOT ≤ 50ms):
Key point: TTFT structure stays stable and TPOT carries no extra decode overhead. The optimization targets KV-Cache reading and queuing stages without affecting generation quality.
6. Why This Is "The End of the Prefilling Era"
Traditional LLM serving is an "online computation" paradigm — every request computes KV-Cache from scratch, making prefill a mandatory, compute-intensive stage.
Agentic workloads upend this:
DeepSeek's DualPath tells the industry: when parameter count is no longer the bottleneck, I/O is the Achilles' heel. Compute isn't lacking — compute is waiting for data.
This matches a hardware trend: from Ampere to Blackwell, GPU compute improved 28.8x, but PCIe bandwidth only 2.0x and HBM capacity 2.4x. The I/O-to-compute ratio worsened 14.4x. Each new GPU generation over-provisions compute while the pipe feeding it narrows relatively.
The industry is responding: PCIe 6.0 (256GB/s x16) and PCIe 7.0 (512GB/s x16) double bandwidth repeatedly; CXL pools DRAM; NVIDIA Dynamo/NIXL, LMCache, and SGLang HiCache all build layered caching. DualPath is the systems-architecture answer to this trend.
7. DeepSeek's "Hippocampus"
One analogy from the video: DeepSeek is building an efficient hippocampus for AGI.
The hippocampus is the brain's memory hub — it doesn't "compute" (that's the cortex's job); it converts short-term memory to long-term and retrieves on demand. DualPath treats KV-Cache not as a byproduct of computation but as a first-class citizen: storage, transport, caching, and scheduling are all designed around the KV-Cache lifecycle.
It's an upgrade in systems philosophy. The old goal: "make GPUs compute faster." The new goal: "make GPUs never wait for data."
8. Beyond the Paper: Ecosystem Validation
DualPath isn't an isolated effort. Multiple concurrent projects push the same direction:
| Project | Direction | Relation | |---------|-----------|----------| | NVIDIA Dynamo / NIXL | Data-movement abstraction layer | DualPath's RDMA transfer depends on such infrastructure | | LMCache | CPU/SSD hierarchical cache + prefetch | Where KV-Cache lives | | SGLang HiCache | GPU/CPU/Storage three-tier cache | Similar layering idea | | Mooncake | Distributed DRAM cache + affinity scheduling | Baseline discussed in the paper, limited by DRAM cost | | Dell PowerScale + vLLM + LMCache | Enterprise storage in the hot path | Public validation: 1s TTFT @ 131K context vs 17s baseline |
These projects point to one conclusion: KV-Cache storage and transport are becoming the next battleground of AI infrastructure.