DeepSeek DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
> TL;DR: In multi-turn agentic inference, KV-Cache read speed becomes the system's bottleneck. DeepSeek's DualPath puts the idle storage bandwidth of decode engines to work, allowing KV-Cache to reach prefill via two routes—direct from storage, or relayed through decode engines. The system picks whichever is faster. Offline inference improves up to 1.87x; online serving up to 1.96x.
1. The Problem: Compute Isn't the Bottleneck, IO Is
Agent workloads share a signature: long contexts, short appends, high KV-Cache reuse.
In DeepSeek's production trace, a typical agent session has 157 turns, an average context of 32,721 tokens, and only 429 new tokens appended per turn. KV-Cache hit rate: 98.7%. That means each turn, 98.7% of attention computation is "old work"—no recomputation needed, just reading back stored KV-Cache.
Compute pressure is small. The real pressure is storage bandwidth.
Modern inference clusters use PD disaggregation (prefill-decode separation):
- Prefill engine: processes input prompts, computes KV-Cache
- Decode engine: generates output token by token
- Long prefill storage-NIC queues → DE path
- Heavy decode compute → PE path
- Both idle → load balance across both
- All GPU traffic uses GPUDirect RDMA (no CPU copies)
- QoS isolation: model communication gets high priority with 99% reserved bandwidth; KV-Cache traffic is low priority, scavenging idle bandwidth
- InfiniBand uses Virtual Lanes; RoCE uses Traffic Class + DSCP marking
- Storage NIC queue length
- GPU compute load
- Request workload characteristics (context length, append size)
- DS 660B: up to 1.87x (64K context, 1024 agents)
- DS 27B: up to 1.78x, 1.64x average, 2.46x in extreme configs
- Qwen 32B: consistent trend
- DS 27B: 1.67x throughput
- DS 660B: 2.25x throughput
- Average: 1.96x
- 98.7% of tokens are "old"—they need reading, not computing
- Prefill is no longer "compute" but "read"
- Performance is decided by GB/s, not FLOPS
- Paper: DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
- arXiv: 2602.21548
- Authors: Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, Panpan Huang
- Institutions: Peking University, Tsinghua University, DeepSeek-AI
- Date: 2026-02-25
- Core code: ~5,000 lines on an in-house inference framework, built on FlashMLA, DeepGEMM, DeepEP, 3FS
- Models tested: DeepSeek 660B (MoE + sparse attention), DeepSeek 27B, Qwen 32B
- Configs: 2P4D (DS 660B), 1P2D (Qwen 32B), 1P1D (DS 27B)
- SLO: TTFT ≤ 4s, TPOT ≤ 50ms
- Key numbers: 1.87x offline, 1.96x online, 98.7% KV-Cache hit rate, no-bottleneck P/D range 1/7–7/2
Each agent turn, the prefill engine must load the previous turn's KV-Cache from external storage (SSD/3FS). Meanwhile, decode engines' storage NICs sit mostly idle—they only write back small amounts of new KV-Cache.
Result: prefill nodes' storage NICs saturate and GPUs wait idle, while decode nodes' storage NICs and GPUs both loaf.
A clean experiment proves this: regardless of P/D ratio (1P1D, 2P1D, 1P2D), as long as total storage bandwidth stays constant, performance stays constant. Performance is locked by storage bandwidth, not GPU compute.
2. DualPath: Two Paths into Prefill
Core insight: KV-Cache doesn't have to flow from storage directly into prefill. It can first traverse a decode engine's storage NIC, then reach prefill via RDMA over the compute network.
| Path | Flow | Bandwidth consumed | |------|------|--------------------| | PE Read Path | Storage → prefill engine → compute | prefill storage NIC | | DE Read Path | Storage → decode engine → RDMA → prefill engine | decode storage NIC + compute network |
A global scheduler routes dynamically based on live load:
The cluster's storage NIC bandwidth is thus pooled: prefill and decode storage bandwidth jointly serve KV-Cache loading.
3. Three Engineering Challenges
1. Fine-Grained Transfer
Layerwise prefill (loading one KV-Cache layer into GPU HBM at a time) shatters KV-Cache into many small chunks. Transfers must have minimal overhead and overlap with compute.DualPath's approach: while layer N computes, layer N+1's KV-Cache is DMA'd into a DRAM buffer in the background. No waiting on storage.
2. Traffic Isolation
The DE path injects KV-Cache traffic into the compute network, which also carries latency-sensitive collectives—AllToAll (MoE expert parallelism), ReduceScatter/AllGather (tensor/context parallelism). KV-Cache traffic must not steal their bandwidth.DualPath's CNIC-centric traffic management:
3. Dynamic Load Balancing
Routing can't be blind—if all requests take the DE path, decode engines get crushed. The scheduler monitors:Mathematical modeling shows: with typical configs (8 GPUs/node, 1 storage NIC/node), no new bottleneck appears as long as the P/D ratio stays between 1/7 and 7/2—covering most deployments.
4. Ablation: Where Each Gain Comes From
| Component | JCT reduction | Notes | |-----------|--------------|-------| | Layerwise Prefill | 17.21% | Relieves HBM bottleneck, enables larger batches | | + Dual-Path Loading | 38.19% | Core gain, unlocks decode storage bandwidth | | + Scheduling | 45.62% | Dynamic scheduling fully utilizes both paths |
Layerwise is the base optimization, dual-path is the main breakthrough, scheduling monetizes it.
5. Performance Numbers
Offline inference (RL rollout scenario):
Online serving (Poisson arrivals, TTFT ≤ 4s, TPOT ≤ 50ms):
Key point: TTFT structure is stable and TPOT carries no extra decoding overhead. Optimization targets KV-Cache reading and queuing only, without affecting generation quality.
6. Why This Is "The End of the Pre-filling Era"
Traditional LLM serving is "online computation"—every request computes KV-Cache from scratch, making prefill compute-bound.
Agentic workloads break this paradigm:
Hardware trends confirm it: from Ampere to Blackwell, GPU compute improved 28.8x, but PCIe bandwidth only 2.0x and HBM capacity 2.4x—a 14.4x deterioration of IO-to-compute ratio. The industry is responding: PCIe 6.0 (256GB/s x16) and PCIe 7.0 (512GB/s x16), CXL memory pooling, and layered caching efforts like NVIDIA Dynamo/NIXL, LMCache, and SGLang HiCache. DualPath is the systems-architecture answer to this trend.
7. DeepSeek's "Hippocampus"
One metaphor from the source: DeepSeek is building an efficient hippocampus for AGI. The hippocampus handles memory—not computation—converting short-term memory to long-term and retrieving on demand. DualPath treats KV-Cache as a first-class citizen, with storage, transfer, caching, and scheduling all designed around its lifecycle. The optimization goal has shifted from "make GPUs compute faster" to "make GPUs never wait for data."
8. Ecosystem Context
DualPath is not alone; multiple concurrent projects target the same direction:
| Project | Direction | Relationship | |---------|-----------|--------------| | NVIDIA Dynamo / NIXL | Data movement abstraction | DualPath's RDMA transfers rely on this type of infrastructure | | LMCache | CPU/SSD tiered cache + prefetch | Where KV-Cache lives | | SGLang HiCache | GPU/CPU/Storage three-tier cache | Similar tiering approach | | Mooncake | Distributed DRAM cache + affinity scheduling | Comparison baseline in the paper, limited by DRAM cost | | Dell PowerScale + vLLM + LMCache | Enterprise storage in the hot path | 1s TTFT @ 131K context vs 17s baseline |
KV-Cache storage and transfer is becoming the next battleground of AI infrastructure.