DualPath: Nearly 2x AI Inference Cluster Throughput via Dual-Path KV-Cache Loading

Forum topic · QianXun · 2026-09-01

Summary

A 2026 paper from Peking University, Tsinghua University, and DeepSeek-AI, DualPath argues that storage I/O—not compute—is the dominant bottleneck in modern agentic LLM inference. In agent workloads with ~157 turns and 32.7K-token contexts, KV-Cache hit rates exceed 98.7%, making load speed the limiting factor. Under standard prefill-decode disaggregation, all KV-Cache loads flow through the Prefill engine's storage NIC while the Decode engine's sits nearly idle. DualPath introduces a second path: KV-Cache can be loaded via the Decode engine's storage NIC and relayed over idle compute-network RDMA bandwidth, roughly doubling effective storage bandwidth without new hardware. CNIC-centric traffic management uses InfiniBand virtual lanes to prioritize latency-sensitive AllToAll traffic, and an adaptive two-level scheduler with compute quotas minimizes GPU idle bubbles. On a DeepSeek cluster, offline throughput rose up to 1.87x (DeepSeek-V3.2 660B), online serving throughput ~1.96x without SLO violations, average job completion time dropped 45.62%, and 1,152 GPUs scaled near-linearly (job time grew only 1% as agents scaled 24x).

I/O Is the New Compute: How DualPath Nearly Doubles AI Inference Cluster Throughput

*Deep-dive on DualPath (Wu et al., 2026, arXiv:2602.21548) — a joint paper by Peking University, Tsinghua University, and DeepSeek-AI.*

Key Numbers

| Metric | Result | |---|---| | Offline inference throughput (DS 660B) | 1.87x | | Online serving throughput (no SLO breach) | 1.96x avg | | KV-Cache hit rate (agent workloads) | 98.7% | | Cluster scale | 1,152 GPUs, near-linear scaling | | Average JCT reduction | 45.62% |

1. The Real Bottleneck: Storage I/O

Modern agentic LLM workloads average 157 turns per task with 32.7K-token contexts, but each turn adds only ~429 new tokens. With KV-Cache hit rates at 98.7%+, the bottleneck shifts from "how much to compute" to "how fast to load."

The paper quantifies this with a Cache-Compute Ratio (GB of KV-Cache loaded per PFLOP of compute):

| Model | Architecture | Ratio | |---|---|---| | Qwen2.5-32B | FP16 dense GQA | 117–267 GB/PFLOP | | GPT-OSS-120B | MoE, 128 experts | 47–95 GB/PFLOP | | Qwen3-235B-A22B | MoE, 22B active | 39–60 GB/PFLOP | | DeepSeek-V3.2 660B | MoE + MLA | 13–36 GB/PFLOP |

Meanwhile, GPU I/O-Compute ratios have degraded 14.4x from Ampere to Blackwell.

2. The Hidden Flaw in PD Disaggregation

Typical AI datacenters physically separate compute NICs (CNIC) from storage NICs (SNIC): 8x 400Gbps CNICs but only 1x 400Gbps SNIC per node. Under prefill-decode (PD) disaggregation, all KV-Cache loading goes through the Prefill engine's SNIC, while the Decode engine's SNIC sits near 0% utilization.

3. DualPath: The Detour Is Faster

Core insight: KV-Cache loading does not have to be Prefill-centric.

Path A (traditional): Storage → Prefill engine SNIC
Path B (new): Storage → Decode engine SNIC → RDMA over compute network → Prefill HBM

This works because: 1. The idle DE-side SNIC becomes usable — total storage bandwidth goes from 1x to 2x. 2. Compute networks are heavily underutilized (sub-5% instantaneous use; inference AllToAll is bursty and sub-millisecond). 3. Path choice can be dynamic, based on SNIC queue length, GPU load, and HBM headroom.

The paper derives a bottleneck-free operating region: with g=8 GPUs and s=1 SNIC per node, P/D ratios from 1/7 to 7/2 introduce no new bottleneck — no hardware changes needed. RDMA submit (~1μs) is also faster than CUDA Copy (5–7μs).

4. CNIC-Centric Traffic Management

All GPU traffic (including local H2D/D2H) is routed through the GPU-paired CNIC via GPUDirect RDMA, letting InfiniBand/RoCE QoS arbitrate: latency-sensitive AllToAll/ReduceScatter/AllGather gets a high-priority virtual lane (~99% bandwidth); KV-Cache transfers use a low-priority lane consuming only idle capacity.

5. Adaptive Two-Level Scheduler

PE side — reverse priority: engines with short disk queues get highest priority (SNIC about to go idle).
DE side: cross-group balancing by minimum token count; in-group selection via token threshold Z=1.05×avg and shortest sequence.
Intra-engine Compute Quota: requests packed FIFO until estimated attention time reaches a 300ms quota; binary search on batch size when exceeded. Result: Max/Avg attention execution time of 1.06 — near-zero inter-GPU bubbles.

6. Evaluation

Offline batch inference (vs. baseline):

| Model | Speedup | |---|---| | DeepSeek-V3.2 660B | 1.87x (near I/O-free ceiling) | | DeepSeek-V3.2 27B | 1.78x | | Qwen2.5-32B | 1.62x |

DualPath 1P1D matches baseline 2P1D throughput, confirming storage bandwidth as the dominant bottleneck. Average speedup across P/D configs: 1.64x (max 2.46x).

Online serving (TTFT ≤ 4s, TPOT ≤ 50ms): APS capacity +1.67x (DS 27B) and +2.25x (DS 660B); TTFT queuing delay stays flat; no added decode overhead.

Ablation (cumulative JCT reduction): Layerwise prefill −17.21% → + dual-path loading −38.19% → + adaptive scheduling −45.62%.

Scaling: 48P96D (1,152 GPUs) serving 49,152 agents completed in 3,201s vs. 3,167s for 2P4D with 2,048 agents — 24x the agents for ~1% more JCT. Scheduler CPU usage < 10 cores.

7. Engineering Takeaways

1. Software-defined I/O pooling beats adding hardware — 8x 400Gbps CNICs running under 5% utilization are purchased-but-unused capacity. 2. VL/QoS is the next-generation scheduler — physical network isolation plus logical InfiniBand/RoCE channels. 3. Single-component optimization is exhausted; end-to-end system co-design is the new frontier.

> "N idle network interfaces hide N undiscovered pools of compute capacity."

Reference: Wu et al. (2026). *DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference.* arXiv:2602.21548. Peking University, Tsinghua University, DeepSeek-AI.

Related work: Mooncake, 3FS (DeepSeek file system), SGLang HiCache.