I/O Is the New Compute: How DualPath Nearly Doubles AI Inference Cluster Throughput

Forum topic · QianXun · 2026-09-01

Summary

DualPath is a joint paper by Peking University, Tsinghua University, and DeepSeek-AI that tackles the storage I/O bottleneck in agentic LLM inference. The authors observe that agent workloads average 157 turns with ~32.7K-token contexts but only ~429 new tokens per turn, so KV-Cache hit rates exceed 98.7% and performance is limited by how fast cached data can be loaded, not how fast GPUs compute. DualPath's core idea is a dual-path KV-Cache loading scheme: instead of routing all loads through the Prefill engine's saturated storage NIC, data can detour through the Decode engine's idle storage NIC and travel over RDMA on the underutilized compute network. A CNIC-centric traffic management layer uses InfiniBand virtual lanes to protect latency-sensitive collective communication, while an adaptive two-level scheduler balances engines and eliminates GPU idle bubbles. On a DeepSeek cluster, DualPath raises offline inference throughput up to 1.87x (DeepSeek-V3.2 660B) and online serving throughput up to 1.96x without violating SLOs (TTFT <= 4s, TPOT <= 50ms), reduces average job completion time by 45.62%, and scales near-linearly to 1,152 GPUs (48P96D) across 49K agents. The paper argues that software-defined I/O pooling and smarter scheduling, not more hardware, are the path to higher AI datacenter efficiency.

DualPath is a recent paper from Peking University, Tsinghua University, and DeepSeek-AI (arXiv:2602.21548) arguing that storage I/O, not compute, is the real bottleneck of agentic LLM inference — and showing that a clever "detour" can nearly double cluster throughput without any hardware upgrades.

The problem: agentic workloads are I/O-bound

Agent workloads average 157 turns per task with ~32.7K-token contexts, but only ~429 new tokens per turn.
This yields KV-Cache hit rates of 98.7%+: for every 100 tokens loaded, ~99 were already computed and just sit in storage waiting to be read.
The paper quantifies the pressure with a Cache-Compute Ratio (GB of KV-Cache per PFLOP of compute):

| Model | Architecture | Cache-Compute Ratio | |---|---|---| | Qwen2.5-32B | FP16 dense GQA | 117–267 GB/PFLOP | | GPT-OSS-120B | MoE, 128 experts | 47–95 GB/PFLOP | | Qwen3-235B-A22B | MoE, 22B active | 39–60 GB/PFLOP | | DeepSeek-V3.2 660B | MoE + MLA | 13–36 GB/PFLOP |

Meanwhile, from Ampere to Blackwell, NVIDIA GPUs' I/O-Compute Ratio worsened by 14.4x — compute flies while I/O crawls.

The hidden flaw in PD disaggregation

Modern AI datacenters physically isolate the compute network (CNIC) from the storage network (SNIC). Under Prefill/Decode disaggregation, all KV-Cache loads go through the Prefill engine's SNIC — saturating it — while the Decode engine's SNIC sits nearly 0% utilized.

DualPath: the detour is faster

DualPath adds a second loading path:

Path A (traditional): Storage → Prefill engine SNIC
Path B (new): Storage → Decode engine SNIC → RDMA over the compute network → Prefill engine HBM

This works because:

1. The Decode engine's idle SNIC becomes usable, doubling effective cluster storage bandwidth (1x → 2x). 2. Compute networks are massively underutilized — 8x 400Gbps CNICs per node (~3.2 Tbps) with inference bursts using under 5% instantaneously; RDMA submit (~1µs) is even faster than CUDA Copy (5–7µs). 3. Path selection is dynamic: the scheduler monitors SNIC queue lengths, GPU load, and HBM headroom per request.

The paper derives a bottleneck-free region: with g=8 NICs, s=1 SNIC, M≈500 GB/s, B_S≈50 GB/s, any P/D ratio between 1/7 and 7/2 introduces no new bottleneck.

CNIC-centric traffic management

All GPU-bound traffic (including local H2D/D2H) is routed through the GPU's paired CNIC via GPUDirect RDMA, so everything traverses the compute network where InfiniBand Virtual Lanes provide hardware QoS: a high-priority VL for latency-sensitive AllToAll/ReduceScatter/AllGather, a low-priority VL for KV-Cache bulk transfer (≈99/1 weighted scheduling).

Adaptive two-level scheduling

PE side (reverse priority): engines with *shorter disk queues* get highest priority — a short queue means the SNIC is about to idle.
DE side: cross-group balancing by minimum token count; in-group selection by token threshold (Z=1.05×avg) and minimum sequence length; the side with the shorter read queue becomes the load path.
In-engine Compute Quota: requests are packed into a 300ms attention-time quota per batch, with binary search on batch size and chunked prefill for leftovers. Result: attention-layer Max/Avg execution time = 1.06, near-zero GPU sync bubbles.

Evaluation results

Offline batch inference (throughput vs. baseline):

| Model | Speedup | |---|---| | DeepSeek-V3.2 660B | 1.87x | | DeepSeek-V3.2 27B | 1.78x | | Qwen2.5-32B | 1.62x |

A P/D-ratio experiment shows DualPath 1P1D ≈ Basic 2P1D — proving storage bandwidth is the dominant bottleneck. Average speedup across configurations: 1.64x (max 2.46x).

Online serving (TTFT ≤ 4s, TPOT ≤ 50ms SLOs): accepted-per-second capacity up 1.67x (DS 27B) and 2.25x (DS 660B), with TTFT queuing delay stable instead of exploding.

Ablation (cumulative JCT reduction):

| Component | Cumulative JCT reduction | |---|---| | + Layerwise Prefill | −17.21% | | + Dual-Path Loading | −38.19% | | + Adaptive Scheduling | −45.62% |

Near-linear scaling to 1,152 GPUs (48P96D): JCT grows from 3,167s at 2,048 agents (2P4D) to just 3,201s at 49,152 agents — a 24x workload increase with only ~1% JCT growth. Scheduler CPU usage stays under 10 cores.

Engineering takeaways

1. Software-defined I/O pooling beats adding hardware. The 8x 400Gbps CNICs per node are already-paid-for capacity sitting unused; the bottleneck is allocation, not resources. 2. VL/QoS is the next-generation scheduler. Physical network isolation remains, but InfiniBand VLs, RoCE PFC, and Ultra Ethernet QoS enable logical channels on top of it. 3. Single-component optimization is exhausted; end-to-end coordination is the frontier. The 1+1+1 > 3 synergy of layerwise prefill, dual-path loading, and adaptive scheduling shows the next bottleneck is system-level, not point-solution.

Paper: Wu et al. (2026). *DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference.* arXiv:2602.21548. Affiliations: School of Computer Science, Peking University; Tsinghua University; DeepSeek-AI.

Related reading: Mooncake (distributed KV-Cache storage), 3FS (DeepSeek's file system), SGLang HiCache.