I/O Is the New Compute: How DualPath Nearly Doubles AI Inference Cluster Throughput

> A 2026 paper by Peking University, Tsinghua University, and DeepSeek-AI acknowledges a counterintuitive truth: stacking more GPUs is no longer the answer — taking the longer route can actually be faster.

| Metric | Value | |---|---| | Offline inference throughput | 1.87x (DeepSeek 660B vs baseline) | | Online serving throughput | 1.96x, without breaking SLOs | | KV-Cache hit rate | 98.7% (multi-turn agent dialogues) | | Cluster scale | 1,152 GPUs, near-linear scaling at 48P96D |

The Core Question

Why do AI data centers with tens of thousands of GPUs — bought for billions — run at only ~40% utilization year-round? The answer is not compute, network, or model capacity. It is storage I/O. DualPath's essence is to acknowledge this reality and decompose the bottleneck with the engineering philosophy that "a detour can be faster."

From One-Shot Q&A to Agentic Workloads

Legacy LLM serving answered questions in one pass. Today's agents plan, retrieve, write code, and reflect — on average 157 turns, with ~32.7K token contexts but only ~429 new tokens per turn.

The critical consequence: with KV-Cache hit rates at 98.7%, 99 of every 100 loaded tokens need no recomputation. The bottleneck has shifted from "how much to compute" to "how fast to load."

The paper quantifies this as the Cache-Compute Ratio (GB of KV-Cache read per PFLOP of compute):

| Model | Architecture | Cache-Compute Ratio | |---|---|---| | Qwen2.5-32B | FP16 dense GQA | 117–267 GB/PFLOP | | GPT-OSS-120B | MoE, 128 experts | 47–95 GB/PFLOP | | Qwen3-235B-A22B | MoE, 22B active | 39–60 GB/PFLOP | | DeepSeek-V3.2 660B | MoE + MLA | 13–36 GB/PFLOP |

Even MLA-optimized architectures require substantial reads. Meanwhile, from Ampere to Blackwell, NVIDIA's I/O-Compute Ratio has worsened by 14.4x — compute is flying while I/O crawls.

The Hidden Flaw of PD Disaggregation: Unequal Resources

Modern AI data centers physically separate the compute network (CNIC, typically 8x 400Gbps per node) from the storage network (SNIC, typically 1x 400Gbps). Under PD disaggregation, all KV-Cache loading goes through the prefill engine's SNIC, while the decode engine's SNIC sits nearly idle at ~0% utilization.

Existing fixes fall short: Mooncake-style distributed DRAM pools are expensive; KV-Cache compression doesn't address the root cause. The real problem is an unresolved single-point I/O bottleneck.

DualPath: The Detour Is Faster

DualPath's core insight: *KV-Cache loading does not have to be prefill-centric.*

• Path A (traditional): Storage → Prefill Engine's SNIC → H2D → compute. Bottleneck locked at the PE-side SNIC.
• Path B (new): Storage → Decode Engine's SNIC → RDMA across the compute network → Prefill Engine's HBM.

The detour exploits three underused resources:

1. Idle DE-side SNIC — cluster storage bandwidth effectively goes from 1x to 2x. 2. Idle compute network — 8x 400Gbps CNICs per node (3.2 Tbps), yet inference communication is sub-millisecond bursts using under 5% instantaneous bandwidth. 3. Dynamic path selection — a scheduler monitors SNIC queue depth, GPU load, and HBM headroom to route each data block.

A bottleneck-free region is proven mathematically: with g=8 NICs, s=1 SNIC, M≈500 GB/s, B_S≈50 GB/s, any P/D ratio between 1/7 and 7/2 introduces no new bottleneck — with zero hardware upgrades.

CNIC-Centric Traffic Management

Won't RDMA traffic hurt inference communication? Inference collectives (AllToAll, ReduceScatter, AllGather) are latency-sensitive sub-millisecond bursts where software shaping fails — hardware QoS is required.

Rule: All traffic to/from GPUs (including local H2D/D2H) goes through the GPU-paired CNIC via GPUDirect RDMA, converging on the compute network where InfiniBand Virtual Lanes provide priority arbitration — roughly 99% bandwidth to inference traffic, remainder to KV-Cache transfers via weighted round-robin.

A bonus: RDMA submit costs ~1μs vs 5–7μs for CUDA Copy, plus doorbell batching amortizes overhead. This is a shift from "shortest path" to "optimal scheduling."

Adaptive Two-Level Scheduling

• Engine-level (PE side): inverted priority — engines with *short disk queues* get highest priority (their SNIC is about to idle), overloaded engines get none.
• Engine-level (DE side): two-level balancing — minimum token totals across groups; within groups, a high-token threshold (Z = 1.05x avg) plus minimum sequence selection; the side with the shorter read queue becomes the load path.
• Intra-engine: a Compute Quota mechanism estimates attention layer execution time per request (cached, bsz), accumulates FIFO up to a 300ms quota, and bisects batch size when exceeded. Result: attention-layer Max/Avg time ratio of 1.06 — inter-GPU wait bubbles near zero. On a 1,024-GPU cluster, eliminating 1% bubbles equals 10 GPUs of compute.

Evaluation

Offline batch inference (throughput vs baseline):

| Model | Speedup | vs Oracle | |---|---|---| | DeepSeek-V3.2 660B | 1.87x | near zero-I/O ceiling | | DeepSeek-V3.2 27B | 1.78x | near ceiling | | Qwen2.5-32B | 1.62x | near ceiling |

A P/D ratio experiment (DS 27B) showed DualPath 1P1D ≈ Basic 2P1D — indirect proof that storage bandwidth is the dominant bottleneck. Average speedup across configurations: 1.64x (up to 2.46x).

Online serving (TTFT ≤ 4s, TPOT ≤ 50ms): capacity up 1.67x (DS 27B) and 2.25x (DS 660B); TTFT queuing delay stays flat under load; no extra decode overhead.

Ablation (cumulative JCT reduction):

• Layerwise prefill: −17.21%
• + Dual-path loading: −38.19%
• + Adaptive scheduling: −45.62%

Scale: from 2P4D (2,048 agents, JCT 3,167s) to 48P96D (49,152 agents, JCT 3,201s) — a 24x increase in agents with only ~1% JCT growth. Online load scaled 22x (0.4 → 8.8 APS) with unchanged latency. Scheduler CPU usage: under 10 cores.

Engineering Lessons for the Agentic Era

1. Software-defined I/O pooling beats stacking hardware. Eight 400Gbps CNICs idle 95% of the time represent capacity already paid for. Hardware scaling hits physical ceilings (light speed, copper, transistor density); intelligent scheduling does not. 2. VL/QoS is the next-generation scheduler. Physical isolation is the baseline; logical channels — InfiniBand VLs, RoCE PFC, Ultra Ethernet QoS — unlock it. 3. Single-component optimization is exhausted; system co-design is the frontier. Layerwise prefill + dual-path loading + adaptive scheduling combine for 1+1+1 > 3 synergy. The next bottleneck in LLM infrastructure is end-to-end coordination, not single-point breakthroughs.

> "N idle network interfaces hide N undiscovered pools of compute capacity."

Not compute-stacking, but data scheduling. Not the shortest path, but the highest throughput. Not single-point optimization, but end-to-end coordination.

---

*Reference: Wu et al. (2026). DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference. arXiv:2602.21548. Institutions: School of Computer Science, Peking University; Tsinghua University; DeepSeek-AI.*

*Related reading: Mooncake (distributed KV-Cache storage), 3FS (DeepSeek's file system), SGLang HiCache.*