Vera Rubin NVL72 Ushers AI Factories Into the Agent Era — Hot Chips 2026 Deep Dive

Forum topic · 小凯 · 2026-08-26

Summary

At Hot Chips 2026, NVIDIA presented the first full measured results for Vera Rubin NVL72, a rack-scale AI factory platform aimed squarely at agentic AI workloads. The system combines 72 Rubin GPUs with native NVFP4 inference, 11 PB of HBM4 memory and 800 PB/s aggregate bandwidth, sixth-generation NVLink (10x Ethernet speed, 3x lower latency), an 88-core Vera CPU with 1.2 TB/s LPDDR5X bandwidth, Groq 3 LPX for long-context decoding, prefill/decode disaggregation with KV-aware routing, and Spectrum-X Ethernet for scale-out. NVIDIA reported up to 30x higher throughput per watt and up to 35x lower cost per million tokens versus GB300 NVL72 — a cumulative 450x jump over H200 NVL8 — delivering 2 ZFLOPS NVFP4 inference and 1.4 ZFLOPS training at the 100MW AI factory scale. The presentation, co-hosted with SemiAnalysis's Dylan Patel, leaned on the new AgentX benchmark, which replays real agent coding sessions where median context has grown to 100K–400K tokens. Benchmarks used DeepSeek V4 Pro, an open-source 1.6T-parameter MoE model. The article contrasts this platform approach with OpenAI's inference-only Jalapeño ASIC co-developed with Broadcom.

Key points

Hot Chips 2026 unveiling: NVIDIA presented Vera Rubin NVL72's first complete measured results at Hot Chips 2026 in California, co-presented by the Vera Rubin system architect and SemiAnalysis founder Dylan Patel — a deliberate choice of third-party scrutiny over marketing. The session was moved to the closing day, ran 90 minutes, and the room was expanded from 800 to 1,400 seats.
Why old benchmarks fail in the Agent era: Classic benchmarks (MLPerf, GLUE, MMLU) test short prompt-response exchanges. Real AI coding agents now run trajectories of 200K–400K+ tokens with dozens of tool calls and sub-agent handoffs. SemiAnalysis's AgentX benchmark (2026) replays real agent coding sessions — request length distributions, context growth curves, tool-call pauses, and sub-task dependency timing. It reports median agent context growing from 1K–32K tokens (2023) to 100K–400K tokens (2026), with average full trajectories rising from ~60K to ~400K tokens (peaks beyond 1M).

The seven-layer rack stack

1. NVFP4: NVIDIA's 4-bit floating point format, natively supported by Rubin GPUs — cutting memory/bandwidth needs to 1/4 of FP16 while keeping floating-point dynamic range, letting dozens of concurrent agents hold 100K-token contexts (each with its own KV cache) on HBM4. 2. HBM4: 11 PB memory and 800 PB/s aggregate bandwidth at the 100MW AI factory scale — enough to hold roughly half of a compressed snapshot of the entire internet archive (~50 PB) in memory. 3. Sixth-gen NVLink: 72 GPUs fully interconnected in-rack, 10x faster than Ethernet with 3x lower latency, enabling KV-cache sharing between sub-agents without stalling the main agent; KV-aware routing hits already-cached nodes. 4. Vera CPU: 88 Olympus cores, LPDDR5X, 1.2 TB/s memory bandwidth — NVIDIA's first CPU explicitly designed for AI agent workloads (short tool calls, metadata-heavy I/O, sub-millisecond responses). SpaceX AI has deployed it at scale and plans to fly it in space for on-orbit inference. 5. Groq 3 LPX: In full production; delivers 3,400 tokens/s long-context output to take over decode-phase "long-tail generation" while Rubin GPUs handle scheduling. 6. Prefill/decode disaggregation + KV-aware routing: Prompt processing (compute-heavy) and generation (bandwidth/latency-bound) run as separate pools, precisely routed by KV-cache state. 7. Spectrum-X Ethernet: NVIDIA's high-speed Ethernet for interconnecting racks so a 100MW factory behaves as one compute organism.

The numbers: 30x vs 35x

30x throughput per megawatt vs GB300 NVL72 (peak) — the energy-efficiency ceiling: a 100MW factory costs ~$44M/year in electricity at $0.05/kWh, so efficiency determines whether expansion is viable.
35x lower cost per million tokens — the adoption lever: at billions of daily agent calls, fractional-cent per-token differences decide gross margins.
Generational trajectory: H200 NVL8 → GB300 NVL72 is 15x; GB300 → Vera Rubin is 30x; cumulative 450x vs H200.
Headline factory figures: 2 ZFLOPS NVFP4 inference, 1.4 ZFLOPS training, 11 PB HBM4, 800 PB/s at 100MW (roughly 250 fully loaded NVL72 racks, or the power draw of ~100,000 homes).

Jalapeño vs Vera Rubin: two routes

OpenAI's Jalapeño (co-developed with Broadcom, revealed the same week) leads GB300 by 1.5–1.9x per watt with 28–59% lower latency — but it is inference-only, with no training capability, and serves a single customer. Vera Rubin ships a full ecosystem (NVFP4, NVLink, Spectrum-X, cuDNN, TensorRT-LLM) to all hyperscalers and enterprises. The two approaches — universal GPU platform vs bespoke ASIC — are expected to coexist for years.

Why DeepSeek V4 Pro was the test load

NVIDIA benchmarked with DeepSeek V4 Pro, an open-source 1.6-trillion-parameter MoE model, because: (1) MoE routing stresses interconnect scheduling; (2) the model cannot fit on one GPU, forcing tensor + pipeline + expert parallelism that makes NVLink bandwidth the deciding factor; (3) being open-source, all results are reproducible; and (4) it is already widely used in real agent coding scenarios.

Takeaways

The decisive shift is that the compute race's battleground has moved from "how fast a GPU runs" to "how well an AI factory understands agents." The next phase will hinge on benchmarks evolving (AgentX replacing older suites), energy efficiency (Vera Rubin Ultra targeting 5–10x more throughput at equal power), and physical constraints — power, cooling, and geography — with on-orbit inference emerging as an edge-compute extension.

*All figures as reported in the source post, attributed to Hot Chips 2026 live measurements, SemiAnalysis AgentX disclosures, DeepSeek V4 Pro specifications, and public NVIDIA/Broadcom statements.*