Vera Rubin NVL72: NVIDIA Ushers AI Factories Into the Agent Era

> One-line warning: when a single rack's throughput makes the previous flagship rack look like an abacus, the compute race is no longer "who is stronger" — it's "who first understands what Agents are doing."

Hot Chips: From Slide-Deck Launches to a Rack Weighing Machine

Hot Chips, running since 1989, exists for one thing: making chip designers expose their lowest-level design details to peers. Attendees are PhDs who will catch you on memory bandwidth, NVLink topology, or power curves on the spot. NVIDIA chose this venue — not a consumer keynote — for the first full measured presentation of Vera Rubin NVL72, co-presented by the system architect alongside SemiAnalysis founder Dylan Patel, with every number open to scrutiny. The session was scheduled as the 90-minute finale of Hot Chips 2026, with the room expanded from 800 to 1,400 seats and still standing-room only.

Why Old Benchmarks Failed the Agent Era

Legacy AI benchmarks (MLPerf, GLUE, MMLU) were designed for one-shot Q&A. Real 2026 production workloads look very different. A coding agent in a single session may:

1. Read 200 lines of code context 2. Call grep to find definitions 3. Spawn sub-agents reading git commits 4. Recombine conclusions, then survive 5 follow-up rounds 5. Produce trajectories exceeding 300K tokens with 12 tool calls and 6 sub-task returns

SemiAnalysis's AgentX benchmark replays real agent coding sessions — request length distributions, context growth curves, tool-call pauses, and sub-task dependency timing. Its core finding: median agent context lengths grew from 1K–32K tokens (2023) to 100K–400K tokens (2026), with single trajectories averaging ~400K tokens and peaking above 1M.

Vera Rubin NVL72: A Rack-Scale AI Factory

Vera Rubin NVL72 is not a GPU — it's a complete rack-level AI factory: 72 Rubin GPUs, networking, and CPU in one standard 19-inch rack, built around one goal: letting hundreds of users simultaneously get a 160 tokens/second agent experience.

Layer 1 — NVFP4: NVIDIA's 4-bit floating-point format. Compared to FP16, model size and bandwidth drop to 1/4; compared to INT4, it retains floating-point dynamic range. This lets 100K-token contexts coexist with dozens of concurrent agents' KV caches in HBM4.

Layer 2 — HBM4: 11 PB of memory with 800 PB/s aggregate bandwidth at the 100MW AI factory scale. For reference, a compressed snapshot of the entire internet archive is roughly 50 PB.

Layer 3 — Sixth-gen NVLink: All-to-all interconnect for the 72 GPUs — 10x faster than Ethernet with 3x lower latency. This enables KV-cache-aware routing so sub-agents' KV cache transfers never block the main agent.

Layer 4 — Vera CPU: 88 Olympus cores, LPDDR5X, 1.2 TB/s memory bandwidth — NVIDIA's first CPU explicitly designed for AI agent workloads: bursts of short tool calls, massive metadata reads/writes, and sub-millisecond responses to keep pace with GPU token output. SpaceX AI has reportedly deployed Vera CPU at scale and plans to send it to orbit for real-time inference on satellite clusters.

Layer 5 — Groq 3 LPX: The LPU-derived accelerator, in full mass production, delivers 3,400 tokens/s on long-context output, taking over long-tail decode generation while the main model handles KV cache scheduling.

Layer 6 — Prefill/decode separation + KV-aware routing: Prefill (batch compute on Rubin GPUs) and decode (bandwidth/latency on Groq 3 LPX) run in separate compute pools, routed precisely by KV cache state.

Layer 7 — Spectrum-X Ethernet: High-speed Ethernet binding multiple racks into one 100MW factory-scale compute organism.

30x vs 35x: Which Number Matters More?

NVIDIA's headline figures:

• Throughput per megawatt: up to 30x vs GB300 NVL72
• Cost per million tokens: down up to 35x

The 30x is an energy-efficiency story — electricity is the largest operating cost of an AI factory (~$44M/year for a 100MW facility at $0.05/kWh). The 35x is a unit-economics story — at billions of daily agent calls, per-token cost differences decide gross margins. Both together mean NVIDIA redefined the unit economics of agent inference. Historical context: GB300 NVL72 was 15x over H200 NVL8; Vera Rubin is 30x over GB300 — a 450x leap over three generations, an exponential discontinuity rather than a smooth curve.

Same-Week Rivalry: OpenAI Jalapeño vs Vera Rubin

OpenAI's Jalapeño, co-developed with Broadcom, is a custom inference ASIC. Its numbers are impressive: 1.5–1.9x better performance per watt than GB300 and 28–59% lower latency. But it is inference-only, with no training capability, and serves a single customer. Vera Rubin NVL72 delivers 2 ZFLOPS inference plus 1.4 ZFLOPS training on one hardware stack, sold with a full ecosystem (NVFP4, NVLink, Spectrum-X, cuDNN, TensorRT-LLM) to all hyperscalers. OpenAI is betting "I don't need the world to run my hardware"; NVIDIA is betting "every AI factory should look like this." Both routes will likely coexist for years.

Why DeepSeek V4 Pro Was the Best Stress Test

NVIDIA benchmarked with DeepSeek V4 Pro — an open-source 1.6-trillion-parameter MoE model — rather than Llama, Qwen, or Gemini, because: (1) MoE routing stresses interconnect scheduling; (2) 1.6T parameters can't fit on one card, forcing tensor + pipeline + expert parallelism, where NVLink 6's 10x-over-Ethernet bandwidth matters; (3) being open-source, all numbers are reproducible — no cherry-tuning suspicions; and (4) it's already widely used in agent coding, making AgentX runs reflect real customer workloads.

The 100MW AI Factory as a New Unit

A 100MW AI factory is a compute unit of measurement — equal to the power of 100,000 households, or a Nimitz-class carrier's catapults, or 250 NVL72 racks at full load. At this scale, Vera Rubin delivers: 2 ZFLOPS NVFP4 inference, 1.4 ZFLOPS training, 11 PB HBM4, 800 PB/s aggregate bandwidth. A company's core asset may soon be how many 100MW factories it can operate — power, land, cooling, and networking are becoming the scarce resources of the AI age.

Boundary Conditions and What to Watch

• Hardware co-design: The rack stack (NVLink + Groq 3 LPX + Vera CPU + Spectrum-X) is one machine's four organs; any one missing causes visible agent stalls.
• Benchmarks: AgentX may replace old benchmarks faster than hardware refreshes; winners will be vendors who keep up with benchmark evolution.
• Energy efficiency: 30x per megawatt isn't the endpoint — a next-gen Vera Rubin Ultra must find 5–10x more throughput at the same power budget.
• Ecosystem: General-purpose GPU factories and vertical custom ASICs will run in parallel for the next five years.
• Geopolitics: With Vera CPU already doing inference in orbit, the compute race's boundary is no longer the data center but the physical world.

Epilogue

In the week around August 25, 2026, chip people, model people, and agent people were all discussing the same engineering question: can one rack give 100 users a simultaneous 160 tokens/s agent experience? Vera Rubin NVL72 answers bluntly: yes — 30x per megawatt, 35x per token, 2 ZFLOPS per AI factory. The real takeaway: the compute race's main battlefield has shifted from "how fast the GPU runs" to "how well the AI factory understands Agents."

*Based on public measured data from Hot Chips 2026, SemiAnalysis AgentX disclosures, DeepSeek V4 Pro official specifications, and public statements from NVIDIA and Broadcom. All key figures are traceable to original sources.*