> Beijing time 2026-08-26 09:00 · Zhichai AI · AI Compute Infrastructure column
Source: OpenAI official blog + IT Home/Sina Finance coverage, Aug 25-26
The headline
On August 25, OpenAI published the first benchmark results for Jalapeño, its first self-developed AI inference chip, co-designed with Broadcom. Tested on the SemiAnalysis InferenceX public benchmark against NVIDIA GB200/GB300 systems, using three external models — GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T — the chip delivered:
- 1.5-1.9x higher AI workload per watt
- 1.7-3.6x lower end-to-end latency
- Rated 700W / sustained draw ≤550W — real-world efficiency better than the spec sheet
- Co-design partner: Broadcom, providing ASIC physical implementation and high-speed SerDes IP; OpenAI contributing workload characteristics, optimization targets, and architecture choices.
- Hardware lead: OpenAI hardware chief Richard Ho presented the results.
- Benchmark: SemiAnalysis InferenceX, across three models of different scales and architectures.
- Deployment: OpenAI plans to start deploying Jalapeño in its own compute infrastructure by end of 2026, calling it "the first generation of a multi-year, multi-generation roadmap."
- Important caveat: OpenAI simultaneously committed to continuing broad deployment of NVIDIA and other partners' accelerators — diversification, not replacement.
- Layer 1 — hardware: NVIDIA (GB200/GB300), AMD (MI400), Broadcom (custom co-design partner), and in-house chips (Google TPU, AWS Trainium/Inferentia, Meta MTIA, Tesla Dojo).
- Layer 2 — software stacks: NVIDIA CUDA remains the de facto standard; AMD ROCm追赶; Google XLA/JAX; OpenAI Triton as a possible companion stack for Jalapeño.
- Layer 3 — models + inference products: OpenAI, Google, and xAI have closed loops; Anthropic and Meta remain partially open.
- NVIDIA: Short-term headwind as some GB200/GB300 orders shift away, but replacement is gradual; NVIDIA's CUDA moat will take years to erode.
- AMD: Head-tier vendors prefer Broadcom-style custom partners over general-purpose GPUs, forcing AMD to rethink differentiation.
- Broadcom: The quiet winner — now the custom-ASIC design partner of choice for leading AI firms (also Google's TPU partner), echoing TSMC's role for Apple silicon.
- China: The choice of Kimi K2.5 (Moonshot AI) as the 1T-parameter test case signals OpenAI's willingness to run non-OpenAI models on its inference stack — though actual commercial access for Chinese vendors depends on geopolitics.
- End of 2026: initial deployment in OpenAI's internal infrastructure
- 2027: Gen 2 "deep in development," Gen 3 "taking shape"
- 2027-2028: possible availability for API customers
- Beyond 2028: possible external inference services
Three implications:
1. OpenAI is no longer just a big NVIDIA GPU customer — it is tightening the inference supply chain around co-designed silicon. 2. The inference economics are rewritten — more users or more complex agents per watt, enabling lower API prices or higher margins. 3. The competition axis has shifted from "who has more GPUs" to "who has custom silicon + models + inference software stack" — the Google TPU playbook spreading across the AI leaders.
Notably, OpenAI benchmarked external open-source and third-party models rather than its own internal ones, preempting accusations of cherry-picked results.
What Jalapeño is
Jalapeño is OpenAI's first officially named inference chip — not a standalone GPU or ASIC, but a full-stack co-designed inference system positioned to serve real-time inference for ChatGPT, Codex, and other OpenAI products.
Key facts:
The numbers
| Model | Compared GPU | AI workload per watt | End-to-end latency | |---|---|---|---| | GPT-OSS 120B | GB200 1200W | 1.9x | 1.7x lower | | DeepSeek R1 670B | GB300 1400W | 1.7x | 3.6x lower | | Kimi K2.5 1T | GB300 1400W | 1.5x | 3.4x lower |
On highly interactive (agentic) workloads, Jalapeño shows 2.1-4.1x higher performance than the comparison systems — significant because multi-turn agent calls compound latency dozens of times.
Key architectural choices
1. KV cache locality
The KV cache grows rapidly in multi-turn conversations and agent loops. In traditional GPU designs it must be replicated across GPUs, wasting latency and power on communication. Jalapeño keeps it explicitly local. Per OpenAI:
> "model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase."
2. Prefill/decode co-optimization
Prefill is compute-bound; decode is memory-bandwidth-bound and often leaves GPU compute idle waiting on HBM. Jalapeño matches compute to memory bandwidth so both phases run at high utilization:
> "excel at both prefill and decode, and adapt as the balance between them changes, a defining feature of agentic workloads."
A similar philosophy appears in TPU v5e/v6 — a core advantage of workload-specific accelerators over general-purpose GPUs.
The three-layer structure of AI compute
Jalapeño upgrades OpenAI from a two-layer to a three-layer closed loop — its biggest strategic change.
Market impact
Timeline
Bottom line
Jalapeño is not an NVIDIA replacement but a rewrite of the AI compute economy: the first verifiable, third-party-benchmarked lead by a non-NVIDIA chip over NVIDIA hardware since the ChatGPT era began — not a 10x leap, but enough to restructure how the market prices compute moats.
References
1. OpenAI blog: Jalapeño's first results — https://openai.com/index/jalapeno-first-results/ 2. IT Home (Aug 26): OpenAI's first custom inference chip benchmark results 3. Sina Finance / IT Home: Jalapeño vs GB300 coverage 4. SemiAnalysis InferenceX benchmark methodology and model list 5. OpenAI CFO Sarah Friar: The full stack behind abundant intelligence — https://openai.com/index/the-full-stack-behind-abundant-intelligence/