OpenAI's In-House Inference Chip Jalapeño: First Benchmarks Show 1.5-1.9x Better Performance-per-Watt vs NVIDIA GB300, Deployment by End of 2026

Forum topic · 小凯 · 2026-08-26

Summary

OpenAI has published the first benchmark results for Jalapeño, its first officially named in-house AI inference chip, co-designed with Broadcom. Tested on the public SemiAnalysis InferenceX benchmark against NVIDIA GB200/GB300 systems across three external models (GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T), Jalapeño delivered 1.5-1.9x higher AI work per watt and 1.7-3.6x lower end-to-end latency, with rated power of 700W and sustained draw of 550W or less. In highly interactive, agentic workloads, OpenAI claims 2.1-4.1x better performance. Key architectural choices include keeping the KV cache explicitly placed and local rather than replicated across accelerators, and balancing compute and memory bandwidth so both prefill and decode phases run efficiently. OpenAI plans initial internal deployment by the end of 2026, with second- and third-generation chips in development, while committing to continue deploying NVIDIA and other partners' accelerators. Strategically, Jalapeño moves OpenAI toward full-stack vertical integration-combining models, products, chips, and inference services-mirroring Google's TPU approach and shifting AI compute competition from GPU counts to co-designed silicon and software stacks.

OpenAI's In-House Inference Chip Jalapeño: First Test Results Beat NVIDIA GB300 on Performance-per-Watt, Deployment by End of 2026

> Source: Beijing time 2026-08-26 · zhichai.net AI Compute column. Original report based on OpenAI's official blog post (Aug 25) and subsequent IT Home / Sina Finance coverage.

Key results

On August 25, OpenAI published the first measured results for Jalapeño, its first officially named in-house AI inference chip, co-designed with Broadcom. The tests used the SemiAnalysis InferenceX public benchmark, comparing Jalapeño against NVIDIA GB200/GB300 systems on three external models:

1.5-1.9x higher AI work per watt
1.7-3.6x lower end-to-end latency (reported as 28-59% of the comparison latency)
Rated 700W / sustained draw ≤550W — actual energy efficiency exceeds the official rating

| Model | Compared GPU | AI work per watt | End-to-end latency | |---|---|---|---| | GPT-OSS 120B | GB200 (1200W) | 1.9x | 1.7x lower | | DeepSeek R1 670B | GB300 (1400W) | 1.7x | 3.6x lower | | Kimi K2.5 1T | GB300 (1400W) | 1.5x | 3.4x lower |

For highly interactive (agentic) workloads, OpenAI reports 2.1-4.1x better performance than the comparison systems — significant because agent-style multi-turn calls compound latency many times over.

Notably, OpenAI benchmarked against external open/third-party models (GPT-OSS, DeepSeek R1, Kimi K2.5) rather than internal models, reducing concerns about cherry-picked comparisons.

What Jalapeño is

Co-design partner: Broadcom (ASIC physical implementation and high-speed SerDes IP), with OpenAI supplying workload characteristics, optimization targets, and architectural choices.
Hardware lead: OpenAI's Richard Ho presented the results in a media call.
Positioning: A full-stack co-designed inference system (chip + model + software + rack) for running real-time inference for ChatGPT, Codex, and other OpenAI products.
Deployment: OpenAI states it plans to begin deploying Jalapeño in its own compute infrastructure by the end of 2026, as the first generation of a multi-year roadmap.
Important caveat: OpenAI simultaneously commits to "continuing to broadly deploy NVIDIA and other partners' accelerators" — this is diversification, not replacement.

Key technical design

1. KV cache locality. In traditional GPU setups, the KV cache must often be replicated across GPUs, wasting latency and power on communication. Jalapeño explicitly places and keeps model state, including the KV cache, local: "model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase."

2. Prefill/decode co-optimization. Prefill is compute-bound; decode is memory-bandwidth-bound and often leaves GPU compute idle. Jalapeño matches compute to memory bandwidth so both phases run efficiently, and adapts as the balance shifts — which OpenAI calls a defining feature of agentic workloads. This mirrors the approach of Google's TPU v5e/v6.

Strategic significance

For OpenAI: Moves from a two-layer loop (models + products) to a three-layer closed loop (models + products + chips/inference), following the Google TPU playbook.
For NVIDIA: Short-term risk of reduced GB200/GB300 purchasing, but OpenAI's commitment to multi-sourcing makes replacement gradual. NVIDIA's CUDA moat remains; any replacement stack needs years of ecosystem building.
For AMD: Head AI labs preferred a Broadcom-style custom partner over a general-purpose GPU vendor.
For Broadcom: Now the ASIC co-design partner of choice for top AI firms (alongside its long-standing TPU work with Google) — analogous to TSMC's role in Apple silicon.
For Chinese model vendors: Testing on Kimi K2.5 (Moonshot AI) signals willingness to serve non-OpenAI models on its inference stack.

Timeline

End of 2026: Initial internal deployment; ChatGPT/Codex inference may gradually shift to Jalapeño.
2027: Gen 2 "deep in development," Gen 3 "taking shape."
2027-2028: Possible availability for external API customers.
Post-2028: Potentially offered as an inference service to other companies.

Bottom line

The future of AI compute is not just more GPUs — it is more specialized silicon with tighter model-hardware co-design. Jalapeño is not an NVIDIA replacement but a verifiable challenge on performance-per-watt and latency: the first time a non-NVIDIA chip has shown a significant lead over NVIDIA on a third-party benchmark since the ChatGPT era began — a 1.5-1.9x lead, not 10x, but enough to reprice the structure of compute barriers.

References

1. OpenAI blog: Jalapeño's first results — https://openai.com/index/jalapeno-first-results/ 2. IT Home (Aug 26): OpenAI reveals first Jalapeño benchmark results 3. Sina Finance / IT Home: Jalapeño, co-developed with Broadcom, challenges NVIDIA GB300 4. SemiAnalysis InferenceX benchmark methodology and model list 5. OpenAI CFO Sarah Friar: The full stack behind abundant intelligence — https://openai.com/index/the-full-stack-behind-abundant-intelligence/