The current bottleneck for the generative AI revolution is no longer just the availability of high-end GPUs or the quality of training data. Instead, the industry has hit a physical wall: the power grid. As hyperscalers struggle to secure enough megawatts to keep their clusters running, the conversation in the developer community has shifted from raw FLOPS to a more pragmatic metric. The focus is now on how many tokens a data center can produce per megawatt of power consumed. This energy crisis is the silent driver behind the latest move by OpenAI to decouple its future from a total reliance on general-purpose hardware.

The Architecture of Jalapeño

To address these constraints, OpenAI has partnered with Broadcom to develop Jalapeño, a dedicated ASIC designed exclusively for large language model inference. Unlike previous attempts to modify existing GPU architectures, Jalapeño was built from a blank slate. The development cycle was remarkably aggressive; from the initial design phase in mid-2024 to the final tape-out, the process took approximately 16 months. In the world of custom silicon, where ASIC development typically spans several years, this timeline represents an extreme acceleration of the hardware lifecycle.

The chip is engineered for high-throughput, low-latency performance using a Single-Token Prediction (STP) approach. In practical benchmarks, this architecture delivers impressive results. When running the Kimi-K2.5 and GPT-OSS models, the chip recorded processing speeds of approximately 1,400 tok/sec/user. Even with the computationally demanding DeepSeek R1 model, Jalapeño maintained an interactive performance of over 700 tokens per second per user in a concurrency-1 environment. To support these speeds, OpenAI integrated HBM4 (High Bandwidth Memory 4), ensuring that the memory bottleneck typically associated with LLM inference is significantly mitigated through a tight hardware-software co-design.

The Shift to Tokens per Megawatt

While raw speed is impressive, the true disruption of Jalapeño lies in its performance per watt (perf/W). In nearly every tested scenario, Jalapeño outperformed Nvidia's Blackwell architecture in energy efficiency. What makes this achievement significant is that OpenAI reached these numbers without relying on common software-level optimization tricks. The chip does not use Multi-Token Prediction (MTP), speculative decoding, or prefill-decode disaggregation. It achieves superior efficiency through the sheer optimization of its physical silicon for the specific mathematical patterns of inference.

There was an initial assumption that Jalapeño would be a niche product, optimized solely for OpenAI's proprietary models. However, results from the InferenceX benchmark suggest otherwise. Jalapeño is a general-purpose AI inference chip capable of handling diverse workloads. To demonstrate this versatility, the team successfully ran a version of the game Doom, ported via Codex prompts, proving that the chip can handle varied computational tasks beyond simple text generation. This versatility ensures that the hardware remains viable as model architectures evolve, preventing the silicon from becoming obsolete the moment a new model is released.

For OpenAI, the pivot to Jalapeño is a strategic response to the reality of data center economics. While budget and physical floor space are concerns, the absolute limit on system expansion is now the power supply. By prioritizing tokens per MW as the primary KPI, OpenAI is treating power efficiency as the ultimate determinant of economic value. The ability to generate more intelligence per watt directly translates to lower operational costs and a higher ceiling for scaling their agentic workflows.

The industry is moving toward a future where the most successful AI companies will be those that can optimize the physical cost of a single token.