Inference is becoming the expensive part people can feel

Training a frontier model gets most of the attention. Inference is the work that happens after training, every time a person or an agent sends a request and waits for an answer. It is repeated constantly, and it is where delay, power consumption and capacity limits become visible to users.

That is why the chip announcement from OpenAI matters even though there is no new model in it. The company says its first custom inference chip, Jalapeño, is designed to avoid a familiar trade-off: systems are often tuned for either high throughput or fast individual responses. OpenAI says Jalapeño can improve both in the operating ranges it tested.

The result is a company report, not an independent benchmark result. Still, it gives a more concrete look at what OpenAI thinks its next infrastructure bottleneck will be: serving capable models quickly enough, often enough and with less energy per useful answer.

What OpenAI measured

OpenAI says it tested Jalapeño with the public InferenceX benchmark from SemiAnalysis. The company used three open-weight models: GPT-OSS 120B, DeepSeek R1 and Kimi K2.5. It compares the full serving process rather than only a chip-level calculation, which is closer to what an application experiences.

Across those three models, OpenAI reports 1.5 to 1.9 times more peak AI work per watt and 1.7 to 3.6 times lower end-to-end latency than the comparison systems. It says the chip is rated at 700 watts, while measured sustained power on the workloads was at or below 550 watts. The company normalised performance using each comparison chip’s published power rating.

The numbers should be read as OpenAI’s own results. The article gives useful methodological detail, but it does not replace a broad set of independent measurements, a public price comparison or evidence from a live customer fleet. Those will matter once the hardware is deployed.

Why an inference system is more than a chip

Generating an answer has several different bottlenecks. Reading a long prompt needs compute. Producing one token after another leans heavily on memory bandwidth. Moving model state between chips can add delay. A fast accelerator can still lose time if the rest of the system has to wait for data to arrive.

OpenAI says Jalapeño was designed around that whole chain: compute, memory, networking, software and the rack-scale system. It highlights keeping the model state used during generation close to the relevant compute and reducing data movement between stages.

That is an engineering argument rather than a consumer feature. But it is relevant to anyone using an agent that takes many sequential steps. Small delays compound quickly when a system has to plan, call tools, read results and try again.

The first deployment is still ahead

OpenAI says it plans to begin deploying Jalapeño in its compute infrastructure by the end of 2026. It also says it will continue to use accelerators from NVIDIA and other suppliers for training and inference. A custom chip is therefore an addition to its supply strategy, not an instant replacement for the existing market.

The company says AI systems helped shorten aspects of the chip design and programming process. It reports that, for selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than existing human-written implementations. Those figures concern selected blocks, not the full model.

The meaningful test will be less tidy than a benchmark chart. It will be whether OpenAI can manufacture, qualify, operate and update the system at scale while retaining the speed and efficiency gains it reports now.

What is confirmed, what OpenAI says, and what is open

Confirmed: OpenAI published its Jalapeño results on 25 August 2026. Its announcement describes Jalapeño as its first custom inference chip, names the three open-weight models used for public-benchmark testing and says deployment is planned by the end of the year.

OpenAI’s claims: Jalapeño delivers a better combined balance of throughput, power efficiency and latency than the comparison systems tested, and a full-stack design can make future AI products faster and less costly to serve.

Open questions: independent reproduction of the results, the exact comparison configuration and commercial terms, behaviour with frontier closed models in production, manufacturing volume, fleet reliability and the extent to which efficiency gains reduce prices rather than simply enable more demand.

Sources

  1. OpenAI — Jalapeño’s first results show industry-leading speed and efficiency in AI inferencePrimary OpenAI announcement, published 25 August 2026. Source for the system description, reported benchmark setup, performance figures, power-rating methodology and planned deployment timing.
  2. OpenAI — The full stack behind abundant intelligencePrimary OpenAI company context, published 25 August 2026. Source for how OpenAI frames custom inference hardware within its broader infrastructure strategy.