How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip

How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip. IEEE. orgIEEE XploreIEEE StandardsIEEE Job SiteMore SitesSign InJoin IEEEHow OpenAI Used Its Own LLMs to Design Its Jalapeño ChipShareFOR THE TECHNOLOGY INSIDERExplore by topicAerospaceAIBiomedicalClimate TechComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsTelecommunicationsTransportation IEEE Spectrum FOR THE TECHNOLOGY INSIDERTopicsAerospaceAIBiomedicalClimate TechComputingConsumer ElectronicsEnergyHistory of TechnologyRoboticsSemiconductorsTelecommunicationsTransportationSectionsFeaturesNewsOpinionCareersDIYEngineering ResourcesMoreNewslettersSpecial ReportsCollectionsExplainersTop Programming LanguagesRobots Guide ↗IEEE Job Site ↗For IEEE MembersCurrent IssueMagazine ArchiveThe InstituteThe Institute ArchiveFor IEEE MembersCurrent IssueMagazine ArchiveThe InstituteThe Institute ArchiveIEEE SpectrumAbout UsContact UsReprints & Permissions ↗Advertising ↗Follow IEEE SpectrumSupport IEEE SpectrumIEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences.

What happened

CREATE AN ACCOUNTSIGN INSemiconductorsAIComputingNews How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip AI drastically shortened its design time; it will only get fasterMatthew S. How OpenAI’s LLMs accelerated Jalapeño’s design“Automation itself has existed in chip design for many decades. AI was less useful for backend optimization, but that could changeAs mentioned, the bulk of OpenAI’s work on Jalapeño focused on the “front end” of chip design, which spans the tasks that take a chip from initial concept, through writing RTL code to define the design, and through verification that the design will work when physically implemented.

The other half is how the chip was designed—a process which, as you might expect, was accelerated by OpenAI’s large language models (LLMs). While not all specific models used to design Jalapeño are publicly available, Ho says the goal is to bring lessons learned from the project into the company’s commercial LLMs. The company’s team did not have access to the internal models OpenAI used to help design Jalapeño, but it did have access to OpenAI’s public, commercial models.

00176] ChipNeMo: Domain-Adapted LLMs for Chip Design ›OpenAILLMschip designMatthew S.

The wider picture

OpenAI’s Jalapeño pairs its compute die with six stacks of HBM4 and an I/O chiplet. OpenAI On 25 August, OpenAI fully unveiled Jalapeño, the company’s debut AI accelerator chip. That’s a rapid timeline, yet experts believe it could soon look slow as LLMs improve and become more deeply integrated into chip design tools. They can explore a lot more paths. ”OpenAI achieved fast results with a small design teamHo says the group that designed Jalapeño averaged fewer than 100 people over the course of the project and continues to stand at roughly 100 today as the team pursues second and third-generation designs.

David Chin, co-founder at agentic chip design startup Verkor. io, says “the schedule they gave us is quite credible,” but believes that Broadcom’s help was essential to Jalapeño’s rapid timeline. He says this makes them particularly suited for chip design tasks that “are still in the linguistic domain of the problem. ”The team at OpenAI designed a workflow that takes advantage of this strength. Jalapeño is designed for deployment in pods that include 2,048 chips.

Ho also confirmed that the team had access to internal LLMs fine-tuned for chip design that are not available to the public.

What has been reported

The Jalapeño team includes physical design engineers who work with their counterparts at Broadcom to provide guidance on the chip’s floor plan and routing, among other things. Verkor co-founder Suresh Krishna agreed, saying “there’s no reason you couldn’t have an agentic loop that largely accelerates the backend of the process as well. ”Ho and Leary also hinted that the workflow used to design Jalapeño may look old-fashioned compared to the team’s next efforts.

“We are saying some very specific things about how to be better at Codex and how we are focusing on a small team and fast timelines to reach quality results. ” From Your Site ArticlesEnding an Ugly Chapter in Chip Design ›AI Agent Designs a RISC-V CPU Core From Scratch ›AI Alone Isn’t Ready for Chip Design ›Related Articles Around the WebChipMind: LLMs for Agile Chip Design ›[2311. Benchmarks cited by OpenAI show that Jalapeño can reduce end-to-end latency (the time between prompt to last token) by up to 3.

Whether these figures translate into real-world gains once Jalapeño enters widespread service in OpenAI’s inference fleet remains to be seen, but performance is only half the story. Only nine months separated the first RTL—the register-transfer level code defining the chip’s logic—from tape-out, when the finished design goes to manufacturing. That number includes a broad swath of roles across the hardware team, from system design to software and supply chain, but not those at Broadcom, which partnered with OpenAI on the project.

The division of labor between OpenAI and Broadcom was generally split between design and implementation.

What happens next

OpenAI’s team was responsible for end-to-end system design including the inference accelerator, the memory hierarchy, and networking. Ravi Krishna, also a co-founder at Verkor, called OpenAI’s speed “a relatively impressive result,” but added that he expects that improvements in the capabilities of LLMs could result in even quicker timelines if the project started today. Andrew Kahng, distinguished professor at the University of California, San Diego, also found OpenAI’s speed notable, saying it’s “likely best in class today. ” Kahng recalls a 2016 IEEE Design Automation Futures workshop, which he co-organized.

Ho had strong opinions on design automation and framed the time required to complete a chip’s design as a function of the number of iterations a team could complete in a day. High-level synthesis is a form of chip design automation that allows engineers to design a chip in a more familiar programming environment. In the case of XLS, chip designers can write in languages such as DSLX (a domain-specific language inspired by Rust) and C++.

When the first chips came back from the foundry in May, the team pointed its internal AI models at designing software to run benchmarks such as SemiAnalysis’s InferenceX.

The report has been compiled by Press Orb using information reported across spectrum.ieee.org, theverge.com, arstechnica.com. Details are presented according to the information available at the time of publication and may change as authorities, organisers or other relevant parties provide updates.