OpenAI is actively exploring alternatives to Nvidia’s artificial intelligence chips due to dissatisfaction with their performance in specific high-speed tasks. While Nvidia remains the primary provider for most of the startup’s operations, OpenAI is seeking new hardware to improve the speed of user interactions within its ChatGPT and Codex platforms.
This shift in strategy focuses on the “inference” phase of AI, where a trained model generates responses to user prompts. According to eight sources familiar with the matter, OpenAI has been evaluating different hardware options since last year to address these concerns.
Performance Concerns in AI Inference
OpenAI’s dissatisfaction stems from the speed at which current Nvidia hardware delivers responses for specialized tasks. These include software development and instances where AI interacts directly with other software. The company is specifically looking for hardware that can eventually handle approximately 10% of its total inference computing requirements.
A primary technical hurdle involves how chips handle memory. Nvidia’s graphics processing units (GPUs) typically rely on external memory, which creates latency as data is retrieved during the inference process. In contrast, OpenAI is searching for chips with high levels of embedded memory, known as SRAM. By integrating more memory directly onto the silicon, chips can process data much faster, which is critical for maintaining a smooth user experience as millions of people use AI bots simultaneously.
Impact on Coding Tools and User Experience
The performance issues have been particularly noticeable in Codex, OpenAI’s tool for generating computer code. Internal staff have reportedly attributed some of the limitations of Codex to the GPU-based technology provided by Nvidia. OpenAI CEO Sam Altman recently noted that speed is a top priority for customers using coding models, even if it is less critical for casual users of the standard ChatGPT app.
To address these speed requirements, OpenAI has already taken steps to diversify its hardware. The company recently announced a commercial partnership with Cerebras, a startup that develops specialized chips with high SRAM levels. OpenAI has also considered hardware from Groq, though those discussions reportedly stalled after Nvidia entered its own licensing agreement with that firm.
Shifting Dynamics in the AI Chip Market
The search for alternatives comes at a delicate time for the relationship between the two AI industry leaders. OpenAI and Nvidia have been in long-running investment talks, with Nvidia previously indicating plans to invest up to $100 billion in the startup. However, these negotiations have extended for months longer than expected as OpenAI continues to sign deals with rival chipmakers, including AMD.
Despite the search for alternatives, OpenAI executives have defended their ongoing partnership with Nvidia. A company representative stated that Nvidia still provides the best performance per dollar for most inference tasks. Nevertheless, the move toward custom and specialized hardware mirrors strategies used by other tech giants like Google, Amazon, and Meta, all of which have developed proprietary chips to manage their growing AI workloads.
