When a typical business finds its supply chain costs have risen, it will absorb the increase to keep its customers happy. Nvidia isn’t a typical business. It enjoys gross profit margins of 75% and commands between 70% and 90% of the global market for artificial-intelligence chips. So when the companies that make memory components put their prices up, Nvidia passed the hike on. It’s now raising the prices of servers containing its accelerator processors by more than 15%, according to Bloomberg News.
That won’t deter big tech customers — some of the most well-capitalized companies in history — from buying Nvidia chips. But it will probably force a reevaluation of how companies like Meta Platforms Inc., Microsoft Corp. and OpenAI use them. The US firms will need to start thinking more like their counterparts in China, who’ve learned from years of export bans to work around chip constraints by introducing more efficient software.
Take Hangzhou-based DeepSeek. After US export controls blocked it from buying Nvidia’s most powerful chips, it figured out how to compress the memory needs of its AI models by writing a clever mathematical shorthand called multi-head latent attention. It cut the company’s chip memory needs by about 96%, allowing it to build a world-class product at a fraction of the typical hardware costs.
Silicon Valley could have pursued similar innovations to use semiconductors more efficiently. In fact, it should have. The use of Nvidia chips to process tasks for end customers is remarkably inefficient. One founder of an AI chip startup recently told me that AI firms were using less than 15% of the theoretical capacity of Nvidia’s graphics processing units (GPUs). Others estimate that figure to be as low as 5%.
The problem isn’t the GPUs themselves — remarkably powerful machines that can do millions of mathematical calculations at once — but the flood of information they must handle when thousands are stitched together in a data center. The software1 powering them is good at telling a single chip how to process graphics or perform a calculation. But it wasn’t designed to distribute the data from a large AI model across thousands of chips simultaneously. That leads to massive data traffic jams, and chips sitting idle as they wait for the software to move numbers between them.
Until now, tech giants have dealt with the problem by just buying more chips to make up for the inefficiencies. They could afford to, and being in a fast-moving AI race made it difficult to take the time to rewrite millions of lines of entrenched code for running processors, especially when Nvidia was constantly releasing new chip architectures.
See more: The Quantum Computer Revolution Is Tantalizingly Close
OpenAI and Meta have tried, to their credit, building new software tools and a programming language to enhance performance and “keep GPUs busy.” But these have turned out to be piecemeal efforts that haven’t solved the broader infrastructure issues with data-center software.
For all the talk of persistent, stratospheric demand for compute keeping the AI boom alive, even industry insiders say computing power hit a hard physical limit years ago. “We're right now in a world where we need more performance than ever, and the physics has run out,” one AI semiconductor entrepreneur recently told me.
Tech companies can’t just wait around for chips to get faster. (They certainly won’t get any cheaper.) And Nvidia knows all too well how this can end — with Chinese companies who’ve learned to work around export constraints building hyper-efficient AI tools that are becoming increasingly popular in the West. Capable AI models that don’t need much power from chips pose an obvious threat to Nvidia. That’s why it recently spent $6 billion on licenses from San Francisco-based startup Poolside AI, whose software acts as a kind of traffic cop for the data circulating between thousands of GPUs in data centers, reorganizing flows so they work more efficiently.
Poolside’s Co-Chief Executive Officer Eiso Kant told me recently that the chaotic nature of data-center infrastructure means that if building AI models is an industrialized process, “you spend all of your time improving the factory itself."
“People see these huge fundraises, tens of billions... and they think that's what it takes to train a very capable model, and that's not correct,” he added. Kant says Poolside was able to train its coding model Laguna in about eight weeks, using efficiency techniques to build a system that surpasses much larger models that can cost hundreds of millions of dollars to develop.
Poolside’s approach shows the direction in which the AI race may be shifting. The winners of the first phase of the battle succeeded on brute financial strength: Whoever had the most money to buy vast amounts of computing power stayed in the lead. But China’s growing influence in AI has shown that capital alone no longer guarantees dominance, and Nvidia’s price hike makes such an approach even harder to justify. Alphabet Inc.’s Australian A$5.5 billion ($3.9 billion) bond sale earlier this month to fund its own data centers shows that even the tech giants may be getting close to hitting limits on how much cash they have to spend.
Silicon Valley can’t just throw money at AI anymore to stay in the lead. Instead, it needs to adopt China’s frugal software playbook and start writing smarter code.
1. Specifically the programming language and compilers like Nvidia's CUDA.
A message from Advisor Perspectives and VettaFi: Discover something new! Click here to register for our upcoming webcasts.
Bloomberg News provided this article. For more articles like this please visit
bloomberg.com.
Read more articles by Parmy Olson
