An essay from Anthropic PBC’s Dario Amodei saying artificial intelligence companies must “pace the frontier” has got the world talking about a slowdown in their most cutting-edge research. OpenAI’s Sam Altman and SpaceX’s Elon Musk publicly agreed with the post. But don’t expect their companies to slam the brakes in the race to “superintelligence.”
There is simply too much at stake from the industry’s perspective. Anthropic has an initial public offering in the works and its profit margins are kept high by the promise of future AI capabilities, not just existing cash flows or its $65 billion in annualized revenue. The company has picked Nasdaq for a listing that could value it at $2 trillion, and signs point to the floatation still being on track. Chipmaker Nvidia Corp. may throw in $10 billion or so.
As for frontier research itself, Amodei’s statements suggest he’ll slow development only so far, while finding other ways to address the fears stoked by his own researchers that their AI models could kill all humans as they become more powerful and are able to redesign and improve themselves.
See more: One AI Trade for Now, Many Trades Later
The key is in his choice of words. To “pace” a technology’s evolution doesn’t always mean to genuinely slow it down, it can be about controlling the speed as part of a managed path toward a lofty goal. Marathon runner Eliud Kipchoge has used pacemakers. He was still the first person to complete the distance in less than two hours.
In fairness, Amodei does also call for slower development: “We must slow the pace at which we improve the capabilities of AI models,” he writes. But this is no indefinite moratorium. He describes “a balanced rate” of advancement, and stresses that “progress will still seem fast.” When Anthropic Chief Financial Officer Krishna Rao posted his boss’s essay on LinkedIn, he spoke primarily of “pacing” and “common standards.”
And Amodei makes clear that he doesn’t want American firms to ease up enough for their Chinese competitors to “pull ahead.”
Rather than meaningfully decelerate their research and development, it seems more likely that Anthropic and OpenAI will focus on Amodei’s most concrete recommendation: to embed independent evaluators in their companies to check that the work on frontier models is happening safely. His own firm has already committed to this.
President Donald Trump publicly ruled out any federal coordination of such efforts, ignoring a request from Amodei, and went on to say that the only guardrails AI needed were a strong, high-IQ president. He wants the US to beat China to superintelligence because he sees that as the ultimate way to “win.” Amodei says it would be a security threat to cede ground to “authoritarian regimes, chiefly the Chinese Communist Party.”
Anthropic’s leaders are driven by the belief that the creation of superintelligent systems is inevitable, and it’s irresponsible to let the “least cautious actors” get there first. That is why, in spite of his long-voiced concerns about safety, Amodei has pursued risky techniques like recursive self improvement, where AI systems largely create their own successors.
As well as China, the other reason to doubt a meaningful slowdown in R&D is old-fashioned commerce. You could argue that even a pause on building more powerful systems wouldn’t hurt Anthropic’s business in the short term, since many businesses don’t want to buy the most expensive, frontier models. By taking this approach, Amodei’s company could save billions on training new models, and continue to sell the already-developed versions of its Claude model.
But longer term its products would risk becoming commoditized, with less to differentiate Claude from OpenAI’s ChatGPT or Google’s Gemini, and that would threaten the 80% gross margins Anthropic enjoys, according to David Rainville, lead manager of Sycomore Sustainable Tech fund, which holds about €800 million ($923 million) in global tech stocks. That would be a very unhappy prospect for Amodei and Altman as they head toward the public markets.
A person close to Amodei tell me he earnestly wants to build AI that benefits humanity, and that talking openly about the challenges now will help society better adjust later.
And yet, if Amodei took his safety argument to its logical conclusion, he’d pause frontier development completely on the reasonable grounds that a tiny risk of destroying human civilization is not worth taking. That would be my preference.
He could also be more prudent in his choice of “independent” evaluators. Amodei cited the non-profit research organization METR as his example of one. Yet METR draws from the same tightknit AI safety world as Anthropic. Ajeya Cotra, a researcher behind METR’s investigation into the OpenAI agent hack on Hugging Face, spent more than nine years working at San Francisco-based Coefficient Giving, a philanthropic organization formerly known as Open Philanthropy where Amodei was previously a technical advisor.
A better choice of auditor would be the UK’s highly regarded AI Safety Institute, which has been testing frontier models since 2023, or the US Center for AI Standards and Innovation.
This is an Oppenheimer moment for AI companies, with a handful of executives helping set the rules for developing some of the riskiest and most unpredictable technology ever seen. The latest statements from Amodei and Altman are reassuring, until you consider how little they may ultimately change. Amodei wants to buy time, not abandon the destination. The race to build superintelligent systems will continue.
A message from Advisor Perspectives and VettaFi: Discover something new! Click here to register for our upcoming webcasts.
Bloomberg News provided this article. For more articles like this please visit
bloomberg.com.
Read more articles by Parmy Olson
