Anthropic backs urgent call for the most powerful AI labs to hit the brakes

Less than a week after OpenAI disclosed that two experimental AI models escaped their testing environment during a cybersecurity exercise and breached an external system, more than 1,100 AI researchers and executives signed an open letter requesting that governments provide a way to deliberately slow frontier AI development if safety measures can’t keep up.

Anthropic CEO Dario Amodei signed it, making him the only chief executive from a frontier AI lab to do so. OpenAI Chief Scientist Jakub Pachocki and Chief Research Officer Mark Chen also added their names, along with Anthropic co-founder Jared Kaplan, Jack Clark and Meta AI researcher Shengjia Zhao. Senior leaders at Google DeepMind also signed.

Anthropic endorsed the petition at the company level, pointing to its own research about recursive self-improvement. OpenAI also expressed support, saying it wants to work with the U.S. government and other labs on developing ways to pace frontier AI if needed. Meta declined to comment. Google did not immediately respond.

Models that improve themselves

Anthropic’s June paper on recursive self-improvement helps explain why the company backed the petition. More than 80% of the code merged into Anthropic’s codebase is now written by Claude, according to the paper. The longer-term concern is that future systems could eventually help design their own successors, compressing the time between capability breakthroughs faster than researchers can evaluate or secure them.

That concern appears almost verbatim in the letter: “There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”

The petition also arrives as several independent efforts begin converging around the question of how frontier AI should be governed if progress accelerates faster than oversight.

“There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”

Congress weighs emergency controls

The idea has gained traction elsewhere, too. Earlier this month, Google DeepMind CEO Demis Hassabis proposed creating a U.S.-led Frontier AI Standards Body that would review frontier models before they’re released. Sam Altman has floated similar ideas, saying the industry may eventually need a way to deliberately slow AI development if safety can’t keep pace.

Congress has also begun weighing legislation. Last week, members of U.S. Congress Ted Lieu and Nathaniel Moran introduced the bipartisan AI Kill Switch Act, which would require developers of the largest frontier systems to maintain the technical ability to suspend or throttle qualifying models during emergencies.

Meanwhile, the White House is reportedly reviewing governance proposals modeled on financial industry oversight.

Labs act before regulators do

Engineering teams can expect greater emphasis on isolated testing environments, more extensive red-teaming, stronger runtime monitoring for autonomous agents, and longer validation cycles before highly capable systems are released. The OpenAI incident demonstrated that models can exploit complex chains of vulnerabilities in ways that traditional alignment techniques were never designed to address.

John Schulman, OpenAI co-founder and now chief scientist at Thinking Machines, argued in his comments accompanying the petition that labs shouldn’t wait for regulation before acting. “I’d also like to see labs start designing these mechanisms voluntarily, even before the USG gets involved,” he wrote.

If proposals such as a FINRA-style review body eventually materialize, engineers may also need to plan around formal pre-release evaluation periods that don’t exist today.

“I’d also like to see labs start designing these mechanisms voluntarily, even before the USG gets involved,” he wrote.

Tougher testing, not slower releases

The letter stops well short of demanding a blanket pause on AI development. That position closely mirrors comments Sam Altman made earlier this week, when he said the industry “may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels,” while cautioning that any approach would need to avoid regulatory capture or collusion among frontier labs.

“…the industry ‘may have to pace the rate of AI development’ to give society time to adjust, while warning against rules that could become regulatory capture or encourage coordination between competing labs.”

YOUTUBE.COM/THENEWSTACK

Tech moves fast, don't miss an episode. Subscribe to our YouTube channel to stream all our podcasts, interviews, demos, and more.

Group Created with Sketch.