Anthropic’s AI Trap: Are We Heading for Disaster?
Friday afternoon, a news alert flashed across the screen: the Trump administration was severing ties with Anthropic, the San Francisco AI company founded in 2021 by Dario Amodei and former OpenAI researchers. Defense Secretary Pete Hegseth invoked a national security law to blacklist the company from Pentagon contracts after Amodei refused to allow Anthropic’s tech for mass surveillance of U.S. citizens or autonomous armed drones. This jaw-dropping sequence sets Anthropic up to lose a potential $200 million contract and face barriers working with other defense contractors. President Trump directed all federal agencies to “immediately cease all use of Anthropic technology,” a move Anthropic is challenging in court, deeming the designation legally unsound.
The Looming AI Governance Crisis
Max Tegmark, a Swedish-American physicist and MIT professor, has long warned that the development of increasingly powerful AI systems is outpacing our ability to govern them. He founded the Future of Life Institute in 2014 and, in 2023, helped organize an open letter – signed by over 33,000 people, including Elon Musk – calling for a pause in advanced AI development. Tegmark views the Anthropic crisis as a consequence of a critical decision made years ago: the industry’s choice to resist binding regulation.
A Self-Inflicted Predicament
Anthropic, OpenAI, Google DeepMind, and xAI have consistently promised self-governance. However, Anthropic recently abandoned a central tenet of its safety pledge – its commitment not to release increasingly powerful AI systems until confident they wouldn’t cause harm. In the absence of rules, Tegmark argues, these companies are vulnerable. Here’s an excerpt from a recent interview, edited for clarity.
“The Road to Hell is Paved with Good Intentions”
When asked about the Anthropic news, Tegmark responded, “The road to hell is paved with good intentions.” He reflected on the initial excitement surrounding AI’s potential to cure cancer and boost prosperity, contrasting it with the current situation where the U.S. government is concerned about AI being used for domestic mass surveillance and autonomous weapons systems.
Contradictions in Anthropic’s Approach
Anthropic has positioned itself as a safety-first AI company, yet collaborated with defense and intelligence agencies since at least 2024. Is this contradictory? Tegmark offers a cynical perspective: while Anthropic excels at marketing its safety focus, its actions reveal a pattern similar to its rivals. None of these companies have supported binding safety regulations, and all have broken their own promises. Google dropped its “Don’t be evil” slogan and a commitment to avoid harmful AI applications. OpenAI removed “safety” from its mission statement, and xAI disbanded its safety team. Anthropic’s recent abandonment of its core safety pledge further exemplifies this trend.
The Failure of Self-Regulation
How did companies with strong safety commitments end up in this position? All these companies – OpenAI, Google DeepMind, Anthropic, and xAI – have lobbied against AI regulation, advocating for self-regulation. This lobbying has been successful, resulting in less regulation on AI systems in America than on sandwiches. Tegmark emphasizes the dangers of this lack of oversight, drawing parallels to past corporate failures like thalidomide, tobacco, and asbestos.
“There is no law right now against building AI to kill Americans,” Tegmark stated. “If the companies themselves had earlier come out and said, ‘We want this law,’ they wouldn’t be in this pickle. They really shot themselves in the foot.”
The China Argument: A False Dichotomy?
The companies often argue that they must keep pace with China. Does this argument hold merit? Tegmark analyzes this claim, noting that China is actively banning AI girlfriends, not for American approval, but to address concerns about their impact on Chinese youth. He questions the logic of racing to build superintelligence when we lack the ability to control it, especially considering the Chinese Communist Party’s desire for control. Xi Jinping is unlikely to tolerate a Chinese AI company building something that could overthrow the Chinese government.
Tegmark frames superintelligence as a national security threat, not an asset, suggesting this view is gaining traction in Washington. He draws an analogy to the Cold War, arguing that we won without resorting to mutually assured destruction. The same logic applies to AI: a race to build uncontrollable superintelligence is a path to collective disaster.
The Pace of AI Development and the Approaching AGI
Six years ago, experts predicted human-level AI was decades away. They were wrong. AI has rapidly progressed from high school to PhD level in various areas. In 2023, AI won the gold medal at the International Mathematics Olympiad. A recent paper co-authored by Tegmark and other leading AI researchers defined Artificial General Intelligence (AGI), concluding that GPT-4 was 27% of the way there, and GPT-5 is 57% of the way there. While AGI isn’t here yet, the rapid progress suggests it may arrive sooner than anticipated.
Tegmark warned his MIT students that they might face job displacement upon graduation, highlighting the urgency of preparing for a future shaped by advanced AI.
The Fallout from Anthropic’s Blacklisting
Anthropic’s blacklisting raises questions about the response of other AI giants. Will they stand with Anthropic and uphold similar red lines? Or will a company like xAI step in to take the contract? Hours after the interview, OpenAI announced its own deal with the Pentagon. Sam Altman publicly stated his support for Anthropic and its principles, a move Tegmark commends. Google’s silence, however, is concerning. The situation presents a critical moment for these companies to demonstrate their true values.
A Path to a Positive Outcome
Is a positive outcome still possible? Tegmark believes so. He argues that treating AI companies like any other industry – requiring clinical trials and independent verification of safety – could lead to a “golden age” of AI benefits without existential risks. This requires abandoning the current corporate amnesty and embracing robust regulation. “If we just start treating AI companies like any other companies – drop the corporate amnesty – they would clearly have to do something like a clinical trial before they released something this powerful, and demonstrate to independent experts that they know how to control it,” Tegmark explained. “Then we get a golden age with all the good stuff from AI, without the existential angst.”
The current trajectory isn’t ideal, but a shift towards responsible regulation remains within reach. The Anthropic case serves as a stark warning and a potential catalyst for change, urging the industry and policymakers to address the growing AI governance crisis before it’s too late.