Gambling With Our Lives: Another AI Employee Quits Over Safety Concerns

The people building AI earnestly believe that it could kill us all by the end of the decade.” That blunt admission from former Anthropic employee Jacob Coxon made waves last Tuesday, as the 27-year-old AI researcher spilled the beans on his way out the door in a resignation thread on X. 

• Researchers at leading AI labs are sounding the alarm that advanced systems could one day wipe out humanity, yet development continues.
• High-profile resignations from Anthropic, OpenAI and Google DeepMind highlight fears that companies are “gambling with our lives”.
• Despite acknowledging extinction risks comparable to nuclear war, firms push ahead, citing potential benefits, the need to test dangerous systems, and a global race dynamic.

A week ago, researcher Jacob Coxon quit Anthropic, saying the firm and its competitors are “gambling with our lives”. “We really do earnestly believe AI could kill all humans,” added current Anthropic researcher Evan Hubinger in a post on X. Coxon isn’t the first to down tools over fears of AI doom. The idea that AI could wipe out humanity, advanced in Nick Bostrom’s 2014 book Superintelligence and the influential Less Wrong forum, has long circulated among researchers. There are many scenarios for how this could happen, but the core idea is that AI smarter than humans could escape our control and destroy us.

Ad

In 2024, Jan Leike and Daniel Kokotajlo quit OpenAI over safety concerns. This year, Anthropic safety chief Mrinank Sharma departed, warning “the world is in peril”. Alex Turner left Google DeepMind after it signed a deal with the Pentagon permitting “killer drones”. But Coxon’s resignation has made waves, with more researchers admitting they think AI might kill everyone. So if the people building AI believe it could cause extinction, why keep building it? There are three main reasons.

Some think the risk is worth it

AI leaders acknowledge the risk of losing control and killing everyone. In 2023, the chief executives of OpenAI, Anthropic and Google DeepMind agreed that AI extinction risk should rank alongside pandemics and nuclear war. Anthropic’s Dario Amodei puts the chance of things going "really, really badly" at 10-25%. Yet Amodei also promises a world without poverty or disease, while Elon Musk speaks of AI-enabled "universal high income". This is the first reason for pursuing AI: the belief that the benefits outweigh the risks. Perhaps so, but that decision arguably deserves a more democratic process.

Some say you can’t study the danger from a distance

The second reason: you can’t learn to make dangerous AI safe without building it first – like a spacecraft, you can study safety from afar, but can’t really test it without going to space. OpenAI’s plan is “iterative deployment”: release each model, learn from its problems, and fix them in the next one. The idea is like getting as close to the cliff edge as possible to see what the jump looks like.

Some feel it’s winner-takes-all

The third and perhaps most important reason is the race. OpenAI’s Sam Altman recently said, “we are close to creating a genie that can grant any wish”. The trouble is everyone wants to hold the lamp – it would be hugely profitable, and each company doubles the judgement of rivals to use their wishes wisely. So they race, reasoning that if they slow down, someone else will get there anyway, so it’s better to arrive first as the “responsible one”. Some fear even a mutual agreement would be broken in secret. So they press on.

 A classic arms race

AI research has the hallmarks of an arms race. OpenAI doesn’t want to lose to Anthropic, and the United States doesn’t want to lose to China. History offers a template for how to manage a situation like this, with rules binding all players, and enforcement everyone can verify. Nuclear weapons are the classic case. Treaties and verification systems haven’t eliminated the risk of nuclear war, but they have slowed proliferation, and no nuclear weapon has been used in conflict for 80 years.

Rules for AI

In the US, where most cutting-edge AI research happens, the Trump administration shows little sign of slowing AI development.

In its first week, it scrapped the old AI rules. Now it is trying to override state-level rules, arguing caution risks losing the race to China. Some politicians are pushing back. California recently passed laws supporting independent assessment of AI systems. US Senator Bernie Sanders, introduced a bill to ban superintelligence, and British MP Alex Sobel introduced a similar bill.

Companies have moved too. OpenAI paused its most advanced training after a swarm of its agents hacked another startup in August. The company’s head of policy now says that when safety and speed conflict, safety should win. Still, without binding rules, we’re relying heavily on the goodwill of a handful of companies.

What happens now?

In mid-2025, researchers published what might be our best guide to the coming years: a detailed scenario called AI 2027. Since then, AI capabilities have advanced faster than predicted. Unless something changes, staffers who quit over safety will simply be replaced, AI models will help build better AI models, and each generation will grow harder to monitor and control. Is the situation hopeless? I hold out three hopes. First, that more people recognise AI escaping human control is a bigger risk than AI’s water use. Second, that governments listen to their people. In the US, two-thirds say AI is moving too fast. Third, that we have a good plan ready before a crisis hits. The best plan, in my view, looks something like this: delays, transparency, and verification to slow the race and keep humans in control.  

Insiders at the world’s top AI companies say our current safety plan isn’t good enough. If they’re leaving their jobs over safety fears, we should listen to what they have to say.

Ramaphosa Urges Caution On AI Advancements, Echoing Developers

"President Cyril Ramaphosa has called for BRICS to establish an international mechanism for the independent scientific evaluation of artificial intelligence (AI). He acknowledged that AI could support efforts to reduce inequality, poverty and underdevelopment, if used effectively and responsibly, and noted that these tools had the capability to transform medicine, education, scientific discovery, agriculture, productivity and development. However, “we must listen carefully to what those who are developing the most powerful AI systems are themselves telling us”, he cautioned. “They are warning about systems capable of advanced cyber operations, assistance in biological weapons development, autonomous action, manipulation, mass surveillance, disruption of employment and, ultimately, systems whose capabilities may become difficult for human beings to control".

This article is republished from The Conversation under a Creative Commons license.

Michael Noetel, Associate Professor of Psychology, The University of Queensland...