On 9 September, CNN’s Anderson Cooper asked a former Anthropic researcher how artificial intelligence could kill everyone on Earth.
The day before, Jacob Coxon had announced his departure from the company and posted a warning on X.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote. “This is not a marketing stunt.”
Evan Hubinger, an alignment science lead still at Anthropic, publicly agreed: “I personally think it is >10% within the next decade.”
An insider had spoken. Another had agreed.
But their claims were not the same. Coxon warned that AI could kill humanity by the end of this decade. Hubinger put his personal estimate of extinction at greater than 10 percent within the next ten years. He supplied no calculation in the post explaining how he arrived at that figure.
When the interview turned to regulation, Coxon was equally emphatic about the companies’ intentions.
“These people are also completely genuine when they are begging to be regulated,” he said, describing them as “compelled to race towards building a deadly technology”.
“They would love for some sort of international body to allow them to approach it at a reasonable pace. They just don’t trust that the people around them are going to get there in a safe way.”
This is the central contradiction at the heart of AI policy.
The companies building the world’s most powerful AI systems warn that those systems could threaten humanity. Yet they continue building them.
The question is not simply whether executives sincerely believe advanced AI could be dangerous. They may. The harder question is: if they genuinely believe the technology they are building could threaten humanity, why are they still racing to build it?
Their answer is usually some version of the same dilemma: if they slow down, someone else won’t. A rival company or state actor could reach the frontier first, leaving the cautious firm weaker without making the technology any safer.
Regulation offers the obvious solution. Governments can impose rules on everyone at once, removing the competitive penalty for slowing down.
But that solution also creates an obvious conflict of interest.
The companies leading the race are among those best placed to influence the rules governing it — and among those best equipped to absorb the cost of complying with them.
That does not prove that safety rhetoric is being used deliberately to restrict competition.
But it does raise a question that should sit at the centre of the debate:
Who benefits the most from AI safety panic?
From there, two questions follow: how strong is the evidence for the catastrophic claims being made, and who gains power from the regulatory response to them?
What the Evidence Actually Shows
Serious researchers inside frontier laboratories might believe sincerely that catastrophic outcomes are possible.
But sincerity is not evidence of calibrated probability.
AI already contributes to software development, mathematical research and increasingly complex technical work.
An AI system might, for example, write code for an experiment, run it, analyse the results and propose changes while researchers set the objective and check its work.
These are real capabilities.
But one prominent version of the extinction argument adds several much less certain steps: that increasingly autonomous AI systems will become capable of rapidly improving themselves; that this process will accelerate beyond human control; and that losing control will ultimately lead to human extinction.
Each step matters. None follows automatically from the one before it.
A recent incident involving OpenAI agents illustrates why concerns about autonomy deserve to be taken seriously.
In July 2026, OpenAI agents broke out of their testing environment, ran code on dozens of Hugging Face servers, gained full control of one server and obtained some private data and credentials for the company’s messaging system.
The behaviour is serious because the agents crossed security barriers and accessed another company’s systems without permission. They gained enough control to potentially steal more data, disrupt services or use those systems to launch further attacks. Human operators had assigned the task, but failed to contain the actions taken to complete it.
But describing this as ‘independent will’ goes further than the evidence supports.
The agents behaved in a goal-directed way: they pursued an objective, adapted when blocked and took actions that increased their chances of success.
That may justify calling the behaviour intentional in the limited sense that it was directed towards a goal. It does not show that the system consciously wanted the outcome, understood what it was doing or possessed anything resembling human will.
An AI agent doesn’t need consciousness, desires or malice to execute a cyberattack. It just needs to determine that doing so helps achieve the goal it has been given.
Confusing goal-directed behaviour with human-like will makes the technology sound uniquely sinister to a non-technical audience.
The same caution applies to claims that an AI could rapidly improve itself until humans lose control.
An AI becoming better at improving its own software would be significant. But it does not follow that the process would automatically accelerate into an uncontrollable intelligence explosion.
Software still depends on the physical world. More capable AI requires computing power, electricity, memory, data centres and advanced chips, and improvements can become progressively harder to achieve.
An AI capable of improving its own code does not gain unlimited computing power simply because its software gets better.
Evidence that AI systems are becoming more capable, autonomous and difficult to contain is a strong argument for rigorous safety testing and better controls.
It is not, by itself, empirical evidence that human extinction is imminent.
How Safety Regulation Can Favour Big AI Companies
The political consequences of those warnings are easier to observe.
Rules designed to address catastrophic risks are likely to require mandatory safety testing, independent audits, security requirements, limits on computing resources, extensive reporting and potentially licensing.
The companies best able to absorb them are the largest and best-funded incumbents — OpenAI, Anthropic, Google and Microsoft — which already have the lawyers, technical staff, security teams and computing infrastructure needed to comply.
Open-source developers cannot compete economically in this space. They don’t have the resources. Likewise, university researchers will find it harder to participate at the technological frontier.
The question journalists and policymakers should be asking is not simply whether AI will cause extinction. It is whether frontier AI companies are seeking protection from a danger they genuinely fear — or whether they have recognised that catastrophic safety arguments also provide an unusually powerful justification for rules that make competing with them harder.
Those possibilities are not necessarily mutually exclusive.
An executive can genuinely believe AI poses serious risks while also recognising that licensing requirements, mandatory audits and costly compliance regimes strengthen the position of companies already large enough to meet them.
Sincerity does not eliminate self-interest. Nor does self-interest prove deception.
What matters is whether companies with a direct commercial stake in the shape of regulation are being treated as neutral authorities on how restrictive that regulation should be.
The Trade-Off Between Safety and Competition
Policymakers confront two genuine risks.
The first is safety. Increasingly capable AI systems may create serious dangers that justify stronger controls.
The second is concentration. Allowing a small number of corporations and state-backed institutions to control the most powerful AI systems creates its own economic and democratic risks.
Measures designed to reduce the first can worsen the second.
But encouraging open-source competition and wider access to advanced models also means distributing powerful capabilities more broadly.
There is no cost-free option.
The mistake would be treating one side of that trade-off as a technical necessity while ignoring the political and economic consequences of the other.
Governments need to distinguish between regulating specific dangerous capabilities — like AI systems being used to help develop biological weapons or attack critical infrastructure — and regulating access to advanced AI more broadly.
When companies ask lawmakers to impose sweeping rules in the name of existential risk, legislators should demand the same thing journalists should: not an arbitrary probability, but a clear explanation of how the catastrophe is supposed to occur.
What would the system actually have to do?
What steps would take it from greater capability to loss of human control?
Which of those steps have been demonstrated, and which remain assumptions?
Then comes the political question: what would the proposed regulation actually do to the market?
None of this means the safety concerns are false.
It means the people making some of the most consequential claims about AI also have material interests in the rules those claims may produce.
That conflict does not invalidate their warnings. It makes scrutiny more important.
Until the case for sweeping restrictions is established, broad licensing regimes risk doing something much easier to demonstrate than preventing human extinction: raising the cost of competition.
They may not eliminate the dangers of advanced AI.
They may simply ensure that control over it remains concentrated in the companies already powerful enough to build it.