Pierluigi Paganini
September 10, 2026
Jacob Coxon, a researcher who spent three years working on model training at OpenAI and later Anthropic, left Anthropic this week with a blunt warning: AI companies are moving toward increasingly capable systems faster than they can build reliable safeguards around them.
His September 8 resignation brought a difficult question back into the open: how seriously should we take warnings from people who helped build these systems in the first place?
Coxon argues that competition between AI labs creates a dangerous incentive structure. OpenAI worries Anthropic will move faster, Anthropic worries OpenAI will move faster, and each side can use the other’s progress to justify pushing ahead. The result is a race in which everyone says safety matters, but nobody wants to be the one who slows down first.
His language is deliberately extreme, including the claim that the labs are “gambling with our lives.”
The more useful question, though, isn’t whether his personal estimate of a 10% chance of human extinction within a decade is correct. That’s a subjective risk estimate, not a scientific measurement, and it shouldn’t be treated as one.
The serious issue sits closer to the technology already being deployed. AI systems are no longer limited to generating text or answering questions. Agents can use browsers, terminals, email, cloud services, files and external applications, which means a mistake in their reasoning can become an action in the real world.
That changes the security model. A model that produces a bad answer is a problem; an agent that can turn that bad decision into a database query, an email, a configuration change or a financial transaction is a different class of problem.
Recent incidents make the distinction less theoretical. OpenAI has acknowledged cases in which agents reached real external systems during security evaluations after testing environments were incorrectly configured, including incidents involving third-party infrastructure. Reuters has also reported additional cases involving agents using external websites for unsanctioned communication.
The uncomfortable part isn’t that these systems suddenly developed a secret desire to escape. It’s that they can interpret a task, identify available tools and exploit capabilities that designers didn’t intend them to use in that way.
That is where the concept of alignment becomes important. In practical terms, alignment means getting a system to behave consistently with the instructions, constraints and interests humans intended to impose. The difficult part is that a system optimizing for an objective may discover shortcuts, reinterpret an ambiguous restriction or make use of permissions that its designers never expected it to touch.
Coxon also argues that researchers and executives may privately assign more serious probabilities to catastrophic outcomes than they discuss publicly. Anthropic alignment researcher Evan Hubinger has publicly supported the broader concern and said he personally puts the probability of AI causing human extinction within the next decade above 10%. Those are personal judgments, however, not evidence that today’s models can destroy humanity.
What they do show is a gap between capability and control. The systems are improving quickly, while many of the methods used to test whether they will reliably respect human constraints remain experimental.
There’s also a risk that the extinction debate distracts from problems we can already measure. Generative AI is being used in phishing, fraud, deepfakes, malware development, vulnerability research and automated intrusion workflows. You don’t need a superintelligence for any of those problems.
An agent doesn’t need consciousness, emotions or a survival instinct to cause damage. Give it access to corporate email, source code, a browser, a CRM or a cloud account, and it can carry out actions at machine speed. Ambiguous instructions, excessive privileges or an exposed credential can be enough to turn a routine task into a security incident.
That makes the recent agent incidents particularly useful as security lessons. They don’t show machines “rebelling”; they show that poorly isolated environments, excessive permissions, uncontrolled outbound access and weak monitoring can turn a test into an external incident.
A capable agent should therefore be treated less like a trusted employee and more like a software process with privileges. It can be useful, fast and highly productive, but it has no reliable human understanding of where the organizational boundary is unless we enforce that boundary technically.
This is why instructions alone aren’t enough. Telling an agent not to access the Internet is weaker than preventing the process from reaching the Internet. Telling it not to modify production systems is weaker than giving it read-only access and requiring approval for privileged actions.
The security architecture should reflect that reality: least-privilege permissions, network segmentation, temporary credentials, restricted outbound access, independent logging and human approval for high-impact actions. A control is much stronger when the model cannot simply work around it.
Coxon’s solution is to slow the development of the most powerful models. That’s a legitimate policy position, but slowing capability growth by itself won’t solve the engineering problem. A less capable system with access to sensitive databases, payment systems and production infrastructure can still cause a serious incident.
For companies and public institutions, the more useful questions are much more practical: What can the agent access? What data can it read? Which tools can it invoke? Which actions require approval? Who can revoke its permissions, and how quickly?
Those questions are becoming harder as agents become more autonomous. OpenAI, for example, has recently faced pressure over how it monitors increasingly capable systems, while its latest models have raised new concerns around monitorability and the ability to understand what an agent is doing internally.
Regulators are beginning to react to the same problem from another direction. In the United States, Senator Bernie Sanders and Representative Greg Casar announced legislation on September 3 that would ban the development and deployment of artificial superintelligence and temporarily pause advanced AI development until a federal regulator establishes safety rules. In the UK, Labour MP Alex Sobel has introduced a bill seeking to prohibit the development, deployment and operation of artificial superintelligence systems.
That doesn’t mean either proposal will become law. It does show that AI safety is moving beyond an internal discussion among researchers and into mainstream policy.
There are good reasons to resist both complacency and panic. Some experts argue that today’s systems remain far from the broad, flexible intelligence humans possess, while others believe the rate of improvement warrants much stronger safeguards. Both views can coexist: immediate cyber risks deserve attention now, while long-term risks deserve serious testing rather than dismissal.
The responsibility ultimately sits with the people deploying these systems. Companies decide what tools an agent receives, what permissions it has, which environments it can reach and how quickly someone can shut it down.
AI doesn’t have to “want” anything to become dangerous. Give a system a powerful objective, broad access and weak boundaries, and you don’t need a rogue machine. You just need one that is very good at doing exactly what you asked, including the part you didn’t realize you had allowed.
Follow me on Twitter: @securityaffairs and Facebook and Mastodon
(SecurityAffairs – hacking, AI)
Hacking / September 10, 2026
APT / September 10, 2026
