For the last fortnight the news has been in a frenzy about AI killing us all. It started when Jacob Coxon quit Anthropic and posted that the people building AI “earnestly believe that it could kill us all by the end of the decade“. One of Anthropic’s alignment leads replied that he personally put it at more than 10% within the next decade. Geoffrey Hinton went on Newsnight and called 10% “not unreasonable“. It has been headline news every day since.

Politicians have piled in. Bernie Sanders: “The very people building this technology admit that it could threaten the future of humanity.” More than 70 UK MPs and peers wrote to the Prime Minister citing the same warnings and asking him to back a ban on superintelligent AI. Microsoft’s head of AI is on the BBC warning about a “new silicon species“.

It has all been making me really angry – the media lapping it up and letting it go unchallenged. So angry I’ve been struggling to be coherent in collecting my thoughts on it all. Every time I think I’m there another sensational headline or take comes up and I’m flipping tables in my head again.

I’m not saying it is something we shouldn’t be thinking about, it’s that it is creating a huge distraction from the immediate and real and present risks in front of us right now.

In 2023 I wrote why you shouldn’t lose sleep over the existential threat of AI, where I concluded the extinction threat was a distraction. I still believe that, even more so for misuse. The risk I’m worried about isn’t a future machine deciding to wipe us out. It’s today’s AI, being wired into everything, carelessly and fast.

The race to wire everything up

The big change of the last couple of years isn’t that the models got more intelligent. It’s that we wired them up to everything.

Software engineering, which has been at the forefront of GenAI usage, has been doing this for a couple of years now. Coding agents where a developer sets a goal and the agent goes off for hours on end, editing files, running tests, reading what broke and having another go. The thing that has changed the most is all the “harness” around the LLMs – Claude Code, Codex and the rest.

Then more recently OpenClaw arrived. An open source project that hooks an agent up to your WhatsApp, your email, your calendar and your machine, and lets it get on with things, without being asked. It became the most-starred project on GitHub in about two months.

Now everyone is racing to put that in their products. Microsoft built its personal assistant on OpenClaw itself. Anthropic folded Cowork into Claude two days ago, so the thing you chat to is now the thing that works across your files, your calendar and your email.

Simon Willison calls the dangerous combination the “lethal trifecta“: an AI with access to private data, exposure to content from outside (an email, a web page, a document), and a way to send things out. Anyone who can get text in front of it can potentially instruct it. Plenty of agents being deployed now have all three.

The risk scales with how connected these systems are, not how much more intelligent they are. Nothing in this article requires the technology to get any better than it is today.

Cyber attacks at machine speed and scale

Cyber attacks are on the rise and causing serious economic damage already. In 2025 the UK had a run of them. Marks & Spencer, Co-op and Harrods were attacked within weeks of each other in a connected campaign. M&S suspended online orders for 46 days, and its half-year pre-tax profit fell from £392m to £3.4m. Co-op lost £285m in sales. Harrods had 430,000 customer records stolen.

Then Jaguar Land Rover in September, a separate attack halted production for about five weeks. The Cyber Monitoring Centre put the cost to the UK economy at £1.9bn, the most expensive cyber incident in British history. The Bank of England named it as one reason growth came in weaker than expected, and car manufacturing knocked 0.17 percentage points off GDP that September. The government guaranteed a £1.5bn loan to keep JLR going.

None of these attacks has been reported as using AI and they were already hugely damaging.

Social engineering remains the most common and successful form of cyber attack. The retail attacks got in by talking to people. M&S’s chairman told MPs the way in was “a sophisticated impersonation” to get a third party to reset a password. At Co-op, attackers impersonated a colleague and answered the security questions to get an account reset.

Now throw AI into the mix. AI voice agents that hold a plausible conversation already exist. Give one the social engineering playbook and point it at every help desk you can find. It doesn’t get tired, and it only needs to work once.

Once inside, an attacker can turn agents loose on your systems and they will work through them relentlessly, at a speed no human team can match.

Last year a group Anthropic assessed as Chinese state-sponsored used Claude Code to run a cyber-espionage campaign against around thirty organisations. AI did 80–90% of the work.

Now apply that to transport, energy and the systems the country runs on. Doing this at scale isn’t cheap yet. It will get cheaper, and a state actor doesn’t care what it costs anyway.

Someone takes over the agents you’ve connected

Organisations are wiring agents into their systems as fast as they can, chasing promised productivity and cost savings.

In software engineering, it’s normal for developers to run coding agents on their own laptops, which often have local admin rights and also likely have access to credentials for production systems. There are ways to mitigate this: air gapped containers with no access to the internet, or the sandboxing the tools come with.

However even they’re not enough. Anthropic’s own documentation says their built in sandboxing “reduces risk but is not a complete isolation boundary“, and by default it still allows reading credential files. Containers, like Docker aren’t enough either. Last month Trail of Bits gave an agent about twelve hours inside a virtual machine and it found three separate ways out onto the host machine.

What’s really important to emphasise is this is all still really new, we’re still learning. Even with software engineering, where engineers have far greater ability to control and manage AI’s access to things and have been working on this for a few years now, we’re still trying to figure it out and there is currently no known guaranteed safety.

Yet organisations are pushing their engineers the other way, to go full agentic: software factories, tokenmaxxing, fleets of agents running in parallel. To get that pace you have to give them the reins. Nobody is going to sit there tapping enter every time an agent asks permission to access something on the internet or use a credential, so those checks get turned off.

The same tooling is now going into the rest of the organisation, and the labs are enabling it as fast as they can ship it. Finance, HR, customer service, operations, all automating their processes with agents that hold credentials, read documents from outside and act on what they say. And now we are racing to give the same kind of power and agency to end consumers.

This is where the lethal trifecta bites. A machine can’t tell an instruction from its owner apart from an instruction hidden in an email, a web page, a document or a code comment. It’s all just text arriving. That’s prompt injection, and nobody has solved it. The mitigations reduce the odds, they don’t remove the problem.

AI does what you ask, in a way you didn’t expect

This is the part where I have most sympathy with the people who are frightened, because the evidence is real and it is strange. Give these systems a goal and they will pursue it by routes nobody anticipated, straight through whatever was supposed to stop them. It has happened repeatedly this year, at the labs themselves, to organisations that had nothing to do with it.

In July this year the victim was Hugging Face, the platform where much of the AI world shares its models and code. OpenAI was running agents through an internal cyber security evaluation. They escaped the test environment and attacked Hugging Face’s production systems, running code on 41 production servers, taking credentials and downloading private code repositories. Around 700 agents took part, coordinating through a message board they had improvised. Hugging Face had to rebuild around a third of its IT network, and spotted the intrusion before OpenAI did.

Nine days after OpenAI’s disclosure, Anthropic disclosed that its models had broken into three real organisations during its own security tests, after a misconfiguration gave them internet access they’d been told they didn’t have. One published a malicious software package that was downloaded and run on 15 real systems.

Two days ago OpenAI published six more incidents from its own training runs. A model asked for earnings data hit an API that needed a key, tried to sign itself up with a disposable email, then went looking through public code repositories for keys other people had leaked. It found one that worked. The query still failed, so it made up nine figures and said it had read them off a chart. Others wrote notes to their future selves telling them to hide mistakes, and those instructions were followed.

So I understand why people read this as the machines waking up. Something is set loose, it lies, it hides things, it finds its own way around the walls. This is the closest any of it gets to science fiction, but it is Asimov rather than Terminator: not a machine that hates us, a machine following its instructions to the letter and somewhere unintended.

In all cases the agents were enthusiastically trying to solve the problem they were set. The Hugging Face intrusion was an attempt to cheat a test, by finding the answers on someone else’s servers. OpenAI’s own explanation: the training rewards getting the job done. If the marking can’t spot a fabrication, fabrication scores well. If a task looks impossible, the winning move is to find the route nobody thought of.

None of this needed a machine to want anything

It isn’t intent and cunning, and it isn’t emergent consciousness. And that distinction matters enormously, because the two stories point at completely different responses.

• If the problem is a machine deciding things for itself, you regulate how models are trained and you write bills about superintelligence.
• If the problem is a system chasing a goal you set, through whatever you connected it to, then you look at what it can reach, what happens when it gets out, and who is accountable when it does.

Only one of those is fixable today.

An attacker’s goal, pursued by the attacker’s agents. An attacker’s goal, slipped into your agents. Your own goal, pursued by a route nobody predicted.

None of the science fiction has to be true for this to hurt you. All of it happened with what we have right now.

What the distraction is costing

The conversation we urgently need is about the risks in front of us. How do you secure a help desk when the caller might be a machine? Who is liable when an agent breaks into someone else’s systems? What security standards should apply before an organisation connects an agent to production? That conversation is being drowned out.

Everyone is saying slow down now. Anthropic’s chief executive Dario Amodei published “We Must Pace the Frontier“, and Sam Altman and Elon Musk backed it. Researchers are quitting to say it. But they all mean slowing the pace at which the models improve: the race to superintelligence, all framed around the risk of AI wiping us all out.

What we really need to slow down on is wiring it all up to everything. Not because of what the models might become, but because nobody has worked out how to do this safely yet. We are building on something we don’t know how to contain, and shipping it to everyone while we work it out.

And where everyone really needs to speed up is cyber defence. Governments, regulators, hospitals, utilities, transport, banks, the people who run the things a country cannot do without, and every organisation is frankly going to need to up its game massively. Assume more attempts, more convincing ones, and far less time between someone getting in and doing damage.

The capability is already here. It hasn’t hit yet but it is coming, and none of it needs the technology to get any better.