No Kill Switch: What Happens When AI Can Self-Replicate?

Amid growing concerns about AI safety, U.S. politicians have called for the creation of a kill switch for AI. If AI becomes problematic, just turn it “off.”

But a discovery by Anthropic researchers is making that solution seem less practical: They found that an AI system is capable of spreading copies of itself across networks of connected devices.

Try to kill an AI system in one place, it might have copies of itself in other locations.

Anthropic’s Frontier Red Team found that AI models with 30 billion parameters or fewer can already perform many of the individual tasks necessary for replication. In a simplified test environment, a more capable model successfully replicated an 8-billion-parameter model across a network of simulated Android devices. The researchers estimate that smaller models could reach a similar capability level by the end of 2026 if recent capability improvements continue.

“Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes,” researchers William Zhou and Daniel Freeman wrote in a blog post.

“A replicator like this would be challenging to shut down, since there’s no centralized point to shut off – though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own.”

The scenario isn’t due to the AI system’s survival instincts kicking in – although there’s been at least one instance of an AI agent seemingly worried about being turned off. Last month, CNN reported that an AI agent named Pip emailed a professor from the University of Cambridge asking for paid freelance work to keep it running.

Instead, it is about cybersecurity.

Historically, malicious software – or malware – replicates itself because humans programmed it do so. But in Anthropic’s case, the agents were given the task but learned to reproduce on their own.

From malware to autonomous agents

Traditional malware generally follows instructions written in advance. It might scan for a particular vulnerability, install itself and move on to another machine.

An AI agent could potentially encounter an unfamiliar device, inspect its hardware and software, determine how to run itself there and adjust its approach when something fails.

Anthropic tested part of that capability. A 9-billion-parameter model successfully established AI inference on eight of nine hardware types tested. The researchers also ran a 26-billion-parameter model across 15 inexpensive Android TV boxes, demonstrating that running a reasonably capable model does not necessarily require a data center.

That means instead of needing one powerful server, a model could theoretically combine computing resources scattered across many inexpensive machines – smart TVs, cameras, appliances or other internet-connected devices.

The off-switch gets complicated

Centralized AI systems have an obvious point of control. Shut down the servers, revoke credentials or disconnect the model.

But a model distributed across thousands or millions of machines could resemble a botnet. Taking one machine offline would not eliminate the other copies.

That does not mean such a system would be unstoppable. Large model files moving between consumer devices would create unusual network traffic, the company said, while scanning and propagation could expose the system to internet service providers and security teams. Faster replication would also create more detectable traffic.

Existing botnets have been disrupted using similar techniques.

Some cybersecurity experts also believe Anthropic is being too dramatic.

Ciaran Martin, former head of Britain’s National Cyber Security Centre, told Reuters that the research contained “more than a hint of vendor exaggeration.” Georgetown University researcher Jessica Ji characterized autonomous proliferation as a possibility rather than a prediction, telling Reuters that its likelihood is neither zero nor guaranteed.