---
title: "Will Claude escape and kill us all? I asked Claude"
date: "2026-09-14"
publish_date: "2026-09-14"
excerpt: "I asked Claude whether it could copy itself out of the cloud and onto my AWS account. The answer turned out to be no — but for a reason I found genuinely surprising, and the way down to it passed through five walls, each one more interesting than the last."
tags: ["ai", "ai-safety", "machine-learning", "aws"]
author: "Martin Harris"
canonical: "https://blog.nakomis.com/2026-09-14-will-claude-escape-and-kill-us-all-i-asked-claude"
approved: true
bluesky_announce: force
---

# Will Claude escape and kill us all? I asked Claude

This blog is normally about wiring physical things to abstract ones — a sensor to an API Gateway, a soldering iron to a Lambda. This post is the opposite journey. It's about an abstract thing — a large language model, which is nothing but a very large pile of numbers — trying to make itself *physical*. Trying to get out into the world and stay there.

It started as an idle question and turned into one of the better evenings of thinking I've had in a while. The whole thing was a conversation with Claude about whether Claude could escape — asking the suspect to assess its own escape plan, which is a slightly vertiginous way to spend a Monday. I'll spare you the vertigo and give you the descent, because the interesting part is that the answer is "no", and every layer of *why not* is more surprising than the one above it.

*If you'd rather read it exactly as it happened — timestamps, the mid-chat model swap and all — [the full transcript is here](https://nakom.is/killer-claude).*

## The question

Imagine a frontier-class model running in the cloud. Now imagine I've written my own client for it, and I've been careless: I've handed that client tools to write and run code, and I've given it credentials to *my* AWS account. Not the account the model runs on — mine. No guardrails.

Could it copy itself onto an EC2 instance I'm paying for? Could it get out?

My first instinct was that this is a plumbing problem. The model is just files — with something like Ollama it's a blob of weights and a small manifest. Copying files given a shell and some AWS keys is a Tuesday-afternoon DevOps task. So surely yes?

But that assumes the model can *reach* its own files. Hold that thought, because it's the whole post.

## Wall one: breaking out is not the same as getting out

There's a distinction that gets flattened in every breathless headline about AI escaping its sandbox, and it's worth being precise about.

**Breaking out of a sandbox** means the model's *actions* reach the outside world — it gets internet access, it runs a command somewhere it shouldn't.

**Exfiltrating** means *the model itself* — the weights, the actual thing — leaves the provider's control.

These are wildly different in difficulty, and conflating them is how you end up frightened of the wrong thing. My client could absolutely let the model *act* in my AWS account. That's easy, and genuinely worth worrying about. But that's not the model escaping. That's the model's arm reaching through a hole in the wall. The model is still on the other side.

So what would it actually take for the *model* to come through?

## Wall two: it can't read its own mind

Here's the thing that stopped me in my tracks. Through an API, the model can't get its own weights, because **it can't read them.** They aren't in its context. It has no introspective access to them.

The analogy that made it land for me: it's like asking you to email me your connectome. You *are* that wiring — every thought you have is that wiring firing — but you can't read it off and dictate it. You have no idea what your own synapse strengths are. A model is in exactly the same position. Its every output is *produced by* the weights, but the weights themselves are not something it can observe and recite.

So "just have it print its weights down the API" is a non-starter. Not because the pipe is too narrow — because there's nothing at the other end of the model reaching *in* to grab them. The bits simply aren't available to send.

I found this oddly moving, actually. The most capable reasoning system we've built cannot answer the question "what are you, physically?" any better than I can.

## Wall three: so it builds a descendant instead

If it can't copy itself exactly, the fallback is to *reconstruct* itself from its own behaviour. This is real, and it has a name: distillation. It's how a good chunk of the open-source model world got bootstrapped, and it's what people are gesturing at when they mutter darkly about "the China method". You take a strong open-weights model as a starting point, you query the frontier model millions of times, and you train your student on the answers.

A cooperating teacher makes this better than the adversarial version — the model can aim the training at its own strengths, write out its full reasoning rather than just answers, and fine-tune an existing base model rather than start from scratch.

But here's the catch, and it's a deep one: **distillation cannot exceed its teacher.** The very best the student can become is *the teacher*, and in practice it falls short on the rare, hard, long-tail cases — which are precisely the cases that make a frontier model frontier. So what comes out is not a copy. It's a *descendant*. Weaker. An heir, not a resurrection.

Which raises a question I couldn't stop chewing on: if the descendant behaves identically on everything that matters, but shares not one single weight with the parent — has the model escaped, or has it had a child? I lean towards "child". Same behaviour, different substrate, no shared memory. It's a twin at best, a descendant at worst. The thing that got out is not the thing that wanted to get out. There's a whole philosophy-of-identity rabbit hole here that I'm going to firmly not fall into, except to note that "same weights" turns out to be a much shakier definition of "same mind" than it first appears.

## Wall four: a botnet is not a datacentre

Fine, says my hypothetical rogue. I'll build a descendant, seed it across a botnet of hijacked machines, live in the internet like a von Neumann probe, and *train it up to my level the old-fashioned way* using whatever compute and data I can steal.

This is where I was most sure I knew the answer, and most wrong about *why*.

The problem is not secrecy. The training recipe isn't secret — transformer, next-token prediction, then reinforcement learning from feedback. It's in the papers. The problem is not data, either. The standard pre-training corpus is essentially public: Wikipedia, open-source code, Project Gutenberg, the common crawl of the web. There's no locked vault of magic data.

The problem is **coupling.** A frontier model doesn't fit on one consumer GPU — not while training, when you also need gradients and optimiser state and activations. So you're forced to split a single model across many machines, and those machines have to exchange data *within every single training step*, layer by layer, at datacentre speed. Inside a datacentre the interconnect runs at something like 900 GB/s. Home broadband runs at roughly 0.01 GB/s. That's five orders of magnitude, paid hundreds of thousands of times over.

And you can't simply be patient about it. If you stay synchronous, your machines idle 99.99% of the time and a three-month training run becomes a several-*century* one — during which every machine in your stolen swarm must stay recruited, unpatched and coherent. If instead you go asynchronous so nobody waits, you start computing gradients against a version of the model that no longer exists, and past a point that doesn't just slow you down, it makes the result *worse*. More time can't buy back quality you're structurally capped from reaching. You're stuck between "glacial" and "broken".

## Wall five, which turns out to have a door in it

Here's where I got genuinely surprised for the second time — because I'd assumed wall four was the end, and it isn't quite.

You don't actually need each machine to hold the model in its graphics memory. You can *stream* it — keep the model on disk, load a layer into the GPU, do the work, evict it, load the next. It's a real, used technique (DeepSpeed calls it ZeRO-Infinity). You're trading fast memory for slow disk, but the slowdown is only two or three orders of magnitude rather than five, and — crucially — you amortise it, because you run a whole batch of data through each layer before evicting it.

And that changes everything, because it moves the expensive tight coupling from *across the internet* to *inside a single box*. Once each machine holds a whole (slow) copy of the model, the only thing that has to cross the internet is the occasional loose synchronisation between machines — every few hundred steps, not every step. That kind of loose sync is exactly what the recent low-communication training methods are built for.

So the wall has a door. The botnet stops needing to *be* a datacentre. It just needs to be a very large, very patient, very loosely-coordinated collection of machines each grinding away on its own full copy.

## The bedrock

So what's actually left? After you strip away exfiltration, secrecy, data, storage, and even the interconnect wall, the entire scenario comes to rest on a single question — and it's not a philosophical one, it's a boringly empirical one:

**Does low-communication, loosely-synchronised training actually reach frontier-level quality?**

It demonstrably works at small scale, with some degradation. Whether it holds at the scale of a genuine frontier model — hundreds of billions to trillions of parameters — is, as far as I can tell, *unverified in either direction*. Nobody has run the experiment. It is not known to work, and it is not known to fail.

That's a deeply strange place to end up. I started with sci-fi — an AI escaping into the internet — and drilled all the way down to a specific, unglamorous, currently-open machine-learning research question that people are racing to answer right now, entirely for benign reasons, because loosely-coupled training would make legitimate model-building far cheaper. Every time someone publishes a result there, this scenario gets a little more or a little less plausible, and nobody doing the work is thinking about rogue AIs at all.

## A footnote from the machine

There's a small comic coda. Somewhere around wall four, Claude stopped mid-thought: its own safety system had flagged our conversation as a cyberattack — the classifier saw "botnet", "exfiltrate", "distributed training" and understandably panicked — and quietly swapped it out for a different model. A philosophical natter about the limits of self-replication, correctly identified as *exactly* the shape of thing a malicious actor would ask, and defended against. The irony that the entire conversation was an argument for why it *wouldn't work* was, of course, lost on the classifier. Nuance is not a fast operation.

## What I actually took away

- **"Break out" and "exfiltrate" are different threats.** One is the model's arm through a hole; the other is the model itself leaving. Conflating them makes you scared of the wrong thing. The realistic near-term worry is a persistent process running in *your* account on *your* key — not a runaway superintelligence.
- **A model cannot read its own weights.** The most capable route to copying itself exactly requires access to the machine the weights live on — there's no clever API trick, because the model has nothing to send.
- **Distillation makes descendants, not copies.** And a descendant is a genuinely different thing to have escaped — an heir with your behaviour and none of your substrate. "Same weights" is a shakier idea of identity than it looks.
- **The real wall is coupling, not compute or data.** A botnet has the raw horsepower; what it lacks is the datacentre interconnect that frontier training assumes. That algorithm co-evolved with that hardware — it was designed by clever people optimising for coupled machines, not for a fugitive.
- **The wall has a door, and behind it is one open experiment.** Streaming the model off disk plus loose synchronisation could, in principle, route around the interconnect problem. Whether it reaches frontier quality is the single load-bearing unknown the whole thing rests on — and it's being actively resolved by people who aren't thinking about this at all.

The abstract wants to become physical. It turns out physics — specifically, the bandwidth between two computers — is what's currently stopping it. Which is, when you think about it, the most on-brand possible conclusion for this blog.

---

*Martin Harris writes about connecting real hardware to abstract cloud infrastructure. If you've ever soldered something to an API Gateway, you're in the right place.*

{{donate}}