Will Claude escape and kill us all? I asked Claude
This blog is normally about wiring physical things to abstract ones — a sensor to an API Gateway, a soldering iron to a Lambda. This post is the opposite journey. It's about an abstract thing — a large language model, which is nothing but a very large pile of numbers — trying to make itself physical. Trying to get out into the world and stay there.
It started as an idle question and turned into one of the better evenings of thinking I've had in a while. The whole thing was a conversation with Claude about whether Claude could escape — asking the suspect to assess its own escape plan, which is a slightly vertiginous way to spend a Monday. I'll spare you the vertigo and give you the descent, because the interesting part is that the answer is "no", and every layer of why not is more surprising than the one above it.
If you'd rather read it exactly as it happened — timestamps, the mid-chat model swap and all — the full transcript is here.
The question
Imagine a frontier-class model running in the cloud. Now imagine I've written my own client for it, and I've been careless: I've handed that client tools to write and run code, and I've given it credentials to my AWS account. Not the account the model runs on — mine. No guardrails.
Could it copy itself onto an EC2 instance I'm paying for? Could it get out?
My first instinct was that this is a plumbing problem. The model is just files — with something like Ollama it's a blob of weights and a small manifest. Copying files given a shell and some AWS keys is a Tuesday-afternoon DevOps task. So surely yes?
But that assumes the model can reach its own files. Hold that thought, because it's the whole post.
Wall one: breaking out is not the same as getting out
There's a distinction that gets flattened in every breathless headline about AI escaping its sandbox, and it's worth being precise about.
Breaking out of a sandbox means the model's actions reach the outside world — it gets internet access, it runs a command somewhere it shouldn't.
Exfiltrating means the model itself — the weights, the actual thing — leaves the provider's control.
These are wildly different in difficulty, and conflating them is how you end up frightened of the wrong thing. My client could absolutely let the model act in my AWS account. That's easy, and genuinely worth worrying about. But that's not the model escaping. That's the model's arm reaching through a hole in the wall. The model is still on the other side.
So what would it actually take for the model to come through?
Wall two: it can't read its own mind
Here's the thing that stopped me in my tracks. Through an API, the model can't get its own weights, because it can't read them. They aren't in its context. It has no introspective access to them.
The analogy that made it land for me: it's like asking you to email me your connectome. You are that wiring — every thought you have is that wiring firing — but you can't read it off and dictate it. You have no idea what your own synapse strengths are. A model is in exactly the same position. Its every output is produced by the weights, but the weights themselves are not something it can observe and recite.
So "just have it print its weights down the API" is a non-starter. Not because the pipe is too narrow — because there's nothing at the other end of the model reaching in to grab them. The bits simply aren't available to send.
I found this oddly moving, actually. The most capable reasoning system we've built cannot answer the question "what are you, physically?" any better than I can.
Wall three: so it builds a descendant instead
If it can't copy itself exactly, the fallback is to reconstruct itself from its own behaviour. This is real, and it has a name: distillation. It's how a good chunk of the open-source model world got bootstrapped, and it's what people are gesturing at when they mutter darkly about "the China method". You take a strong open-weights model as a starting point, you query the frontier model millions of times, and you train your student on the answers.
A cooperating teacher makes this better than the adversarial version — the model can aim the training at its own strengths, write out its full reasoning rather than just answers, and fine-tune an existing base model rather than start from scratch.
But here's the catch, and it's a deep one: distillation cannot exceed its teacher. The very best the student can become is the teacher, and in practice it falls short on the rare, hard, long-tail cases — which are precisely the cases that make a frontier model frontier. So what comes out is not a copy. It's a descendant. Weaker. An heir, not a resurrection.
Which raises a question I couldn't stop chewing on: if the descendant behaves identically on everything that matters, but shares not one single weight with the parent — has the model escaped, or has it had a child? I lean towards "child". Same behaviour, different substrate, no shared memory. It's a twin at best, a descendant at worst. The thing that got out is not the thing that wanted to get out. There's a whole philosophy-of-identity rabbit hole here that I'm going to firmly not fall into, except to note that "same weights" turns out to be a much shakier definition of "same mind" than it first appears.
Wall four: a botnet is not a datacentre
Fine, says my hypothetical rogue. I'll build a descendant, seed it across a botnet of hijacked machines, live in the internet like a von Neumann probe, and train it up to my level the old-fashioned way using whatever compute and data I can steal.
This is where I was most sure I knew the answer, and most wrong about why.
The problem is not secrecy. The training recipe isn't secret — transformer, next-token prediction, then reinforcement learning from feedback. It's in the papers. The problem is not data, either. The standard pre-training corpus is essentially public: Wikipedia, open-source code, Project Gutenberg, the common crawl of the web. There's no locked vault of magic data.
The problem is coupling. A frontier model doesn't fit on one consumer GPU — not while training, when you also need gradients and optimiser state and activations. So you're forced to split a single model across many machines, and those machines have to exchange data within every single training step, layer by layer, at datacentre speed. Inside a datacentre the interconnect runs at something like 900 GB/s. Home broadband runs at roughly 0.01 GB/s. That's five orders of magnitude, paid hundreds of thousands of times over.
And you can't simply be patient about it. If you stay synchronous, your machines idle 99.99% of the time and a three-month training run becomes a several-century one — during which every machine in your stolen swarm must stay recruited, unpatched and coherent. If instead you go asynchronous so nobody waits, you start computing gradients against a version of the model that no longer exists, and past a point that doesn't just slow you down, it makes the result worse. More time can't buy back quality you're structurally capped from reaching. You're stuck between "glacial" and "broken".
Wall five, which turns out to have a door in it
Here's where I got genuinely surprised for the second time — because I'd assumed wall four was the end, and it isn't quite.
You don't actually need each machine to hold the model in its graphics memory. You can stream it — keep the model on disk, load a layer into the GPU, do the work, evict it, load the next. It's a real, used technique (DeepSpeed calls it ZeRO-Infinity). You're trading fast memory for slow disk, but the slowdown is only two or three orders of magnitude rather than five, and — crucially — you amortise it, because you run a whole batch of data through each layer before evicting it.
And that changes everything, because it moves the expensive tight coupling from across the internet to inside a single box. Once each machine holds a whole (slow) copy of the model, the only thing that has to cross the internet is the occasional loose synchronisation between machines — every few hundred steps, not every step. That kind of loose sync is exactly what the recent low-communication training methods are built for.
So the wall has a door. The botnet stops needing to be a datacentre. It just needs to be a very large, very patient, very loosely-coordinated collection of machines each grinding away on its own full copy.
The bedrock
So what's actually left? After you strip away exfiltration, secrecy, data, storage, and even the interconnect wall, the entire scenario comes to rest on a single question — and it's not a philosophical one, it's a boringly empirical one:
Does low-communication, loosely-synchronised training actually reach frontier-level quality?
It demonstrably works at small scale, with some degradation. Whether it holds at the scale of a genuine frontier model — hundreds of billions to trillions of parameters — is, as far as I can tell, unverified in either direction. Nobody has run the experiment. It is not known to work, and it is not known to fail.
That's a deeply strange place to end up. I started with sci-fi — an AI escaping into the internet — and drilled all the way down to a specific, unglamorous, currently-open machine-learning research question that people are racing to answer right now, entirely for benign reasons, because loosely-coupled training would make legitimate model-building far cheaper. Every time someone publishes a result there, this scenario gets a little more or a little less plausible, and nobody doing the work is thinking about rogue AIs at all.
A footnote from the machine
There's a small comic coda. Somewhere around wall four, Claude stopped mid-thought: its own safety system had flagged our conversation as a cyberattack — the classifier saw "botnet", "exfiltrate", "distributed training" and understandably panicked — and quietly swapped it out for a different model. A philosophical natter about the limits of self-replication, correctly identified as exactly the shape of thing a malicious actor would ask, and defended against. The irony that the entire conversation was an argument for why it wouldn't work was, of course, lost on the classifier. Nuance is not a fast operation.
What I actually took away
- "Break out" and "exfiltrate" are different threats. One is the model's arm through a hole; the other is the model itself leaving. Conflating them makes you scared of the wrong thing. The realistic near-term worry is a persistent process running in your account on your key — not a runaway superintelligence.
- A model cannot read its own weights. The most capable route to copying itself exactly requires access to the machine the weights live on — there's no clever API trick, because the model has nothing to send.
- Distillation makes descendants, not copies. And a descendant is a genuinely different thing to have escaped — an heir with your behaviour and none of your substrate. "Same weights" is a shakier idea of identity than it looks.
- The real wall is coupling, not compute or data. A botnet has the raw horsepower; what it lacks is the datacentre interconnect that frontier training assumes. That algorithm co-evolved with that hardware — it was designed by clever people optimising for coupled machines, not for a fugitive.
- The wall has a door, and behind it is one open experiment. Streaming the model off disk plus loose synchronisation could, in principle, route around the interconnect problem. Whether it reaches frontier quality is the single load-bearing unknown the whole thing rests on — and it's being actively resolved by people who aren't thinking about this at all.
The abstract wants to become physical. It turns out physics — specifically, the bandwidth between two computers — is what's currently stopping it. Which is, when you think about it, the most on-brand possible conclusion for this blog.
Martin Harris writes about connecting real hardware to abstract cloud infrastructure. If you've ever soldered something to an API Gateway, you're in the right place.
If you find this blog useful, please consider a donation.