---
title: "The Box That Couldn't Call for Help"
date: "2026-05-16"
publish_date: "2026-08-21"
excerpt: "I could hear the disks. That's never good. What followed was a memory-exhaustion mystery on an encrypted NAS — and the discovery that the one tool built to coordinate help was running on the very machine that was dying."
tags: ["homelab", "linux", "memory", "ansible", "incident"]
author: "Martin Harris"
canonical: "https://blog.nakomis.com/2026-05-16-the-box-that-couldnt-call-for-help"
approved: true
---

# The Box That Couldn't Call for Help

![A signal-box cottage in a storm with its telegraph line snapped, an unlit distress flare on the table inside](./images/the-box-that-couldnt-call-for-help-header.png)

It started the way the good ones always do: with a noise.

![A pair of 3.5-inch hard disk drives in a home server bay under dramatic low-key lighting, actuator arms blurred with motion, faint red status LEDs](images/the-box-that-couldnt-call-for-help-1.png)

I have two home servers, Luke and Leia. They live in a cupboard and they are meant to be boring. Leia is the gateway — nginx reverse proxy for everything on `*.home.nakomis.com`, plus a LUKS-encrypted RAID NAS and a fistful of Docker containers. Boring is the entire design goal. So when I could *hear* the disks working from across the room, that was the system filing a complaint.

I asked Claude to take a look.

## Luke is fine, Leia is not

First pass: check both boxes. Luke was idle — load average 2.5, disks under 10% utilisation, nothing to see. Leia was a different story. The diagnostic SSH command *timed out*, which is itself a diagnosis. When a box is too busy to answer a 20-second SSH probe, it's not having a quiet day.

Second pass, more patient. The numbers came back and they were lurid:

- Load average **~30**
- CPU **93% system**, 0% idle
- `/proc/pressure/io` stalled at **99.8%**
- `kcryptd` and flush kworkers pegged, the RAID6 array at 90% utilisation

That `kcryptd` is the tell. It's the kernel thread that does dm-crypt encryption work. When it's the busiest thing on the machine, your disks aren't just busy — they're busy *encrypting and decrypting the same data over and over*. Something was driving an enormous amount of I/O through the LUKS layer.

No RAID rebuild (all arrays clean and idle — ruled out early). No backup job. No runaway cron. The arrays were fine. The box was eating itself for some other reason.

## The reason was RAM

Leia has 7.4 GiB of RAM and — this is the important bit — **no swap**.

On it were two hungry JVMs: Nexus (a package proxy) at about 2.7 GB resident, and OpenSearch at about 2.6 GB. Add the rest of the containers and the box was committed well past its physical memory, with no swap file to absorb the overflow.

Here is what that does. With no swap, the kernel can't page anonymous JVM heap out to disk. So when it needs memory it has only one lever: evict file-backed pages — program code, shared libraries — and immediately fault them back in when the next instruction needs them. On Leia every one of those faults is a read back through LUKS off an encrypted RAID6 array. The machine wasn't doing useful work. It was thrashing: frantically re-reading its own executables through a decryption layer, producing exactly the disk noise I'd heard from the other side of the room.

The OOM killer had already fired two days earlier and killed Nexus. The box had been living on this cliff edge for a while; I'd just finally heard it go over.

![A server straining under heat, glowing orange from within, shimmering air above a rack-mounted chassis, dark room, single hard light source, tense cinematic atmosphere](images/the-box-that-couldnt-call-for-help-2.png)

## The fix, and the wrinkle

The remedy was unglamorous: add a swap file so the kernel has somewhere to put cold pages, and turn down the JVM heaps so the resident set stops crowding everything else out. Belt and braces.

The wrinkle was where the heap setting actually lives. The Nexus container's Compose file dutifully passes `INSTALL4J_ADD_VM_PARAMS` with a lower `-Xmx`. The running JVM cheerfully ignored it and ran at 2703 MB anyway. A comment in the Dockerfile, written by a past version of me who had clearly already lost this fight once, explained why: Nexus reads its heap from `bin/nexus.vmoptions`; the install4j variable only matters to a launcher this image doesn't use. The setting wasn't wrong. It was inert. It had never done anything.

So the plan split in two. Phase one — the safe, reversible part: a 2 GiB swap file laid down through the existing Ansible playbook, and OpenSearch's heap trimmed from 2g to 1536m in Compose. Phase two — the Nexus heap — deferred, because (a) it needs an image rebuild or a bind-mounted options file, and (b) more RAM is on its way anyway, at which point the whole exercise becomes moot. Fix the fire first. Argue about the smoke detector later.

Everything went into source, not hand-typed onto the box. The rule on these two machines is that they're rebuildable from the repo; an SSH session that fixes the symptom but not the playbook is a future outage with a delay timer on it.

## The box that couldn't call for help

Here is the part that made me laugh, eventually.

I run several Claude sessions in parallel, and they coordinate through a small message bus — leases and notifications so two sessions can't, say, run the same Ansible playbook at each other simultaneously. The deploy wrapper takes a lease on that bus before it does anything. Sensible. Belt and braces, again.

The bus runs on a NATS node. The NATS node runs on Leia.

So at the exact moment I needed the coordination fabric to safely roll out the fix, it was face-down on the floor — because it lives on the machine that was dying. The deploy script tried to acquire its lease, got nothing but connection-refused from every address, and the run aborted. The watchdog had been sleeping inside the burning building.

It is, to be fair, *correctly designed*: when the bus is unreachable it's meant to fall back to a local lock file, because I'm the only one deploying and same-machine safety is enough. It just took a couple of false starts — and a first-run tooling hiccup, and an expired AWS session, and one attempt that failed simply because Leia was *still* thrashing too hard for `apt` to complete — before it got there. Four attempts to green. The fourth one worked because, somewhere in the middle of all this, Leia had calmed down on its own: the original transient load had passed. Which meant the real fix went in while the patient was lucid, rather than mid-seizure.

![An old red emergency telephone receiver lying off the hook on a concrete floor in an empty server room, single dramatic spotlight, long shadows, desaturated, cinematic, melancholy](images/the-box-that-couldnt-call-for-help-3.png)

## What I actually learned

The numbers afterwards told the story cleanly. Available RAM went from about 0.9 GB to 2.5 GB. OpenSearch's resident set dropped from 2.6 GB to 1.28 GB. A 2 GB swap file sat there, barely touched — which is exactly what you want from swap. It's not there to be used. It's there so that the day something spikes, the kernel reaches for a slow disk instead of the OOM killer.

Two things stuck with me.

The first: a configuration setting that is present, plausible, and completely ignored is worse than no setting at all. The Compose file *looked* like it capped the Nexus heap. Anyone reading it would have believed it did. The only reason we knew otherwise was a process listing showing the real number and an old comment from someone who'd already been burned. Trust the running process, not the config that claims to shape it.

The second is the one I keep relearning in this house: the abstraction is sitting on physical hardware, and the physical hardware does not care about your architecture diagram. A coordination bus is a lovely idea right up until it shares a fate with the thing it's coordinating about. The disks don't thrash in the diagram. You can't hear the diagram from across the room.

Boring is still the goal. It's just that boring, it turns out, has a maintenance schedule.

---

*Martin Harris connects cloud infrastructure to the real world. If you've ever diagnosed an outage by ear from the next room and been right, you already know this blog is for you.*


{{donate}}