---
title: "The Refusal That Saved the Filesystem"
date: "2026-08-11"
publish_date: "2026-09-05"
excerpt: "A partition resize script died because parted asked a question nobody was there to answer. That refusal was the only thing between a routine maintenance window and a corrupted root filesystem — and it was the first of seven green signals that turned out to be measuring the wrong thing."
tags: ["infrastructure", "linux", "failure-modes", "ansible", "homelab"]
author: "Martin Harris"
canonical: "https://blog.nakomis.com/2026-08-11-the-refusal-that-saved-the-filesystem"
approved: true
---

# The Refusal That Saved the Filesystem

![A vintage smoke alarm mounted on a ceiling, wired by a visible cable to the light switch of the same dim utility room, dust motes hanging in a single shaft of light](images/the-refusal-that-saved-the-filesystem-1.png)

Cal is a £100 iMac from 2011 that spends its retirement computing embeddings for my home estate. It needed its single 480 GB partition split in two, so the OS root could be made read-only and the data could live on its own partition. Shrink the filesystem, move the partition boundary, carve out the free space. A well-trodden procedure. I even wrote a runbook first.

The runbook said:

> Shrink the FILESYSTEM to 360G (target is 371G — 11G of deliberate slack)

That sentence is wrong in a way that is almost invisible. `resize2fs 360G` means 360 **GiB** — 386.5 GB. `parted … 380GB` means 380 × 10⁹ bytes, which produces a 370.9 GB partition. So the plan, faithfully executed, would have left the filesystem **15.6 GB larger than the partition containing it**. The "11 GB of deliberate slack" was in reality a 15.6 GB overlap. Two tools, both using the letter G, meaning different things by it.

The script ran. The filesystem shrank. And then `parted` printed:

```
Warning: Shrinking a partition can cause data loss, are you sure you want to continue?
```

— and, because it was running with `-s` and script mode cannot answer prompts, it **aborted**.

Not because it detected the overlap. `parted` has no idea about filesystems; it moves boundaries, that's all. It aborted because nobody was there to say yes. A safety property nobody designed, doing the job the arithmetic should have done.

## The success that broke two things

With the units fixed, the partition work finished properly. The playbook flipped the root filesystem to read-only — the entire point of the exercise — and reported success. `ok=127 changed=32 failed=0`. Lovely.

The next reboot sat at a GRUB screen.

Ubuntu's GRUB writes `recordfail=1` at every boot and clears it on a successful startup. With `/boot` on a now-read-only root, the clearing write failed silently, so GRUB assumed every boot was a recovery from a crash — and waited 30 seconds for a keypress that would never come on a headless box.

The fix is not to clear the flag. It comes straight back; the disk is still read-only, by design. The fix is to make the flag *inert*: `GRUB_RECORDFAIL_TIMEOUT=0`. When the symptom is unfixable by construction, fix the consequence instead.

Then the *next* Ansible run failed before executing a single task. Ansible's temp directory lives under `$HOME`, `$HOME` is on the root filesystem, and the root filesystem was now read-only. **The hardening had locked out the tool doing the hardening.** The playbook contains its own remount-rw window for exactly this situation — but fact-gathering happens before any task runs, so it never got the chance. Luke and Leia, the other two servers, already carried `remote_tmp = /tmp/.ansible` for precisely this reason. Cal was never brought onto the pattern, and the gap was harmless right up until it wasn't.

Both failures were caused by the same success.

## Four days of lying quietly

![Two server towers standing in a dark room, each casting a green glowing reflection on the floor while behind them their power cables lie unplugged](images/the-refusal-that-saved-the-filesystem-2.png)

Checking whether the other two servers shared Cal's GRUB exposure turned up something worse. Luke and Leia both had `/` mounted **read-write**, despite fstab asking for `ro` and the estate being documented as read-only at runtime. Both had been that way for at least four days. Nothing had noticed, because nothing was looking.

Luke's fstab was the tell:

```
/dev/disk/by-id/md-uuid-… / ext4 ro,ro,ro,ro,ro,ro,defaults 0 1
```

Six `ro`s. Not a typo — an accumulation. The playbook prepended `ro,` whenever it found root writable. A corrupted fstab makes `systemd-remount-fs` give up and fall back to `rw` at boot. So the next run found root writable again and prepended another `ro`. Six rounds, six `ro`s, each one making the next boot more certain to fail. A small masterpiece of self-perpetuation.

And it wasn't one bug. Three *independent* bugs were keeping those boxes writable, and every one of them reported success:

1. **The playbook preserved whatever state it found.** The task that restores read-only was gated `when: not root_is_rw` — it only restored `ro` if it had *found* `ro`. On an already-writable box it skipped the restore entirely and reported `ok=192 changed=24 failed=0`. This is why running the playbook never fixed anything.
2. **The daily security-update job had the identical shape.** `was_ro=false; restore_ro() { $was_ro || return 0; }` — so once root drifted writable, the job faithfully preserved that state every single morning, and exited 0.
3. **The fstab flip wasn't idempotent** — the prepending bug above. Leia had been fixed months earlier; nobody ported the fix to Luke. Or to Cal, where it sat harmless purely because Cal had only ever flipped once.

The detail that makes the whole thing worth writing about: the playbook contained a task called **"Warn loudly if root could not be returned to read-only"** — written specifically for this situation — gated on the same `not root_is_rw` condition. It could never fire in the one circumstance it existed for. A smoke alarm wired to the light switch of the room it was watching.

## Ask what the state should be, not what it is

The fix, in every case, is the same idea: stop asking *what state is this machine in*, and start asking *what state should it be in*. Intent belongs in fstab. The live mount state is the thing being corrected, so it cannot also be the thing consulted to decide whether to correct it.

There was even a red herring arguing the machines were fine: `chronyd` logged `Read-only file system` hourly on both hosts, which reads like proof that root was mounted `ro`. It wasn't. chrony runs under `ProtectSystem=strict` and sees its own private read-only view of the system. Corroborating evidence that corroborates nothing.

And one detail I'm glad was caught while the new partition was still empty: the 380 GB boundary worked out at sector 742,187,500 — and 742187500 mod 8 is 4. Half a physical sector out on a 512e SSD, which would have made every 4 KiB write a read-modify-write for the life of the machine. Free to fix before data landed; destructive afterwards. The runbook is now sector-based, with a mandatory byte-comparison gate.

## Seven green lights

Across one session, a green signal measured the mechanism rather than the outcome seven times:

- `parted` aborting was the *good* result, and it looked like a failure
- the playbook exited 0 while the machine sat unbootable at a GRUB prompt
- the read-only flip was validated by checking services still ran — nothing checked the management path still worked
- a drained queue proves messages moved, not that anything was embedded; only a *semantic* search proves the vectors landed, because a fulltext search would pass with no embeddings at all
- `failed=0` on a run that had silently skipped the one task that mattered
- `findmnt -no OPTIONS /` reports the VFS flag, which can disagree with the superblock — a root-owned `touch` is the only honest test
- a task-completion notification never arrives if the process hangs; about 90 minutes went on waiting for a signal that could not come, while the log file's mtime sat frozen in plain sight

Each one was caught by checking one level below the claim. None would have been caught by trusting an exit code.

The machine that saved the data was the one that refused to proceed. Everything that said yes was lying.

---

*Martin Harris connects cloud infrastructure to the real world. If your infrastructure has ever reported `failed=0` from a machine that couldn't boot, you're in the right place.*


{{donate}}