The Refusal That Saved the Filesystem#

Cal is a £100 iMac from 2011 that spends its retirement computing embeddings for my home estate. It needed its single 480 GB partition split in two, so the OS root could be made read-only and the data could live on its own partition. Shrink the filesystem, move the partition boundary, carve out the free space. A well-trodden procedure. I even wrote a runbook first.

The runbook said:

Shrink the FILESYSTEM to 360G (target is 371G — 11G of deliberate slack)

That sentence is wrong in a way that is almost invisible. resize2fs 360G means 360 GiB — 386.5 GB. parted … 380GB means 380 × 10⁹ bytes, which produces a 370.9 GB partition. So the plan, faithfully executed, would have left the filesystem 15.6 GB larger than the partition containing it. The "11 GB of deliberate slack" was in reality a 15.6 GB overlap. Two tools, both using the letter G, meaning different things by it.

The script ran. The filesystem shrank. And then parted printed:

Warning: Shrinking a partition can cause data loss, are you sure you want to continue?

— and, because it was running with -s and script mode cannot answer prompts, it aborted.

Not because it detected the overlap. parted has no idea about filesystems; it moves boundaries, that's all. It aborted because nobody was there to say yes. A safety property nobody designed, doing the job the arithmetic should have done.

The success that broke two things#

With the units fixed, the partition work finished properly. The playbook flipped the root filesystem to read-only — the entire point of the exercise — and reported success. ok=127 changed=32 failed=0. Lovely.

The next reboot sat at a GRUB screen.

Ubuntu's GRUB writes recordfail=1 at every boot and clears it on a successful startup. With /boot on a now-read-only root, the clearing write failed silently, so GRUB assumed every boot was a recovery from a crash — and waited 30 seconds for a keypress that would never come on a headless box.

The fix is not to clear the flag. It comes straight back; the disk is still read-only, by design. The fix is to make the flag inert: GRUB_RECORDFAIL_TIMEOUT=0. When the symptom is unfixable by construction, fix the consequence instead.

Then the next Ansible run failed before executing a single task. Ansible's temp directory lives under $HOME, $HOME is on the root filesystem, and the root filesystem was now read-only. The hardening had locked out the tool doing the hardening. The playbook contains its own remount-rw window for exactly this situation — but fact-gathering happens before any task runs, so it never got the chance. Luke and Leia, the other two servers, already carried remote_tmp = /tmp/.ansible for precisely this reason. Cal was never brought onto the pattern, and the gap was harmless right up until it wasn't.

Both failures were caused by the same success.

Four days of lying quietly#

Checking whether the other two servers shared Cal's GRUB exposure turned up something worse. Luke and Leia both had / mounted read-write, despite fstab asking for ro and the estate being documented as read-only at runtime. Both had been that way for at least four days. Nothing had noticed, because nothing was looking.

Luke's fstab was the tell:

/dev/disk/by-id/md-uuid-… / ext4 ro,ro,ro,ro,ro,ro,defaults 0 1

Six ros. Not a typo — an accumulation. The playbook prepended ro, whenever it found root writable. A corrupted fstab makes systemd-remount-fs give up and fall back to rw at boot. So the next run found root writable again and prepended another ro. Six rounds, six ros, each one making the next boot more certain to fail. A small masterpiece of self-perpetuation.

And it wasn't one bug. Three independent bugs were keeping those boxes writable, and every one of them reported success:

1. The playbook preserved whatever state it found. The task that restores read-only was gated when: not root_is_rw — it only restored ro if it had found ro. On an already-writable box it skipped the restore entirely and reported ok=192 changed=24 failed=0. This is why running the playbook never fixed anything.
2. The daily security-update job had the identical shape. was_ro=false; restore_ro() { $was_ro || return 0; } — so once root drifted writable, the job faithfully preserved that state every single morning, and exited 0.
3. The fstab flip wasn't idempotent — the prepending bug above. Leia had been fixed months earlier; nobody ported the fix to Luke. Or to Cal, where it sat harmless purely because Cal had only ever flipped once.

The detail that makes the whole thing worth writing about: the playbook contained a task called "Warn loudly if root could not be returned to read-only" — written specifically for this situation — gated on the same not root_is_rw condition. It could never fire in the one circumstance it existed for. A smoke alarm wired to the light switch of the room it was watching.

Ask what the state should be, not what it is#

The fix, in every case, is the same idea: stop asking what state is this machine in, and start asking what state should it be in. Intent belongs in fstab. The live mount state is the thing being corrected, so it cannot also be the thing consulted to decide whether to correct it.

There was even a red herring arguing the machines were fine: chronyd logged Read-only file system hourly on both hosts, which reads like proof that root was mounted ro. It wasn't. chrony runs under ProtectSystem=strict and sees its own private read-only view of the system. Corroborating evidence that corroborates nothing.

And one detail I'm glad was caught while the new partition was still empty: the 380 GB boundary worked out at sector 742,187,500 — and 742187500 mod 8 is 4. Half a physical sector out on a 512e SSD, which would have made every 4 KiB write a read-modify-write for the life of the machine. Free to fix before data landed; destructive afterwards. The runbook is now sector-based, with a mandatory byte-comparison gate.

Seven green lights#

Across one session, a green signal measured the mechanism rather than the outcome seven times:

• parted aborting was the good result, and it looked like a failure
• the playbook exited 0 while the machine sat unbootable at a GRUB prompt
• the read-only flip was validated by checking services still ran — nothing checked the management path still worked
• a drained queue proves messages moved, not that anything was embedded; only a semantic search proves the vectors landed, because a fulltext search would pass with no embeddings at all
• failed=0 on a run that had silently skipped the one task that mattered
• findmnt -no OPTIONS / reports the VFS flag, which can disagree with the superblock — a root-owned touch is the only honest test
• a task-completion notification never arrives if the process hangs; about 90 minutes went on waiting for a signal that could not come, while the log file's mtime sat frozen in plain sight

Each one was caught by checking one level below the claim. None would have been caught by trusting an exit code.

The machine that saved the data was the one that refused to proceed. Everything that said yes was lying.

---

Martin Harris connects cloud infrastructure to the real world. If your infrastructure has ever reported failed=0 from a machine that couldn't boot, you're in the right place.

---

If you find this blog useful, please consider a donation.