The Disk Is the Resume File
I'm building a NAS. Specifically, a RAID-5 array out of secondhand 6TB drives, on a server called Leia, in my spare room. Before any drive goes into a RAID, I run a full destructive badblocks test — four passes, writing 0xaa, 0x55, 0xff, and 0x00 across the entire surface. After each pattern is written, badblocks -w reads it back to verify before moving on to the next. On a 6TB drive at 150 MB/s, that takes the better part of four days.
Four days is a long time for nothing to go wrong.
The problem with badblocks
badblocks -w is a blunt instrument. It writes, it reads, it tells you if anything went wrong. What it doesn't do is save progress. If the test is interrupted — power cut, reboot, Ctrl-C at the wrong moment — the default answer is to start again from the beginning.
Four more days. Sorry.
Warning:
badblocks -wis a destructive test. Triple-check your device path — this will erase all data on the target device.
badblocks doesn't support resuming in any mode. You can get a progress meter (-s) or report progress to another process (-C), and you can adjust the per-pass chunk size (-c), but there's no built-in checkpoint file or resume flag — not for read-only tests, not for write tests.
So I wrote a resume mechanism. The interesting part isn't the shell script that orchestrates everything — it's how the resume position is detected when there's no checkpoint file.
The insight
A badblocks write test is a state machine with a very specific property: at any given moment during the write phase, the disk encodes exactly what's happened to it. The beginning of the disk (LBA 0) has the current pattern. The end (last LBA) has the previous pattern (or the original data, if you're on pass one). The boundary between them is where the test stopped.
The patterns, in order, are 0xaa, 0x55, 0xff, 0x00. 0xaa and 0x55 exercise adjacent-bit transitions; 0xff and 0x00 catch stuck-at-one and stuck-at-zero faults. Each byte value covers the whole drive before the next one starts, then is read back to verify. If you read the first block and it's all 0x55, and the last block is all 0xaa, you know you're partway through writing pass 2. If first and last are both 0xff, you know the write is done and you're in (or about to start) the read-verify phase for pass 3.
The disk is the resume file. You just have to read it.
Reading it efficiently
The naive approach — scan from block zero until the pattern changes — would take hours on a 6TB drive. There's a better way: binary search.
If block zero is 0x55 and block N is 0xaa, then somewhere between them the pattern changes. Check the midpoint. If it's 0x55, the boundary is in the upper half. If it's 0xaa, it's in the lower half. Repeat until you've narrowed it down to within one block. A 6TB drive contains roughly 1.5 billion 4 KiB blocks (6×10¹²/4096; the exact LBA count depends on the drive), but binary search finds the boundary in about 31 reads (log₂N) regardless.
Each of those reads is a single 4 KiB seek-and-read. Even on a spinning disk, ~31 such seeks complete in well under a second — the detection overhead is negligible next to the days-long test it's resuming.
The remaining problem is latency. Running 31 individual SSH commands — one per binary search iteration — takes roughly as long as a round-trip per command. Even at low latency, that's painful and unnecessary. Instead, the detection runs entirely on the remote host in a single SSH session, via a Python heredoc.
Before invoking it, the script detects the logical block size and total byte count on the remote host:
BLOCK_SIZE=$(ssh "$HOST" "sudo blockdev --getss '$DEVICE'")
DISK_BYTES=$(ssh "$HOST" "sudo blockdev --getsize64 '$DEVICE'")
Using blockdev --getss matters because some drives present 512-byte logical sectors (512e). The detect() function and boundary search both use whatever size the drive reports; the binary search operates in units of that block size. If the boundary falls inside a block, resuming from the previous block boundary will re-write a small already-written region — that's safe, and it's why detecting at the device's block granularity is acceptable even if badblocks -b differs. At a genuine pattern boundary you may see a mixed block — detect() returns None for non-uniform data, and the search treats that as "not yet written with target pattern," which keeps it on the correct side.
One thing has to be got right when this offset is actually handed to badblocks: it's a block index in units of the device's logical sector size, not badblocks' own block size, which defaults to 1024 bytes unless you pass -b. Feed the raw offset to badblocks without accounting for that and, on a 512e drive where getss reports 512, you'd start at half the intended offset — too early on a lucky day, wasting time; too late on an unlucky one, silently skipping unwritten blocks. The fix is to keep the units aligned end to end: always invoke badblocks with -b "$BLOCK_SIZE", so its block size matches the one the detector used. The invocation later in this post does exactly that.
The main detection script is sent in one SSH session and returns a four-field string — label, pass index, phase, and block offset — which the calling script uses to construct the appropriate badblocks invocation:
RESULT=$(ssh "$HOST" "sudo python3 - '$DEVICE' '$DISK_BYTES' '$BLOCK_SIZE'" << 'PYEOF'
import sys, os
device = sys.argv[1]
disk_bytes = int(sys.argv[2])
block_size = int(sys.argv[3])
patterns = [0xaa, 0x55, 0xff, 0x00]
total_blocks = disk_bytes // block_size
def read_block(f, index):
f.seek(index * block_size)
return f.read(block_size)
def detect(data):
"""Return the uniform byte value if every byte is identical, else None."""
if not data:
return None
unique = set(data)
return unique.pop() if len(unique) == 1 else None
def binary_search(f, lo, hi, target_val):
"""Find the first block >= lo whose value is not target_val."""
while hi - lo > 1:
mid = (lo + hi) // 2
mid_val = detect(read_block(f, mid))
if mid_val == target_val:
lo = mid
else:
hi = mid
return hi
def verify_prefix(f, hi, target_val):
"""Confirm every block before hi already holds target_val. Returns the
index of the first block that doesn't, or None if the prefix is clean.
This is the read-verify that badblocks' own resumed write pass will
never perform for this region, since it starts writing at hi."""
for index in range(hi):
if detect(read_block(f, index)) != target_val:
return index
return None
with open(device, 'rb', buffering=0) as f:
first_val = detect(read_block(f, 0))
last_val = detect(read_block(f, total_blocks - 1))
if first_val == last_val and first_val in patterns:
# Both ends are the same test pattern — disk is uniformly filled.
# This is physically indistinguishable from:
# (a) a write pass just completed and verify is about to begin, or
# (b) the disk was already in this state before the test started
# (e.g. previously zeroed or run through a full test).
# Without a trusted external state file from this same run, we cannot
# safely assume earlier passes completed. Emit AMBIGUOUS and let the
# caller restart from pass 0 — the pass index is meaningless here,
# so it's emitted as a placeholder rather than a real value.
print("AMBIGUOUS - write 0")
elif first_val in patterns:
# Write is in progress: LBA 0 has the new pattern, end has old.
pass_index = patterns.index(first_val)
boundary = binary_search(f, 0, total_blocks - 1, first_val)
# badblocks' own read-verify pass for this resumed range will only
# cover [boundary, end) — it never reads back [0, boundary), which
# was written before the interruption but not yet verified. Verify
# it here instead, at the same I/O cost badblocks would have paid.
bad_block = verify_prefix(f, boundary, first_val)
if bad_block is not None:
print(f"FAULT {pass_index} verify {bad_block}")
else:
print(f"WRITE {pass_index} write {boundary}")
else:
# LBA 0 does not contain a known test pattern — disk looks like
# original data. Start from scratch.
print("START 0 write 0")
PYEOF
)
One SSH connection. All the reads happen over the drive's own SATA cable, not your network. The quoted heredoc (<< 'PYEOF') stops bash expanding anything inside it before it's sent — a habit worth keeping any time embedded Python contains $, backticks, or brace-expansion syntax bash might otherwise try to interpret. This particular script sidesteps that risk anyway, by passing every value through sys.argv instead of interpolating it into the heredoc, but the quoting is what makes leaving it that way safe.
The output is a four-field string: state label, pass index (0-based: 0→0xaa, 1→0x55, 2→0xff, 3→0x00; - if not applicable), phase (write to resume a write pass, verify if the prefix check below found a fault, and never a mid-verify resume — see below), and block offset. For example:
WRITE 1 write 734003200
FAULT 1 verify 512339200
AMBIGUOUS - write 0
START 0 write 0
Why the detector never tries to resume mid-verify rather than mid-write: once a pattern's write phase finishes, the whole disk is uniform in that pattern whether or not badblocks has started reading it back — an interruption mid-verify is physically indistinguishable from "write just completed," so it always falls into the AMBIGUOUS, restart-from-pass-0 path already described above. There's no separate mid-verify case to detect because the disk itself doesn't record one.
Putting the four fields to use, the calling script constructs the actual badblocks invocation, keeping the block size explicit throughout so the units the detector used and the units badblocks uses never diverge:
read -r STATE PASS PHASE BLOCK <<< "$RESULT"
PATTERNS=(0xaa 0x55 0xff 0x00)
LAST_BLOCK=$(( DISK_BYTES / BLOCK_SIZE - 1 ))
case "$STATE" in
WRITE)
# Resume the interrupted pattern from the boundary, then run any
# remaining patterns over the whole disk — a start block only makes
# sense for the pass that was actually cut short.
ssh "$HOST" "sudo badblocks -w -b '$BLOCK_SIZE' -t '${PATTERNS[$PASS]}' \
'$DEVICE' '$LAST_BLOCK' '$BLOCK'"
if (( PASS + 1 < ${#PATTERNS[@]} )); then
REMAINING="${PATTERNS[*]:$((PASS + 1))}"
ssh "$HOST" "sudo badblocks -w -b '$BLOCK_SIZE' -t '${REMAINING// /,}' \
'$DEVICE' '$LAST_BLOCK'"
fi
;;
FAULT)
echo "Bad block at offset $BLOCK (pattern index $PASS) — inspect before resuming." >&2
exit 1
;;
AMBIGUOUS|START)
ssh "$HOST" "sudo badblocks -w -b '$BLOCK_SIZE' -t 0xaa,0x55,0xff,0x00 '$DEVICE'"
;;
esac
The edge cases
The detection logic handles four situations:
Write in progress. LBA 0 has pattern N, last LBA has pattern N-1 (or original data for the first pass). Binary search finds the boundary, then a linear scan verifies that everything before the boundary genuinely holds pattern N — the read-verify that the resumed badblocks write will never perform for that region, since it only reads back from the boundary onward. Only once that prefix checks out does the script resume by writing pattern N from the boundary onwards and chaining the remaining passes.
Prefix verification fault. The scan above finds a block before the boundary that doesn't hold the expected pattern. That's a genuine defect surfaced earlier than it otherwise would have been — the script reports the offset and stops rather than silently resuming, which would have buried the fault under the next pass's write.
Ambiguous uniform disk. If LBA 0 and last LBA are both the same test pattern, the disk is uniformly filled with that pattern. This is physically indistinguishable from two different situations: a write pass just completed and badblocks is about to start the read-verify phase; or the disk was already in that state before the test started (previously zeroed, or run through a full test). It also means that if the disk happened to be uniformly 0x00 or 0xff before the test began, content alone cannot tell you whether any passes have actually run.
Without a trusted external state file from this same test run, the only safe choice is to restart from pass 0 and redo all four passes. That can cost up to four full write passes in the worst case, but it guarantees full coverage. Emitting a specific pass index alongside AMBIGUOUS and letting the caller silently skip to it would be incorrect — it would under-test the drive — so the pass-index field is emitted as a placeholder (-) instead: there is nothing meaningful to skip to. When the caller sees AMBIGUOUS, it restarts from the beginning.
Reading /proc/diskstats can help narrow things down: if the sectors-read counter is plausibly ahead of the sectors-written counter for this pass (same boot, same device node, no multipath or device renumbering), it's reasonable to infer that the verify is in progress. But the counters reset on reboot, and cache or read-merge effects make exact offsets fuzzy. Rather than trust uncertain external state, the script takes the conservative path.
Not started. LBA 0 does not contain a known test pattern — disk contents look like original data. Start from scratch.
Each case produces the same output format: which pass we're on, whether the phase to resume is a write or a fault-reporting verify, and which block to resume from (or inspect). The badblocks invocation — or the fault report — is then constructed from those values.
What I actually learned
The useful thing isn't the binary search itself — that's textbook. It's the observation that a write test inherently encodes its own state: nothing needs to be saved externally, because the disk is writing its own checkpoint with every block.
That only works because the patterns are chosen to be uniform and distinct — 0xaa, 0x55, 0xff, 0x00 each cover a different bit pattern and are instantly recognisable from a single block read. It also means the resume logic has to be honest about exactly what a partial write pass has and hasn't verified, rather than assuming the boundary alone is enough.
It also means that when a drive's test is interrupted mid-session, the drive itself still knows exactly where it's been. Four days of work, not wasted.
Martin Harris writes about the part of the stack you can trip over — servers, drives, networks, and the software that sits just above the hardware.
If you find this blog useful, please consider a donation.