The Answer Was a Hole in the Ceiling
I run a Nexus repository on my home network. The idea is straightforward and, I thought, obviously correct: my laptop pulls npm packages, Docker layers and Maven artefacts dozens of times a day, and most of those pulls are for things it has already downloaded before. Put a pull-through cache on the LAN, and the first fetch comes from the internet while every subsequent one comes from a box in the loft at local network speed.
Except it never felt fast. It felt, if anything, slower than just going to the internet.
I had a theory, and it was a good one. The server hosting Nexus stores everything on a four-disk RAID-6 array with LUKS encryption underneath. I had measured fsync performance on that array at around five operations per second. A bare disk manages about a hundred. Parity is expensive: every small synchronous write means reading the old data, reading both parity blocks, recomputing, and writing it all back. Encryption on top of that, on a CPU from 2011.
Five per second is dreadful. It explained everything. Obviously.
The plan: dm-cache
The fix for slow parity writes is well established. dm-cache has been in the kernel since 3.9, and in writeback mode it lets you put a small fast device in front of a large slow one. Writes land on the fast tier and are acknowledged immediately; a kernel thread drains them to the slow origin in the background. The filesystem never knows. LVM drives the whole thing:
lvconvert --type cache --cachemode writeback \
--cachepool fast_lv vg/slow_lv
The theory is lovely. Keep the parity-protected capacity, get mirror-class write latency. My plan was to carve a small RAID-1 out of the same spinning disks, front the RAID-6 with it, and measure the improvement.
There was a wrinkle. Every disk in that box is fully partitioned — zero free sectors. A cache tier needed a new partition, which meant shrinking an existing one, which meant a five-layer unwind: filesystem, logical volume, physical volume, the LUKS container, then the array itself. On a live box. Holding data I'd rather keep.
I spent a while working out how to do that without destroying anything, and I'll come back to it, because it's the most genuinely useful thing in this post. But first I should explain why none of it mattered.
Rebuilding an array three times to change one variable
Before doing surgery on the live machine I wanted a number: how much does removing parity actually buy?
The other server had something convenient — a 3.4 TB RAID-6 array that was completely empty. No logical volumes, no fstab entry, not even mounted. Three and a half terabytes of nothing, which turns out to be a superb laboratory.
Because it was empty, I could stop the array and recreate it at a different RAID level using the same partitions:
mdadm --stop /dev/md3
mdadm --create /dev/md3 --run --level=10 --raid-devices=4 \
--chunk=512 --assume-clean \
--uuid=d80493b6:0148289f:579e0939:f25ced99 \
/dev/sd[abcd]5
Two flags there are doing real work. --assume-clean skips the initial resync, which would otherwise spend hours computing parity for data that doesn't exist. And --uuid= forces the array to keep its original identifier — because recreating an array normally mints a new UUID, which would leave /etc/mdadm/mdadm.conf pointing at an array that no longer exists, and the box wouldn't assemble it at boot. Forcing the old UUID meant I could rebuild the array repeatedly without touching a single config file.
Same disks, same partitions, same LUKS, same CPU. Only the RAID level changed. That's the only way the numbers mean anything.
I ran an identical fio job at each level — 64K random writes with periodic fsync, which is roughly the shape of what Time Machine does to a sparse bundle:
numjobs=1 numjobs=2 latency
RAID-5 x4 35 IOPS 2274 KiB/s 70 IOPS 4524 KiB/s 89.6 ms
RAID-6 x4 44 IOPS 2830 KiB/s 64 IOPS 4146 KiB/s 97.5 ms
RAID-10 x4 68 IOPS 4390 KiB/s 108 IOPS 6946 KiB/s 55.7 ms
RAID-1 x3 — 117 IOPS 7489 KiB/s 50.5 ms
Removing parity is worth about 1.5× on throughput and halves the latency. Real, useful, and considerably less than I'd assumed.
That last point is worth dwelling on. I'd initially estimated the gain at 1.83×, using RAID-1 as a stand-in for RAID-10. Then I measured RAID-10 properly and got 1.69× against RAID-6. Then I remembered the array I actually cared about was RAID-5, not RAID-6, measured that too, and landed at 1.54×. Every time the measurement got more honest, the number got smaller. Proxies flatter.
Everything I eliminated
Armed with a real number, I went back to the actual complaint: why are package pulls slow?
I timed a 41 MB npm tarball from Nexus and got 23 MB/s. Then I timed the same tarball from the public registry, over the internet, and got 34 MB/s.
The internet was beating my LAN. That's absurd, and it should have told me something immediately. It did not.
Instead I started working through the stack, and this is where the post becomes a list of things that were not the problem.
It wasn't the disks. Reading the same file directly on the server, bypassing the network entirely, gave 84–116 MB/s. It was being served out of page cache; the array was barely involved.
It wasn't the CPU. During a transfer, nginx sat at 25% and the Java process at 8%, on a two-core box. Nothing was saturated.
It wasn't TLS. This one is my favourite, because the measurement is so counter-intuitive. The CPU in that machine has no AES-NI, so all encryption is done in software:
AES-256-GCM 27.7 MB/s
ChaCha20-Poly1305 172.5 MB/s
A 6.2× difference — which is exactly why ChaCha20 exists, and exactly what clients already negotiate. So I forced the slow cipher, expecting transfers to crawl.
Nothing happened. Identical speed. If deliberately using a cipher six times slower makes no difference, encryption has headroom to spare and was never the constraint.
It wasn't nginx buffering. The vhost had proxy_buffering off, which is right for a video stream and wrong for bulk file downloads. I was confident. I changed it, deployed it, measured before and after: 23.0 MB/s versus 24.8 MB/s. Two overlapping clouds of noise.
It wasn't Docker's userland proxy. Docker publishes ports via a Go process that can copy every byte through userspace. On a slow CPU that's a genuine tax. It also never showed up in CPU sampling, because modern Docker handles the data path in-kernel.
It wasn't the cable, and it wasn't the network card. I checked, because by this point I was checking everything:
Speed: 1000Mb/s Duplex: Full Link detected: yes
RX errors: 0 crc: 0 frame: 0 fifo: 0 overrun: 0
TX errors: 0 dropped: 0 carrier: 0 collisions: 0
Zero CRC errors across twenty-five million packets. A dodgy cable produces those by the thousand.
The measurement I should have taken first
Somewhere in the middle of all this I did something almost by accident: I copied a 600 MB file off the server over SMB, as a sanity check.
49.6 MB/s.
Same machine. Same network. Same encrypted RAID-6 array. Twice the speed of the HTTP path, reading from the very storage I'd spent two days blaming.
That should have ended it there, and nearly did. But the interesting part came from looking at the actual TCP connection. The service runs in a container, which matters: the socket lives in the container's network namespace, not the host's, so ss on the host shows you nothing at all. You have to go in after it:
PID=$(docker inspect -f '{{.State.Pid}}' gateway-nginx-1)
nsenter -t $PID -n ss -tin
And there it was:
rtt:29.685/1.595 cwnd:527 bytes_retrans:2073298 retrans:0/1466
1,466 retransmissions. Two megabytes of a 41 MB transfer sent twice. About 5% of the data. And the round-trip time climbing from 8 milliseconds at idle to nearly 30 under load — queues filling somewhere, latency tripling, packets dropping.
TCP responds to loss by halving its congestion window. At 5% loss it spends its entire life backing off and cautiously ramping up again. It never gets going.
The topology
My servers are plugged into a 10 gigabit switch. I was rather proud of that switch.
The switch is wired to a mesh WiFi node in the loft. That node reaches the rest of the house wirelessly, to a second node downstairs, which is wired to the main one, which my laptop connects to over WiFi.
So the path from my laptop to my servers crosses two wireless hops. The path from my laptop to the internet crosses one.
That is the whole story. The internet was faster than my LAN because it was, in the most literal sense, closer. The 10 gigabit switch was feeding a wireless bottleneck, doing nothing whatsoever except making me feel good about my rack.
It also explains something that had been nagging me. Early on I'd tried four concurrent downloads, reasoning that if a single stream were window-limited, parallel streams should scale. Four connections gave 19 MB/s in aggregate — worse than one connection's 21. They weren't limited individually. They were fighting each other for one congested link.
The bit that was actually useful
The one piece of genuine engineering I'd salvage from all this is the shrink procedure I worked out and then never needed.
If you ever have to make room for a new partition on a fully allocated disk, the layers unwind from the top down, each left slightly larger than the one above it:
pvresize --setphysicalvolumesize 5400G /dev/mapper/vault # LVM
cryptsetup resize vault --size ... # LUKS
mdadm --grow /dev/md4 --array-size ... # exposed size
# ↳ verify everything still mounts and fscks clean
mdadm --grow /dev/md4 --size ... # commit
sgdisk ... # partitions
The safety valve is --array-size. It changes only the size the array advertises, touches no data, and is instantly reversible. So you shrink what the array claims to be, confirm every filesystem still mounts and passes a check, and only then commit the real component shrink. If anything looks wrong you set it back and you've lost nothing.
And if the free space isn't conveniently at the end of the volume, pvmove will relocate physical extents while the filesystem is mounted and in use, mirroring each extent before switching over, with pvmove --abort to roll back. It's the tool people use to evacuate a dying disk from a volume group without downtime, and it's remarkably calm about it.
I know all of this works. I have tested it thoroughly. I have not used it for anything.
What I'd do differently
Measure the network before blaming the machine. The SMB read that broke the case open took thirty seconds and I ran it on day two. Everything before it was elaborate, well-instrumented investigation of the wrong subsystem.
Change one variable. Two of my wrong answers came from comparisons that moved more than one thing at once. The worst was measuring a crude Python relay without TLS against nginx with TLS, and confidently attributing the entire difference to TLS. It was neither the relay nor TLS. I'd compared two unrelated things and drawn a conclusion about a third.
Distrust a proxy measurement. RAID-1 standing in for RAID-10, RAID-6 standing in for RAID-5 — each substitution was reasonable and each one inflated the answer. 1.83 became 1.69 became 1.54.
A plausible cause is not a measured cause. I asserted four different culprits with real confidence — parity, TLS, nginx buffering, Docker's proxy — and disproved all four. Being able to explain how something would cause the symptom is not evidence that it does.
Three months ago I wrote about building a Raspberry Pi NAS, and that post ended with me plugging in an ethernet cable after a lengthy tour of the Linux storage stack. This one ends with me arranging to drill a hole through a ceiling to run a cable to the loft, after a lengthy tour of the Linux storage stack.
I would like to report that I learned something the first time.
The storage layers were, once again, entirely innocent. LUKS encrypted at line rate. LVM did exactly what it was told. mdadm rebuilt an array from scratch three times without complaint and handed back identical numbers each time. Every abstraction behaved impeccably.
The problem was that some of the bits had to travel through the air.
Martin Harris writes about connecting real hardware to abstract cloud infrastructure. If you have ever benchmarked four RAID levels to discover the answer was a hole in the ceiling, you have arrived at the right blog.
If you find this blog useful, please consider a donation.