Special vdev: 113 TB hanging on two files
The special vdev of a 113 TB pool sat on two qcow2 files — both on the same consumer SSD, with 11% of its endurance left. Moving it meant operating on a live mirror with no room to be wrong. And it surfaced two checks that had been pointed at the wrong thing the whole time.
A ZFS special vdev carries a pool’s metadata: the block pointers, the directory structure, the map of where everything is. You do not have to create one. Once you have, it stops being optional. Lose a special vdev and you do not lose metadata. You lose the pool.
Ours held the metadata of a 113 TB raidz2 pool. Practically everything we run sits on that pool: every SMB share, the share every Swarm service mounts, the photo library, the GitLab runner cache, and the target every backup is written to.
It consisted of two qcow2 files on a consumer SSD with 11% of its endurance left.
Not on the array, then, but on a single Samsung 990 PRO at 89% wear — the same drive the hypervisor booted from. Both halves of the mirror were on the same piece of hardware. A mirror across a single drive is redundancy in name only.
On top of that, a circle we built ourselves: our backups land on a share — and that share comes from the very machine this is about. Which left the special vdev without a single copy outside the box that was carrying it.
And while all that sat there, the same machine held four healthy NVMe drives at 0–1% wear. Passed through to that very guest, for a pool that had not existed for months. The hardware that fixes the problem had been in the chassis the whole time, doing nothing.
Attach before detach
The obvious move would have been to leave the four NVMe with the guest and build a new pool there. Against it stood something we had learned five days earlier: this guest serves storage to its own hypervisor. When it stalls, the layer waiting on it is the one that could bring it back. Passthrough would have moved the redundancy of our most critical vdev entirely inside that guest.
So the drives went to the hypervisor. One pool, zpool_nvme, two 400 GB zvols carved out of it and handed to the guest as ordinary SCSI disks. The media underneath the metadata now belong to the layer that survives a guest failure.
pool: zpool_nvme
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zpool_nvme ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-Samsung_SSD_990_PRO_2TB_…4475V ONLINE 0 0 0
nvme-Samsung_SSD_990_PRO_2TB_…4352W ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
nvme-Samsung_SSD_990_PRO_2TB_…4463B ONLINE 0 0 0
nvme-Samsung_SSD_990_PRO_2TB_…4490Z ONLINE 0 0 0
errors: No known data errorsThe part worth copying is above all the order. ZFS has no command that replaces a special vdev. It has zpool attach and zpool detach, and the only question is which of the two you run first.
Detach first and you spend a few minutes without a second half — on exactly the device whose failure costs 113 TB. Attach first and you spend those same minutes better off than before. So we went from a 2-way to a 3-way to a 4-way mirror and let roughly 124 GB resilver until zero errors were reported. Then we cross-checked the widened mirror in the TrueNAS UI as well — and only then detached the two qcow2 halves.
Redundancy never fell below where it stood at the start. Not even during the minutes nobody had planned for.
That costs nothing but patience, and it changes the nature of the delicate part. A window you have to survive becomes a state you can sit in for as long as you like. After the fourth attach we could have stopped, gone to bed and continued a week later. The point of no return was a single command, issued once all the evidence was on the table.
The old volumes we then deleted rather than left lying around. A detached special half goes stale the moment the pool carries on without it, and can never safely be reattached afterwards — but it very much can be attached. A device that is useless and still reachable is a loaded gun pointed at 113 TB. As a side effect 288 GB came back, and the pool on the boot drive dropped from 39% to 8%. One of the four volumes had no entry in the configuration at all: an earlier single-disk attempt had removed the reference and left the data behind.
qm set moves nothing at all
We describe our VMs’ disks in Ansible, including which storage each disk is supposed to live on. The obvious next step would be to let that declaration converge: if a disk is in the wrong place, Ansible moves it.
That is precisely what we deliberately did not build.
qm set 4110 --scsi0 zpool_nvme:vm-4110-disk-0 looks as though it relocates a disk. It does not. It swaps out the entire drive entry in the VM’s configuration and then points at a volume that was never created on the target. The data stays where it is. The only thing that moves it is qm move-disk, a mirror performed while the machine runs.
So storage is only declared and reported, never applied. The evaluation skips the field before the code comes anywhere near a drive entry, and it never marks a machine as changed — which means a converging run cannot even select a VM on its account. When the declaration disagrees with reality, the audit reports the drift and writes out the qm move-disk command for a human to run.
This is not a gap we intend to close at some point. Some declarations are worth stating and not worth automating, and the most honest way to record that is to make the automation incapable of acting — rather than relying on everyone remembering. Unit tests pin it down, and an integration test asserts that no qm set comes out even with apply mode explicitly on.
One detail we got right by accident first and then kept on purpose: the declaration names the storage, not the name of the volume. Proxmox renumbers on every move — our scsi0 came out as disk-4. Which drive the data is on is worth stating. What the file ends up being called is not.
The check was there. It was pointed at the wrong thing.
Two things went wrong during the maintenance window, and both had the same shape.
The first: we created two PCI mappings through the Proxmox API and verified they were there. They were. qm start refused the VM anyway — missing expected property 'iommugroup'. The endpoint that creates a mapping does not validate it. The validation sits in a function that is only called when a VM starts. The distance between “created” and “usable” therefore stays invisible until the worst conceivable moment. Our own comment in the code claimed that going through the API bought us validation. It does not.
The second: four of five disks moved to the new pool. scsi0 stayed behind on the SSD at 89% wear, and the nightly audit said nothing about it. That was not a bug in the code. A disk nobody declares a storage for is compared against nothing — so a half-finished migration looks like a finished one. The audit answered the question we asked it. The question we had asked was the wrong one.
Checking that something exists is not checking that it is valid. And a check that compares what was declared against what is there cannot report what you forgot to declare.
In both cases a test had frozen the misconception along with everything else. The suite was green on output that Proxmox rejects. That is the point worth taking away: a test does not check whether you have understood a system. It records how you understood it. If that understanding is wrong, the test is the thing defending the bug.
Both are now closed structurally rather than merely noted. iommugroup is mandatory at declaration time, enforced where the mapping is built instead of surfacing at the next VM start. And whether a machine has to be fully declared can be switched on individually: on this guest, every attached disk without an entry fails loudly. On this one and no other — the switch is only worth its noise where enough is at stake if it goes wrong.
What the numbers actually say
We measured immediately after the migration, under live production load, on a dataset that bypasses the cache and was destroyed afterwards.
| profile | IOPS | throughput | latency |
|---|---|---|---|
| 4k sync randwrite, QD1 | 637 | 2.5 MiB/s | avg 1.55 ms, p99 3.9 ms, p99.9 11.6 ms |
| 16k randwrite, 4 jobs QD16 | 60,800 | 950 MiB/s | avg 64 µs, p99 734 µs |
| 1M sequential write | 1,507 | 1,507 MiB/s | — |
The comparison everyone wants is the one with the 688 ms flush latency we had measured on the SATA storage this replaces. The first profile is the comparable shape, and 1.55 ms against 688 ms is a number you enjoy writing down.
It is still not a ratio, and we are not selling it as one. The 688 ms came from a real CI workload under real concurrency; this is synthetic fio on a single thread. The two figures agree in sign and order of magnitude. They do not agree on a number — divide one by the other and you have written a press release, not a measurement.
637 IOPS is on the low side for NVMe, and the reason is right there in the configuration: this pool has no log device of its own, so 4k sync writes land in the in-pool intent log. Worth knowing before someone reads the first row as a verdict on the hardware.
What is still open
The issue this work grew out of set out to establish two independent failure domains for the metadata mirror. It did not fully get there — and that belongs in the article rather than in a ticket.
zpool_nvme consists of two mirrored pairs, written across in stripes. Both 400 GB zvols therefore lie across both pairs. If either pair fails, both halves of the special vdev are gone, and with them the 113 TB pool. That is vastly better than two files on a dying SSD. It is not the property we set out to establish.
On top of that: all four drives hang behind a single PCIe switch, and on this hardware there is no topology that changes it. The hypervisor’s root filesystem is still on the worn SSD that triggered this whole exercise — we moved the metadata off it, not the operating system. And the new pool now carries this guest’s metadata and, in the medium term, further workloads; what used to be an isolated concern has become a shared one. The benchmark itself drove the host’s load average from 16.7 to 40.1 and settled again within two minutes.
None of that argues for having waited longer. It is the difference between a migration that is finished and one that is done — and only one of those two words means you get to stop thinking about it.
What I would keep
- Always attach before you detach. If a swap can be phrased as “widen first, then narrow”, you no longer have to survive the delicate part. You are allowed to sit in it for as long as you like.
- What has been replaced should be deleted. A stale mirror half that can still be attached is not a spare. It is a mistake waiting for a tired human.
- Dangerous automation belongs structurally prevented, not switched off. A switch that can be flipped is one somebody eventually flips. Code that cannot reach the operation is a different kind of promise.
- Ask whether your own check can see absence. Most checks compare what was declared against what is there. Hardly any notice what was never declared — and that is exactly where half-finished work hides.
- Publish the limit you did not reach. The sentence about the striped mirrors is the most useful one in this article, and it describes the thing we did not manage.
Architecture & Governance Lead
Squibble GmbH
Has spent twenty years bringing structure to IT landscapes that grew rather than were designed — as architect, developer, and operator. Writes here about the systems actually running at Squibble and the decisions behind them.
