2.9 KiB
2026-09-06 — pfi-pve NASPool rebuilt as six-wide raidz2 (backplane fault, bays 9/10 dead)
What happened. NASPool (TrueNAS-legacy, 4×mirror on 8 disks) had run two mirrors at single-disk redundancy since ~2026-04-23. Root cause by elimination: R750xs backplane — PERC PHYs 9 and 10 show no link at all (SAS address zero, no error counters); cables replaced by the operator and the PERC replaced 2026-09-05 (serial 3260688 → 29R1754) with no change. Old PERC is a spare, not scrap. Drives in bays 9/10 are healthy but invisible.
Route taken (operator, 2026-09-05 22:59Z): "Route C hybrid". Park everything on ospool
(which holds EVERY guest boot disk and was never touched by an irreversible command), destroy,
recreate NASPool as raidz2 over the six linked disks, copy back. Bays 9/10 = labelled cold spares
(pull a failed disk, move a spare into its bay, zpool replace). Run on one PSU (PSU1 dead since
08-26) — accepted because a power cut mid-copy costs a re-run, not data.
Result. 23:04Z quiesce → 01:20Z parked 1.65T (~200 MB/s, zero source read errors — the copy doubled as the readability test of the two sole-survivor disks; no pre-scrub) → 01:46Z byte-verified → 02:00Z operator GO → 02:01Z destroy + raidz2 create (by-id, ashift=12, lz4, recordsize=1M on backupStore/pve-VMStorage) → 03:53Z restored → 04:03Z backup tier up, scrub started. Guests on ospool never stopped. NASPool now 21.7T usable, any-two-disk failure tolerance.
Prune. Legacy vzdump dump/ (2.4T, 249 files, last written 2026-04-22, superseded by PBS-ANA)
pruned BY OMISSION: only the newest archive per guest (13 keepers, 295G) was copied out; the rest
died with the old pool. Nothing was rm'd.
Correction to the prior brief. It claimed guests' disks lived on NASPool and that Route C
meant everything down. Wrong: all 10 VM boot zvols + 3 CT rootfs are on ospool; NASPool is the
backup/bulk tier (PBS-ANA datastore, restic repos, vzdump dir storage). Read /etc/pve/*.conf
before believing a storage-placement claim.
Gotchas recorded. ana-nas exports pin no fsid=, so clients hold stale handles after a
pool rebuild — unmount before, remount after (ana-docker systemd automount; pbs-ana VM reboot).
logicalreferenced differed 289K vs 280K on a dataset whose content was diff -r identical —
compression-property accounting, not data. rsync --files-from paths are relative to the
source arg; a verify pass run from the wrong cwd returned a meaningless "0 diffs" alongside errors
— read the rc, not the count.
Open. Parked copy ospool/naspool-evac (1.65T) stays until the scrub is clean and one
Backrest + one PBS run succeed, then destroy. @evac snapshots to drop then. Backplane swap at
next colo visit (Dell R750xs 12×3.5" SAS backplane; front-half teardown), then
zpool add NASPool spare <bay9> <bay10>. PSU1 still dead. Runbook:
docs/runbooks/pfi-pve-naspool-rebuild.md. Booth: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/