docs(pfi-pve): NASPool rebuilt as raidz2 — runbook log complete, memory entry

2026-09-05 21:04:14 -07:00
parent 0df9312832
commit ab222133e9
3 changed files with 47 additions and 1 deletions
@@ -0,0 +1,41 @@
+# 2026-09-06 — pfi-pve NASPool rebuilt as six-wide raidz2 (backplane fault, bays 9/10 dead)
+
+**What happened.** NASPool (TrueNAS-legacy, 4×mirror on 8 disks) had run two mirrors at
+single-disk redundancy since ~2026-04-23. Root cause by elimination: R750xs backplane — PERC
+PHYs 9 and 10 show no link at all (SAS address zero, no error counters); cables replaced by the
+operator and the PERC replaced 2026-09-05 (serial 3260688 → 29R1754) with no change. Old PERC is
+a spare, not scrap. Drives in bays 9/10 are healthy but invisible.
+
+**Route taken (operator, 2026-09-05 22:59Z): "Route C hybrid".** Park everything on ospool
+(which holds EVERY guest boot disk and was never touched by an irreversible command), destroy,
+recreate NASPool as raidz2 over the six linked disks, copy back. Bays 9/10 = labelled cold spares
+(pull a failed disk, move a spare into its bay, `zpool replace`). Run on one PSU (PSU1 dead since
+08-26) — accepted because a power cut mid-copy costs a re-run, not data.
+
+**Result.** 23:04Z quiesce → 01:20Z parked 1.65T (~200 MB/s, zero source read errors — the copy
+doubled as the readability test of the two sole-survivor disks; no pre-scrub) → 01:46Z
+byte-verified → 02:00Z operator GO → 02:01Z destroy + raidz2 create (by-id, ashift=12, lz4,
+recordsize=1M on backupStore/pve-VMStorage) → 03:53Z restored → 04:03Z backup tier up, scrub
+started. Guests on ospool never stopped. NASPool now 21.7T usable, any-two-disk failure tolerance.
+
+**Prune.** Legacy vzdump `dump/` (2.4T, 249 files, last written 2026-04-22, superseded by PBS-ANA)
+pruned BY OMISSION: only the newest archive per guest (13 keepers, 295G) was copied out; the rest
+died with the old pool. Nothing was `rm`'d.
+
+**Correction to the prior brief.** It claimed guests' disks lived on NASPool and that Route C
+meant everything down. Wrong: all 10 VM boot zvols + 3 CT rootfs are on ospool; NASPool is the
+backup/bulk tier (PBS-ANA datastore, restic repos, vzdump dir storage). Read `/etc/pve/*.conf`
+before believing a storage-placement claim.
+
+**Gotchas recorded.** ana-nas exports pin no `fsid=`, so clients hold stale handles after a
+pool rebuild — unmount before, remount after (ana-docker systemd automount; pbs-ana VM reboot).
+`logicalreferenced` differed 289K vs 280K on a dataset whose content was `diff -r` identical —
+compression-property accounting, not data. rsync `--files-from` paths are relative to the
+source arg; a verify pass run from the wrong cwd returned a meaningless "0 diffs" alongside errors
+— read the rc, not the count.
+
+**Open.** Parked copy `ospool/naspool-evac` (1.65T) stays until the scrub is clean and one
+Backrest + one PBS run succeed, then destroy. `@evac` snapshots to drop then. Backplane swap at
+next colo visit (Dell R750xs 12×3.5" SAS backplane; front-half teardown), then
+`zpool add NASPool spare <bay9> <bay10>`. PSU1 still dead. Runbook:
+`docs/runbooks/pfi-pve-naspool-rebuild.md`. Booth: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/