docs(pfi-pve): NASPool rebuild runbook — raidz2 hybrid, park on ospool, prune vzdump by omission

2026-09-05 16:04:01 -07:00
parent a51af1887e
commit 1bc7519a24
1 changed files with 87 additions and 0 deletions
@@ -0,0 +1,87 @@
+# pfi-pve — NASPool rebuild as raidz2 (2026-09-05)
+
+Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on
+ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks
+that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane
+fault; cables and PERC both swapped with no change) become labelled cold
+spares. Legacy vzdump archive pruned to the newest file per guest by
+omission: the keepers are copied out, the rest dies with the old pool.
+Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs
+a re-run, not data, because the source of the running copy still exists.
+
+Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/
+
+## Fixed facts (verified 22:24Z–23:05Z, read-only)
+
+- Host `root@10.250.250.31`, PVE 8.3.5, zfs 2.2.7 (no raidz expansion).
+- Every VM boot disk and CT rootfs is on **ospool** (pbs-ana on local-lvm).
+  ospool ONLINE, 2 mirrors, 547G used, 10.2T free. **ospool is never
+  written to by an irreversible command in this runbook.**
+- NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is `dump/`, 249 legacy
+  vzdump files, newest 2026-04-22; images 4.3G; template 124M),
+  backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G,
+  docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS
+  `.system` 1.4G (not restored — irrelevant on Proxmox; stays parked).
+- NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports
+  `/mnt/backup`, `/mnt/backup/pbs-ana`, `/mnt/db`, `/mnt/docker`,
+  `/mnt/pve-VMStorage`, **no fsid= pinned → clients get stale handles
+  after the rebuild and must remount**), CT 112 ana-filebot (same binds),
+  VM 100 pbs-ana (NFS datastore from ana-nas), `pve-truenas` dir storage,
+  ana-docker (`/mnt/backup` systemd automount; containers rest-server,
+  backrest).
+- SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS.
+
+Disk identity (device letters are NOT stable; commands use by-id):
+
+| bay | dev | by-id |
+|---|---|---|
+| 4 | sde | /dev/disk/by-id/scsi-350000398d879ea61 |
+| 5 | sdf | /dev/disk/by-id/scsi-3500003981853ecbd |
+| 6 | sdg | /dev/disk/by-id/scsi-35000039818588ec5 |
+| 7 | sdh | /dev/disk/by-id/scsi-350000397c8218f09 |
+| 8 | sdi | /dev/disk/by-id/scsi-350000397c813e23d |
+| 11 | sdj | /dev/disk/by-id/scsi-350000398d85a1015 |
+
+Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105
+107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106
+(2024-05-31) and qemu-108 (2026-04-20), plus each one's `.log`/`.notes`.
+
+## Steps
+
+1. **Quiesce.** ana-docker: `docker stop backrest rest-server`, stop the
+   `mnt-backup.automount` + `mnt-backup.mount` units. pfi-pve:
+   `qm shutdown 100`, `pct stop 112`, `pct stop 109`,
+   `pvesm set pve-truenas --disable 1`, confirm `fuser -vm /NASPool` empty.
+2. **Park.** `zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac`;
+   `zfs snapshot -r NASPool@evac`; for backupStore, db, docker, webdav,
+   .system: `zfs send -R NASPool/<x>@evac | zfs recv -u ospool/naspool-evac/<x>`.
+   pve-VMStorage: `zfs create ospool/naspool-evac/pve-VMStorage` then rsync
+   images/ template/ snippets/ private/ frompfi/ and the keeper files into it.
+   Runs under nohup, log `/root/naspool-evac.log`.
+3. **GATE — verify.** Per-dataset `used` within a few % of source; rsync
+   `--checksum` re-pass over pve-VMStorage reports zero transfers. Report to
+   operator; wait for go.
+4. **POINT OF NO RETURN.** `zfs unmount -a` for NASPool, `zpool destroy NASPool`.
+   Then per disk, literal paths, one per line:
+   `zpool labelclear -f` and `wipefs -a` on each of the six by-id paths above.
+5. **Create.** `zpool create -o ashift=12 -O compression=lz4 -O atime=off
+   -O xattr=sa NASPool raidz2 <six by-id paths>`. Same pool name so
+   storage.cfg and the CT bind mounts are unchanged.
+6. **Restore.** Receive backupStore, db, docker, webdav back under NASPool
+   (`zfs send -R ospool/naspool-evac/<x>@evac | zfs recv -u NASPool/<x>`),
+   `zfs create NASPool/pve-VMStorage` and rsync the parked copy back. Set
+   `recordsize=1M` on NASPool/backupStore and NASPool/pve-VMStorage before
+   copying data in. Do NOT restore `.system`. Check mountpoints
+   `/NASPool/<x>`; `zfs mount -a`.
+7. **Bring up.** `pvesm set pve-truenas --disable 0`, `pct start 109`,
+   `pct start 112`, `qm start 100`. ana-docker: start automount, then
+   `docker start rest-server backrest`; confirm `/mnt/backup/restic/repo/ana`
+   readable. pbs-ana: datastore `backups` online. `zpool scrub NASPool`.
+8. **Settle.** Keep `ospool/naspool-evac` until the scrub is clean and one
+   Backrest + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`.
+9. **Later.** Backplane swap (Route D) at the next colo visit; then
+   labelclear the two resurfaced disks and `zpool add NASPool spare <a> <b>`.
+
+## Log
+
+(appended as steps complete, UTC)