diff --git a/docs/runbooks/pfi-pve-naspool-rebuild.md b/docs/runbooks/pfi-pve-naspool-rebuild.md new file mode 100644 index 0000000..c4fd796 --- /dev/null +++ b/docs/runbooks/pfi-pve-naspool-rebuild.md @@ -0,0 +1,87 @@ +# pfi-pve — NASPool rebuild as raidz2 (2026-09-05) + +Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on +ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks +that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane +fault; cables and PERC both swapped with no change) become labelled cold +spares. Legacy vzdump archive pruned to the newest file per guest by +omission: the keepers are copied out, the rest dies with the old pool. +Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs +a re-run, not data, because the source of the running copy still exists. + +Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/ + +## Fixed facts (verified 22:24Z–23:05Z, read-only) + +- Host `root@10.250.250.31`, PVE 8.3.5, zfs 2.2.7 (no raidz expansion). +- Every VM boot disk and CT rootfs is on **ospool** (pbs-ana on local-lvm). + ospool ONLINE, 2 mirrors, 547G used, 10.2T free. **ospool is never + written to by an irreversible command in this runbook.** +- NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is `dump/`, 249 legacy + vzdump files, newest 2026-04-22; images 4.3G; template 124M), + backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G, + docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS + `.system` 1.4G (not restored — irrelevant on Proxmox; stays parked). +- NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports + `/mnt/backup`, `/mnt/backup/pbs-ana`, `/mnt/db`, `/mnt/docker`, + `/mnt/pve-VMStorage`, **no fsid= pinned → clients get stale handles + after the rebuild and must remount**), CT 112 ana-filebot (same binds), + VM 100 pbs-ana (NFS datastore from ana-nas), `pve-truenas` dir storage, + ana-docker (`/mnt/backup` systemd automount; containers rest-server, + backrest). +- SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS. + +Disk identity (device letters are NOT stable; commands use by-id): + +| bay | dev | by-id | +|---|---|---| +| 4 | sde | /dev/disk/by-id/scsi-350000398d879ea61 | +| 5 | sdf | /dev/disk/by-id/scsi-3500003981853ecbd | +| 6 | sdg | /dev/disk/by-id/scsi-35000039818588ec5 | +| 7 | sdh | /dev/disk/by-id/scsi-350000397c8218f09 | +| 8 | sdi | /dev/disk/by-id/scsi-350000397c813e23d | +| 11 | sdj | /dev/disk/by-id/scsi-350000398d85a1015 | + +Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105 +107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106 +(2024-05-31) and qemu-108 (2026-04-20), plus each one's `.log`/`.notes`. + +## Steps + +1. **Quiesce.** ana-docker: `docker stop backrest rest-server`, stop the + `mnt-backup.automount` + `mnt-backup.mount` units. pfi-pve: + `qm shutdown 100`, `pct stop 112`, `pct stop 109`, + `pvesm set pve-truenas --disable 1`, confirm `fuser -vm /NASPool` empty. +2. **Park.** `zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac`; + `zfs snapshot -r NASPool@evac`; for backupStore, db, docker, webdav, + .system: `zfs send -R NASPool/@evac | zfs recv -u ospool/naspool-evac/`. + pve-VMStorage: `zfs create ospool/naspool-evac/pve-VMStorage` then rsync + images/ template/ snippets/ private/ frompfi/ and the keeper files into it. + Runs under nohup, log `/root/naspool-evac.log`. +3. **GATE — verify.** Per-dataset `used` within a few % of source; rsync + `--checksum` re-pass over pve-VMStorage reports zero transfers. Report to + operator; wait for go. +4. **POINT OF NO RETURN.** `zfs unmount -a` for NASPool, `zpool destroy NASPool`. + Then per disk, literal paths, one per line: + `zpool labelclear -f` and `wipefs -a` on each of the six by-id paths above. +5. **Create.** `zpool create -o ashift=12 -O compression=lz4 -O atime=off + -O xattr=sa NASPool raidz2 `. Same pool name so + storage.cfg and the CT bind mounts are unchanged. +6. **Restore.** Receive backupStore, db, docker, webdav back under NASPool + (`zfs send -R ospool/naspool-evac/@evac | zfs recv -u NASPool/`), + `zfs create NASPool/pve-VMStorage` and rsync the parked copy back. Set + `recordsize=1M` on NASPool/backupStore and NASPool/pve-VMStorage before + copying data in. Do NOT restore `.system`. Check mountpoints + `/NASPool/`; `zfs mount -a`. +7. **Bring up.** `pvesm set pve-truenas --disable 0`, `pct start 109`, + `pct start 112`, `qm start 100`. ana-docker: start automount, then + `docker start rest-server backrest`; confirm `/mnt/backup/restic/repo/ana` + readable. pbs-ana: datastore `backups` online. `zpool scrub NASPool`. +8. **Settle.** Keep `ospool/naspool-evac` until the scrub is clean and one + Backrest + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`. +9. **Later.** Backplane swap (Route D) at the next colo visit; then + labelclear the two resurfaced disks and `zpool add NASPool spare `. + +## Log + +(appended as steps complete, UTC)