# pfi-pve — NASPool rebuild as raidz2 (2026-09-05) Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane fault; cables and PERC both swapped with no change) become labelled cold spares. Legacy vzdump archive pruned to the newest file per guest by omission: the keepers are copied out, the rest dies with the old pool. Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs a re-run, not data, because the source of the running copy still exists. Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/ ## Fixed facts (verified 22:24Z–23:05Z, read-only) - Host `root@10.250.250.31` (infra-ops user added 2026-09-06 after the rebuild), PVE 8.3.5, zfs 2.2.7 (no raidz expansion). - Every VM boot disk and CT rootfs is on **ospool** (pbs-ana on local-lvm). ospool ONLINE, 2 mirrors, 547G used, 10.2T free. **ospool is never written to by an irreversible command in this runbook.** - NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is `dump/`, 249 legacy vzdump files, newest 2026-04-22; images 4.3G; template 124M), backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G, docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS `.system` 1.4G (not restored — irrelevant on Proxmox; stays parked). - NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports `/mnt/backup`, `/mnt/backup/pbs-ana`, `/mnt/db`, `/mnt/docker`, `/mnt/pve-VMStorage`, **no fsid= pinned → clients get stale handles after the rebuild and must remount**), CT 112 ana-filebot (same binds), VM 100 pbs-ana (NFS datastore from ana-nas), `pve-truenas` dir storage, ana-docker (`/mnt/backup` systemd automount; containers rest-server, backrest). - SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS. Disk identity (device letters are NOT stable; commands use by-id): | bay | dev | by-id | |---|---|---| | 4 | sde | /dev/disk/by-id/scsi-350000398d879ea61 | | 5 | sdf | /dev/disk/by-id/scsi-3500003981853ecbd | | 6 | sdg | /dev/disk/by-id/scsi-35000039818588ec5 | | 7 | sdh | /dev/disk/by-id/scsi-350000397c8218f09 | | 8 | sdi | /dev/disk/by-id/scsi-350000397c813e23d | | 11 | sdj | /dev/disk/by-id/scsi-350000398d85a1015 | Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105 107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106 (2024-05-31) and qemu-108 (2026-04-20), plus each one's `.log`/`.notes`. ## Steps 1. **Quiesce.** ana-docker: `docker stop backrest rest-server`, stop the `mnt-backup.automount` + `mnt-backup.mount` units. pfi-pve: `qm shutdown 100`, `pct stop 112`, `pct stop 109`, `pvesm set pve-truenas --disable 1`, confirm `fuser -vm /NASPool` empty. 2. **Park.** `zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac`; `zfs snapshot -r NASPool@evac`; for backupStore, db, docker, webdav, .system: `zfs send -R NASPool/@evac | zfs recv -u ospool/naspool-evac/`. pve-VMStorage: `zfs create ospool/naspool-evac/pve-VMStorage` then rsync images/ template/ snippets/ private/ frompfi/ and the keeper files into it. Runs under nohup, log `/root/naspool-evac.log`. 3. **GATE — verify.** Per-dataset `used` within a few % of source; rsync `--checksum` re-pass over pve-VMStorage reports zero transfers. Report to operator; wait for go. 4. **POINT OF NO RETURN.** `zfs unmount -a` for NASPool, `zpool destroy NASPool`. Then per disk, literal paths, one per line: `zpool labelclear -f` and `wipefs -a` on each of the six by-id paths above. 5. **Create.** `zpool create -o ashift=12 -O compression=lz4 -O atime=off -O xattr=sa NASPool raidz2 `. Same pool name so storage.cfg and the CT bind mounts are unchanged. 6. **Restore.** Receive backupStore, db, docker, webdav back under NASPool (`zfs send -R ospool/naspool-evac/@evac | zfs recv -u NASPool/`), `zfs create NASPool/pve-VMStorage` and rsync the parked copy back. Set `recordsize=1M` on NASPool/backupStore and NASPool/pve-VMStorage before copying data in. Do NOT restore `.system`. Check mountpoints `/NASPool/`; `zfs mount -a`. 7. **Bring up.** `pvesm set pve-truenas --disable 0`, `pct start 109`, `pct start 112`, `qm start 100`. ana-docker: start automount, then `docker start rest-server backrest`; confirm `/mnt/backup/restic/repo/ana` readable. pbs-ana: datastore `backups` online. `zpool scrub NASPool`. 8. **Settle.** Keep `ospool/naspool-evac` until the scrub is clean and one Backrest + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`. 9. **Later.** Backplane swap (Route D) at the next colo visit; then labelclear the two resurfaced disks and `zpool add NASPool spare `. ## Log (appended as steps complete, UTC) - 22:59Z operator: Route C hybrid, raidz2, bays 9/10 cold spares. - 23:0xZ operator: prune vzdump (newest per guest confirmed 23:20Z), accept one PSU, start ASAP. - 23:04Z quiesced: ana-docker backrest+rest-server stopped, /mnt/backup unmounted; pfi-pve VM100/CT112/CT109 down, pve-truenas disabled; `fuser -vm /NASPool` = kernel mount only. - 23:04Z park copy launched (`/root/naspool-park.sh`, log `/root/naspool-evac.log`); ~180–200 MB/s after warm-up. - 01:20Z PARK DONE (2h15m, ~200 MB/s sustained, no source read errors). All five zfs streams received; 26 keeper files + non-dump dirs rsynced (297G). Total parked 1.65T; ospool 2.18T alloc / 8.72T free. - 01:25Z verify: `logicalreferenced` at @evac identical for backupStore/db/webdav/.system; docker 289K vs 280K is accounting only (`diff -r` clean, 8/8 files). Checksum re-pass (`rsync -anc`) over the 297G file copy launched, log `/root/naspool-verify.log`. - 01:46Z VERIFIED: checksum re-pass over the 26 keepers (297G) = 0 diffs (20 min); non-dump dirs = 0 diffs (re-run with correct cwd after a path slip in the first pass). Parked copy is complete and byte-verified. GATE: waiting for operator go before `zpool destroy NASPool`. - 02:00Z operator GO. Preflight: all six by-id paths resolved to NASPool members (zdb label), serials matched the table, ospool untouched. - 02:01Z `zpool destroy NASPool` executed; six disks labelcleared + wipefs'd by literal by-id path; new NASPool created as raidz2-0 over the same six disks, ONLINE, no errors. Restore (step 6) launched, log `/root/naspool-rebuild.log`. - 03:53Z REBUILD DONE: backupStore/db/docker/webdav received (logicalreferenced identical to parked), pve-VMStorage rsynced back (317.8 GB, rc=0, 26 keepers). NASPool raidz2: 2.48T alloc / 30.3T free raw, 21.7T usable, no errors. `.system` deliberately not restored (stays in the parked copy). - 04:02Z bring-up: pve-truenas re-enabled; CT 109 ana-nas, CT 112 ana-filebot, VM 100 pbs-ana started; NFS exports serving; ana-docker automount restarted, rest-server + backrest healthy (401 on /); `pvesm list pbs-ana` = 116 backups, pve-truenas = 15 entries. - 04:03Z `zpool scrub NASPool` started. Backup tier offline window: 23:04Z → 04:03Z (5h). Guests on ospool never stopped. - OPEN: (a) destroy `ospool/naspool-evac` (1.65T) after the scrub is clean AND one Backrest (08:00Z) + one PBS run succeed; (b) `zfs destroy NASPool/*@evac` snapshots then; (c) backplane swap next colo visit → labelclear the two bay-9/10 disks → `zpool add NASPool spare`; (d) PSU1 still dead; (e) old PERC serial 3260688 = spare. - 04:37Z operator: "tear down and delete truenas". TrueNAS SCALE leftovers `ospool/naspool-evac/.system` (+ children, 1.43G, incl. the 2023 SCALE-22.12.1 samba4 snapshot) destroyed — nothing on the live pool referenced them. Storage id `pve-truenas` removed and re-added as **`naspool-vmstorage`** (same path/content/prune; no guest conf or backup job referenced the old id; 15 entries listed). The TrueNAS 2G swap stubs died with the old partition tables; the two bay-9/10 disks still carry them until the backplane swap. `.system` is no longer in the parked copy — the "stays parked" note above is superseded. - 04:43Z scrub of the new raidz2 NASPool finished: 0B repaired, 0 errors, 40m21s over 2.48T. Remaining gate for destroying `ospool/naspool-evac`: one Backrest run (08:00Z) + one PBS vzdump run (10:00Z) succeed.