vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9a916a759fe6f9333acd1f5229a22d784b357f06
esh-pfi-infrastructure/docs/runbooks/pfi-pve-naspool-rebuild.md
T

8.1 KiB
Raw Blame History

pfi-pve — NASPool rebuild as raidz2 (2026-09-05)

Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane fault; cables and PERC both swapped with no change) become labelled cold spares. Legacy vzdump archive pruned to the newest file per guest by omission: the keepers are copied out, the rest dies with the old pool. Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs a re-run, not data, because the source of the running copy still exists.

Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/

Fixed facts (verified 22:24Z–23:05Z, read-only)

  • Host root@10.250.250.31 (infra-ops user added 2026-09-06 after the rebuild), PVE 8.3.5, zfs 2.2.7 (no raidz expansion).
  • Every VM boot disk and CT rootfs is on ospool (pbs-ana on local-lvm). ospool ONLINE, 2 mirrors, 547G used, 10.2T free. ospool is never written to by an irreversible command in this runbook.
  • NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is dump/, 249 legacy vzdump files, newest 2026-04-22; images 4.3G; template 124M), backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G, docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS .system 1.4G (not restored — irrelevant on Proxmox; stays parked).
  • NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports /mnt/backup, /mnt/backup/pbs-ana, /mnt/db, /mnt/docker, /mnt/pve-VMStorage, no fsid= pinned → clients get stale handles after the rebuild and must remount), CT 112 ana-filebot (same binds), VM 100 pbs-ana (NFS datastore from ana-nas), pve-truenas dir storage, ana-docker (/mnt/backup systemd automount; containers rest-server, backrest).
  • SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS.

Disk identity (device letters are NOT stable; commands use by-id):

bay dev by-id
4 sde /dev/disk/by-id/scsi-350000398d879ea61
5 sdf /dev/disk/by-id/scsi-3500003981853ecbd
6 sdg /dev/disk/by-id/scsi-35000039818588ec5
7 sdh /dev/disk/by-id/scsi-350000397c8218f09
8 sdi /dev/disk/by-id/scsi-350000397c813e23d
11 sdj /dev/disk/by-id/scsi-350000398d85a1015

Keepers from dump/ (newest per guest, ~295G): qemu 101 102 103 104 105 107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106 (2024-05-31) and qemu-108 (2026-04-20), plus each one's .log/.notes.

Steps

  1. Quiesce. ana-docker: docker stop backrest rest-server, stop the mnt-backup.automount + mnt-backup.mount units. pfi-pve: qm shutdown 100, pct stop 112, pct stop 109, pvesm set pve-truenas --disable 1, confirm fuser -vm /NASPool empty.
  2. Park. zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac; zfs snapshot -r NASPool@evac; for backupStore, db, docker, webdav, .system: zfs send -R NASPool/<x>@evac | zfs recv -u ospool/naspool-evac/<x>. pve-VMStorage: zfs create ospool/naspool-evac/pve-VMStorage then rsync images/ template/ snippets/ private/ frompfi/ and the keeper files into it. Runs under nohup, log /root/naspool-evac.log.
  3. GATE — verify. Per-dataset used within a few % of source; rsync --checksum re-pass over pve-VMStorage reports zero transfers. Report to operator; wait for go.
  4. POINT OF NO RETURN. zfs unmount -a for NASPool, zpool destroy NASPool. Then per disk, literal paths, one per line: zpool labelclear -f and wipefs -a on each of the six by-id paths above.
  5. Create. zpool create -o ashift=12 -O compression=lz4 -O atime=off -O xattr=sa NASPool raidz2 <six by-id paths>. Same pool name so storage.cfg and the CT bind mounts are unchanged.
  6. Restore. Receive backupStore, db, docker, webdav back under NASPool (zfs send -R ospool/naspool-evac/<x>@evac | zfs recv -u NASPool/<x>), zfs create NASPool/pve-VMStorage and rsync the parked copy back. Set recordsize=1M on NASPool/backupStore and NASPool/pve-VMStorage before copying data in. Do NOT restore .system. Check mountpoints /NASPool/<x>; zfs mount -a.
  7. Bring up. pvesm set pve-truenas --disable 0, pct start 109, pct start 112, qm start 100. ana-docker: start automount, then docker start rest-server backrest; confirm /mnt/backup/restic/repo/ana readable. pbs-ana: datastore backups online. zpool scrub NASPool.
  8. Settle. Keep ospool/naspool-evac until the scrub is clean and one Backrest + one PBS run succeed. Then zfs destroy -r ospool/naspool-evac.
  9. Later. Backplane swap (Route D) at the next colo visit; then labelclear the two resurfaced disks and zpool add NASPool spare <a> <b>.

Log

(appended as steps complete, UTC)

  • 22:59Z operator: Route C hybrid, raidz2, bays 9/10 cold spares.
  • 23:0xZ operator: prune vzdump (newest per guest confirmed 23:20Z), accept one PSU, start ASAP.
  • 23:04Z quiesced: ana-docker backrest+rest-server stopped, /mnt/backup unmounted; pfi-pve VM100/CT112/CT109 down, pve-truenas disabled; fuser -vm /NASPool = kernel mount only.
  • 23:04Z park copy launched (/root/naspool-park.sh, log /root/naspool-evac.log); ~180–200 MB/s after warm-up.
  • 01:20Z PARK DONE (2h15m, ~200 MB/s sustained, no source read errors). All five zfs streams received; 26 keeper files + non-dump dirs rsynced (297G). Total parked 1.65T; ospool 2.18T alloc / 8.72T free.
  • 01:25Z verify: logicalreferenced at @evac identical for backupStore/db/webdav/.system; docker 289K vs 280K is accounting only (diff -r clean, 8/8 files). Checksum re-pass (rsync -anc) over the 297G file copy launched, log /root/naspool-verify.log.
  • 01:46Z VERIFIED: checksum re-pass over the 26 keepers (297G) = 0 diffs (20 min); non-dump dirs = 0 diffs (re-run with correct cwd after a path slip in the first pass). Parked copy is complete and byte-verified. GATE: waiting for operator go before zpool destroy NASPool.
  • 02:00Z operator GO. Preflight: all six by-id paths resolved to NASPool members (zdb label), serials matched the table, ospool untouched.
  • 02:01Z zpool destroy NASPool executed; six disks labelcleared + wipefs'd by literal by-id path; new NASPool created as raidz2-0 over the same six disks, ONLINE, no errors. Restore (step 6) launched, log /root/naspool-rebuild.log.
  • 03:53Z REBUILD DONE: backupStore/db/docker/webdav received (logicalreferenced identical to parked), pve-VMStorage rsynced back (317.8 GB, rc=0, 26 keepers). NASPool raidz2: 2.48T alloc / 30.3T free raw, 21.7T usable, no errors. .system deliberately not restored (stays in the parked copy).
  • 04:02Z bring-up: pve-truenas re-enabled; CT 109 ana-nas, CT 112 ana-filebot, VM 100 pbs-ana started; NFS exports serving; ana-docker automount restarted, rest-server + backrest healthy (401 on /); pvesm list pbs-ana = 116 backups, pve-truenas = 15 entries.
  • 04:03Z zpool scrub NASPool started. Backup tier offline window: 23:04Z → 04:03Z (5h). Guests on ospool never stopped.
  • OPEN: (a) destroy ospool/naspool-evac (1.65T) after the scrub is clean AND one Backrest (08:00Z) + one PBS run succeed; (b) zfs destroy NASPool/*@evac snapshots then; (c) backplane swap next colo visit → labelclear the two bay-9/10 disks → zpool add NASPool spare; (d) PSU1 still dead; (e) old PERC serial 3260688 = spare.
  • 04:37Z operator: "tear down and delete truenas". TrueNAS SCALE leftovers ospool/naspool-evac/.system (+ children, 1.43G, incl. the 2023 SCALE-22.12.1 samba4 snapshot) destroyed — nothing on the live pool referenced them. Storage id pve-truenas removed and re-added as naspool-vmstorage (same path/content/prune; no guest conf or backup job referenced the old id; 15 entries listed). The TrueNAS 2G swap stubs died with the old partition tables; the two bay-9/10 disks still carry them until the backplane swap. .system is no longer in the parked copy — the "stays parked" note above is superseded.
  • 04:43Z scrub of the new raidz2 NASPool finished: 0B repaired, 0 errors, 40m21s over 2.48T. Remaining gate for destroying ospool/naspool-evac: one Backrest run (08:00Z) + one PBS vzdump run (10:00Z) succeed.
Reference in New Issue View Git Blame Copy Permalink