103 lines
8.1 KiB
Markdown
103 lines
8.1 KiB
Markdown
# pfi-pve — NASPool rebuild as raidz2 (2026-09-05)
|
||
|
||
Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on
|
||
ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks
|
||
that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane
|
||
fault; cables and PERC both swapped with no change) become labelled cold
|
||
spares. Legacy vzdump archive pruned to the newest file per guest by
|
||
omission: the keepers are copied out, the rest dies with the old pool.
|
||
Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs
|
||
a re-run, not data, because the source of the running copy still exists.
|
||
|
||
Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/
|
||
|
||
## Fixed facts (verified 22:24Z–23:05Z, read-only)
|
||
|
||
- Host `root@10.250.250.31` (infra-ops user added 2026-09-06 after the rebuild), PVE 8.3.5, zfs 2.2.7 (no raidz expansion).
|
||
- Every VM boot disk and CT rootfs is on **ospool** (pbs-ana on local-lvm).
|
||
ospool ONLINE, 2 mirrors, 547G used, 10.2T free. **ospool is never
|
||
written to by an irreversible command in this runbook.**
|
||
- NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is `dump/`, 249 legacy
|
||
vzdump files, newest 2026-04-22; images 4.3G; template 124M),
|
||
backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G,
|
||
docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS
|
||
`.system` 1.4G (not restored — irrelevant on Proxmox; stays parked).
|
||
- NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports
|
||
`/mnt/backup`, `/mnt/backup/pbs-ana`, `/mnt/db`, `/mnt/docker`,
|
||
`/mnt/pve-VMStorage`, **no fsid= pinned → clients get stale handles
|
||
after the rebuild and must remount**), CT 112 ana-filebot (same binds),
|
||
VM 100 pbs-ana (NFS datastore from ana-nas), `pve-truenas` dir storage,
|
||
ana-docker (`/mnt/backup` systemd automount; containers rest-server,
|
||
backrest).
|
||
- SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS.
|
||
|
||
Disk identity (device letters are NOT stable; commands use by-id):
|
||
|
||
| bay | dev | by-id |
|
||
|---|---|---|
|
||
| 4 | sde | /dev/disk/by-id/scsi-350000398d879ea61 |
|
||
| 5 | sdf | /dev/disk/by-id/scsi-3500003981853ecbd |
|
||
| 6 | sdg | /dev/disk/by-id/scsi-35000039818588ec5 |
|
||
| 7 | sdh | /dev/disk/by-id/scsi-350000397c8218f09 |
|
||
| 8 | sdi | /dev/disk/by-id/scsi-350000397c813e23d |
|
||
| 11 | sdj | /dev/disk/by-id/scsi-350000398d85a1015 |
|
||
|
||
Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105
|
||
107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106
|
||
(2024-05-31) and qemu-108 (2026-04-20), plus each one's `.log`/`.notes`.
|
||
|
||
## Steps
|
||
|
||
1. **Quiesce.** ana-docker: `docker stop backrest rest-server`, stop the
|
||
`mnt-backup.automount` + `mnt-backup.mount` units. pfi-pve:
|
||
`qm shutdown 100`, `pct stop 112`, `pct stop 109`,
|
||
`pvesm set pve-truenas --disable 1`, confirm `fuser -vm /NASPool` empty.
|
||
2. **Park.** `zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac`;
|
||
`zfs snapshot -r NASPool@evac`; for backupStore, db, docker, webdav,
|
||
.system: `zfs send -R NASPool/<x>@evac | zfs recv -u ospool/naspool-evac/<x>`.
|
||
pve-VMStorage: `zfs create ospool/naspool-evac/pve-VMStorage` then rsync
|
||
images/ template/ snippets/ private/ frompfi/ and the keeper files into it.
|
||
Runs under nohup, log `/root/naspool-evac.log`.
|
||
3. **GATE — verify.** Per-dataset `used` within a few % of source; rsync
|
||
`--checksum` re-pass over pve-VMStorage reports zero transfers. Report to
|
||
operator; wait for go.
|
||
4. **POINT OF NO RETURN.** `zfs unmount -a` for NASPool, `zpool destroy NASPool`.
|
||
Then per disk, literal paths, one per line:
|
||
`zpool labelclear -f` and `wipefs -a` on each of the six by-id paths above.
|
||
5. **Create.** `zpool create -o ashift=12 -O compression=lz4 -O atime=off
|
||
-O xattr=sa NASPool raidz2 <six by-id paths>`. Same pool name so
|
||
storage.cfg and the CT bind mounts are unchanged.
|
||
6. **Restore.** Receive backupStore, db, docker, webdav back under NASPool
|
||
(`zfs send -R ospool/naspool-evac/<x>@evac | zfs recv -u NASPool/<x>`),
|
||
`zfs create NASPool/pve-VMStorage` and rsync the parked copy back. Set
|
||
`recordsize=1M` on NASPool/backupStore and NASPool/pve-VMStorage before
|
||
copying data in. Do NOT restore `.system`. Check mountpoints
|
||
`/NASPool/<x>`; `zfs mount -a`.
|
||
7. **Bring up.** `pvesm set pve-truenas --disable 0`, `pct start 109`,
|
||
`pct start 112`, `qm start 100`. ana-docker: start automount, then
|
||
`docker start rest-server backrest`; confirm `/mnt/backup/restic/repo/ana`
|
||
readable. pbs-ana: datastore `backups` online. `zpool scrub NASPool`.
|
||
8. **Settle.** Keep `ospool/naspool-evac` until the scrub is clean and one
|
||
Backrest + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`.
|
||
9. **Later.** Backplane swap (Route D) at the next colo visit; then
|
||
labelclear the two resurfaced disks and `zpool add NASPool spare <a> <b>`.
|
||
|
||
## Log
|
||
|
||
(appended as steps complete, UTC)
|
||
- 22:59Z operator: Route C hybrid, raidz2, bays 9/10 cold spares.
|
||
- 23:0xZ operator: prune vzdump (newest per guest confirmed 23:20Z), accept one PSU, start ASAP.
|
||
- 23:04Z quiesced: ana-docker backrest+rest-server stopped, /mnt/backup unmounted; pfi-pve VM100/CT112/CT109 down, pve-truenas disabled; `fuser -vm /NASPool` = kernel mount only.
|
||
- 23:04Z park copy launched (`/root/naspool-park.sh`, log `/root/naspool-evac.log`); ~180–200 MB/s after warm-up.
|
||
- 01:20Z PARK DONE (2h15m, ~200 MB/s sustained, no source read errors). All five zfs streams received; 26 keeper files + non-dump dirs rsynced (297G). Total parked 1.65T; ospool 2.18T alloc / 8.72T free.
|
||
- 01:25Z verify: `logicalreferenced` at @evac identical for backupStore/db/webdav/.system; docker 289K vs 280K is accounting only (`diff -r` clean, 8/8 files). Checksum re-pass (`rsync -anc`) over the 297G file copy launched, log `/root/naspool-verify.log`.
|
||
- 01:46Z VERIFIED: checksum re-pass over the 26 keepers (297G) = 0 diffs (20 min); non-dump dirs = 0 diffs (re-run with correct cwd after a path slip in the first pass). Parked copy is complete and byte-verified. GATE: waiting for operator go before `zpool destroy NASPool`.
|
||
- 02:00Z operator GO. Preflight: all six by-id paths resolved to NASPool members (zdb label), serials matched the table, ospool untouched.
|
||
- 02:01Z `zpool destroy NASPool` executed; six disks labelcleared + wipefs'd by literal by-id path; new NASPool created as raidz2-0 over the same six disks, ONLINE, no errors. Restore (step 6) launched, log `/root/naspool-rebuild.log`.
|
||
- 03:53Z REBUILD DONE: backupStore/db/docker/webdav received (logicalreferenced identical to parked), pve-VMStorage rsynced back (317.8 GB, rc=0, 26 keepers). NASPool raidz2: 2.48T alloc / 30.3T free raw, 21.7T usable, no errors. `.system` deliberately not restored (stays in the parked copy).
|
||
- 04:02Z bring-up: pve-truenas re-enabled; CT 109 ana-nas, CT 112 ana-filebot, VM 100 pbs-ana started; NFS exports serving; ana-docker automount restarted, rest-server + backrest healthy (401 on /); `pvesm list pbs-ana` = 116 backups, pve-truenas = 15 entries.
|
||
- 04:03Z `zpool scrub NASPool` started. Backup tier offline window: 23:04Z → 04:03Z (5h). Guests on ospool never stopped.
|
||
- OPEN: (a) destroy `ospool/naspool-evac` (1.65T) after the scrub is clean AND one Backrest (08:00Z) + one PBS run succeed; (b) `zfs destroy NASPool/*@evac` snapshots then; (c) backplane swap next colo visit → labelclear the two bay-9/10 disks → `zpool add NASPool spare`; (d) PSU1 still dead; (e) old PERC serial 3260688 = spare.
|
||
- 04:37Z operator: "tear down and delete truenas". TrueNAS SCALE leftovers `ospool/naspool-evac/.system` (+ children, 1.43G, incl. the 2023 SCALE-22.12.1 samba4 snapshot) destroyed — nothing on the live pool referenced them. Storage id `pve-truenas` removed and re-added as **`naspool-vmstorage`** (same path/content/prune; no guest conf or backup job referenced the old id; 15 entries listed). The TrueNAS 2G swap stubs died with the old partition tables; the two bay-9/10 disks still carry them until the backplane swap. `.system` is no longer in the parked copy — the "stays parked" note above is superseded.
|
||
- 04:43Z scrub of the new raidz2 NASPool finished: 0B repaired, 0 errors, 40m21s over 2.48T. Remaining gate for destroying `ospool/naspool-evac`: one Backrest run (08:00Z) + one PBS vzdump run (10:00Z) succeed.
|