diff --git a/docs/runbooks/pfi-pve-naspool-rebuild.md b/docs/runbooks/pfi-pve-naspool-rebuild.md index 6008e06..f12ca41 100644 --- a/docs/runbooks/pfi-pve-naspool-rebuild.md +++ b/docs/runbooks/pfi-pve-naspool-rebuild.md @@ -94,3 +94,7 @@ Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105 - 01:46Z VERIFIED: checksum re-pass over the 26 keepers (297G) = 0 diffs (20 min); non-dump dirs = 0 diffs (re-run with correct cwd after a path slip in the first pass). Parked copy is complete and byte-verified. GATE: waiting for operator go before `zpool destroy NASPool`. - 02:00Z operator GO. Preflight: all six by-id paths resolved to NASPool members (zdb label), serials matched the table, ospool untouched. - 02:01Z `zpool destroy NASPool` executed; six disks labelcleared + wipefs'd by literal by-id path; new NASPool created as raidz2-0 over the same six disks, ONLINE, no errors. Restore (step 6) launched, log `/root/naspool-rebuild.log`. +- 03:53Z REBUILD DONE: backupStore/db/docker/webdav received (logicalreferenced identical to parked), pve-VMStorage rsynced back (317.8 GB, rc=0, 26 keepers). NASPool raidz2: 2.48T alloc / 30.3T free raw, 21.7T usable, no errors. `.system` deliberately not restored (stays in the parked copy). +- 04:02Z bring-up: pve-truenas re-enabled; CT 109 ana-nas, CT 112 ana-filebot, VM 100 pbs-ana started; NFS exports serving; ana-docker automount restarted, rest-server + backrest healthy (401 on /); `pvesm list pbs-ana` = 116 backups, pve-truenas = 15 entries. +- 04:03Z `zpool scrub NASPool` started. Backup tier offline window: 23:04Z → 04:03Z (5h). Guests on ospool never stopped. +- OPEN: (a) destroy `ospool/naspool-evac` (1.65T) after the scrub is clean AND one Backrest (08:00Z) + one PBS run succeed; (b) `zfs destroy NASPool/*@evac` snapshots then; (c) backplane swap next colo visit → labelclear the two bay-9/10 disks → `zpool add NASPool spare`; (d) PSU1 still dead; (e) old PERC serial 3260688 = spare. diff --git a/persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md b/persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md new file mode 100644 index 0000000..2230f4c --- /dev/null +++ b/persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md @@ -0,0 +1,41 @@ +# 2026-09-06 — pfi-pve NASPool rebuilt as six-wide raidz2 (backplane fault, bays 9/10 dead) + +**What happened.** NASPool (TrueNAS-legacy, 4×mirror on 8 disks) had run two mirrors at +single-disk redundancy since ~2026-04-23. Root cause by elimination: R750xs backplane — PERC +PHYs 9 and 10 show no link at all (SAS address zero, no error counters); cables replaced by the +operator and the PERC replaced 2026-09-05 (serial 3260688 → 29R1754) with no change. Old PERC is +a spare, not scrap. Drives in bays 9/10 are healthy but invisible. + +**Route taken (operator, 2026-09-05 22:59Z): "Route C hybrid".** Park everything on ospool +(which holds EVERY guest boot disk and was never touched by an irreversible command), destroy, +recreate NASPool as raidz2 over the six linked disks, copy back. Bays 9/10 = labelled cold spares +(pull a failed disk, move a spare into its bay, `zpool replace`). Run on one PSU (PSU1 dead since +08-26) — accepted because a power cut mid-copy costs a re-run, not data. + +**Result.** 23:04Z quiesce → 01:20Z parked 1.65T (~200 MB/s, zero source read errors — the copy +doubled as the readability test of the two sole-survivor disks; no pre-scrub) → 01:46Z +byte-verified → 02:00Z operator GO → 02:01Z destroy + raidz2 create (by-id, ashift=12, lz4, +recordsize=1M on backupStore/pve-VMStorage) → 03:53Z restored → 04:03Z backup tier up, scrub +started. Guests on ospool never stopped. NASPool now 21.7T usable, any-two-disk failure tolerance. + +**Prune.** Legacy vzdump `dump/` (2.4T, 249 files, last written 2026-04-22, superseded by PBS-ANA) +pruned BY OMISSION: only the newest archive per guest (13 keepers, 295G) was copied out; the rest +died with the old pool. Nothing was `rm`'d. + +**Correction to the prior brief.** It claimed guests' disks lived on NASPool and that Route C +meant everything down. Wrong: all 10 VM boot zvols + 3 CT rootfs are on ospool; NASPool is the +backup/bulk tier (PBS-ANA datastore, restic repos, vzdump dir storage). Read `/etc/pve/*.conf` +before believing a storage-placement claim. + +**Gotchas recorded.** ana-nas exports pin no `fsid=`, so clients hold stale handles after a +pool rebuild — unmount before, remount after (ana-docker systemd automount; pbs-ana VM reboot). +`logicalreferenced` differed 289K vs 280K on a dataset whose content was `diff -r` identical — +compression-property accounting, not data. rsync `--files-from` paths are relative to the +source arg; a verify pass run from the wrong cwd returned a meaningless "0 diffs" alongside errors +— read the rc, not the count. + +**Open.** Parked copy `ospool/naspool-evac` (1.65T) stays until the scrub is clean and one +Backrest + one PBS run succeed, then destroy. `@evac` snapshots to drop then. Backplane swap at +next colo visit (Dell R750xs 12×3.5" SAS backplane; front-half teardown), then +`zpool add NASPool spare `. PSU1 still dead. Runbook: +`docs/runbooks/pfi-pve-naspool-rebuild.md`. Booth: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/ diff --git a/persistent-memory.md b/persistent-memory.md index 942d860..f75414e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-05_ +_Last updated: 2026-09-06_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -473,6 +473,7 @@ below is a live commitment or a known-open risk._ ## Recent decisions +- `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md` - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md` - `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md` - `[2026-09-04]` **ERP run 3c trained and GATED — the 20x LR cut erased the diversity gain and did NOT remove the −40pp selfharm regression.** LR-robust, so it comes from corpus content. CSAM clean on all three arms. ⚠ A pooled preserve-list test cannot see a single-axis collapse. → `persistent-memory.d/2026-09-04-run3c-trained-and-gated.md`