ops(ana-ml2): pool-health actions landed — tank scrubbed clean + cleared, nvme-cli SMART inventory, zroot 91→73%; root cause of missed scrubs = nvme7 absent 04-23→09-05 (pool DEGRADED, ZED mail unrouted)

- playbooks/ana-ml2-pool-health.yaml: rerunnable elway play (scrub-if-idle, nvme-cli, dangling-image + builder prune; never prune -a)
- servers/ana-ml2/README.md: 8-drive PM1725b inventory with SMART counters, nvme7 absence + alerting-gap note
- persistent-memory: deferred entry closed, outcome + follow-ups (pool-health alerting, nvme7/slot 0-5 watch, boot import race)

persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md

@@ -0,0 +1,43 @@
+# ana-ml2 pool health — the three deferred actions LANDED, and the root cause of the missed scrubs (2026-09-09)
+
+Executed 2026-09-09 00:29–02:02 PT via `playbooks/ana-ml2-pool-health.yaml` (elway, rerunnable) plus
+hand polling. Operator pre-approval: "we'll do all 3 on clean context" (00:30 PT ruling; findings `3e18a04`).
+
+## What was done
+
+1. **`zpool scrub tank`** started 00:29:28 PT, 6.84 TiB at ~1.4 GB/s. **Finished 02:02:12 PT: `scrub repaired 0B in 01:32:44 with 0 errors`.** `zpool clear tank` applied 02:02:38; nvme7n1 CKSUM 2 → 0, pool ONLINE, no known data errors. Seats kept serving throughout (1.25–1.43 GB/s issue rate).
+2. **`nvme-cli` installed** (2.13-2) and every drive's SMART read. nvme7 = Dell PM1725b 1.6 TB, sn
+   `S47VNY0K600221`, fw 1.2.0, `critical_warning 0`, `percentage_used 2%`, spare 100%, **`media_errors 2084`
+   (= `num_err_log_entries`)**, 3083 unsafe shutdowns / 3093 power cycles vs ~200 on the six `S5CU…` siblings.
+   The error-log entries all decode as "Successful Completion" (status 0) — the drive logs them but they carry no
+   LBA. Baseline `media_errors` 2084 at 00:32 PT; **still 2084 at 02:02 PT after the scrub read the whole 6.84 TiB — zero growth**, so the counter is prior-life history, not an active fault; the 2 CKSUM were the stale-block artefact of the 09-05 late resilver, not media. Full table in `servers/ana-ml2/README.md`.
+3. **zroot 91% → 73%**: `docker image prune` (dangling only) reclaimed 80.19 GB, `docker builder prune` 36.54 GB.
+   62 → ~40 images; the tagged idle set (vLLM v0.22–v0.27.1 + 4 nightlies, sglang, aeon-trainer, vibevoice 55 GB,
+   kokoro ×2, infinity, parakeet) was deliberately left — several are rollback seats. Two dangling volumes
+   (`parakeet_parakeet_cache`, `searxng_searxng-data`, 5 GB) also left; not asked for.
+
+## Why tank had no scrub since 04-12 — nvme7 was physically missing for 4½ months
+
+`journalctl --list-boots` + per-boot kernel logs: **boots on 04-23 (×2), 05-06, 06-12 and 08-26 each enumerated
+SEVEN NVMe controllers; only the 09-05 14:26 boot found eight.** PCIe downstream port `0000:02:04.0` (slot 0-5)
+bridged bus 07 every time but nothing answered on it. So from ≤04-23 to 09-05 `tank` was raidz2 **DEGRADED with
+nvme7n1 absent** — one parity margin left, unnoticed. Debian's `/usr/lib/zfs-linux/scrub` and `…/trim` filter
+`zpool list -o health` for `ONLINE` only, which is exactly why the second-Sunday cron scrubbed `zroot` (08-09) and
+skipped `tank`, and why the first-Sunday trim also skipped tank May–Aug and resumed 09-06. The 09-05 cold boot
+(clean systemd shutdown 14:02, power back 14:26) re-seated the drive; ZFS resilvered 638 GB (≈4 months of
+writes) in 1h10 and logged 2 CKSUM on nvme7n1 at import (stale blocks read before the resilver caught up).
+**Nobody was told:** `ZED_EMAIL_ADDR=root`, and there is no MTA on the box (`/var/mail` empty, no
+exim/postfix/msmtp).
+
+## Follow-ups (surfaced, NOT done — operator's call)
+
+- **Pool-health alerting.** A DEGRADED pool for 4½ months is a monitoring hole. Cheapest fix: a Beszel/Uptime-Kuma
+  style check on `zpool status -x` or `zpool list -H -o health` across the ZFS hosts, or point ZED at a real notifier.
+- **nvme7 replace-vs-keep.** Slot 0-5 dropped a drive for months and the drive carries 2084 lifetime media errors
+  and a flapping-grade power-cycle count. If the scrub was clean, keep it but watch `media_errors` at every
+  visit; growth ⇒ replace (any PM1725b 1.6 TB or larger; `zpool replace tank nvme7n1 <new>`). Also worth a reseat
+  / cable check on slot 0-5 at the next hands-on visit — the fault may be the bay, not the drive.
+- **Boot-time import race.** Every boot logs several vdevs UNAVAIL then ONLINE within the same second plus a
+  `vdev.no_replicas` event — the import fires before all NVMes settle. Harmless so far (raidz2 absorbs it) but it
+  is the same pattern that produced the 09-05 checksum errors. A `zfs-import` `After=`/udev-settle tweak would
+  close it.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-09 00:35 PT (fleet-ops: ERP run 7 TRAINING on pfi-gx10 [opening-split slot]; run 6 gated TRANSFERRED after the operator's CSAM adjudication; erp-tune-v6-nvfp4a16 serving on ana-ml2 as `trial`; ESH static-WAN follow-ups + YTVC + webhook repaired; ana-ml2 routes persisted; ana-ml2 tank/zroot actions deferred to the next session)_
+_Last updated: 2026-09-09 02:10 PT (fleet-ops: ana-ml2 pool actions LANDED — tank scrubbed clean + cleared, nvme7 SMART read [2084 lifetime media errors, zero growth], zroot 91→73%; ROOT CAUSE of the missed scrubs = nvme7 physically absent 04-23→09-05, pool DEGRADED and ZED mail to nowhere; ERP run 7 TRAINING on pfi-gx10 [~64/542 at 00:42 PT, adapter ~noon]; erp-tune-v6-nvfp4a16 serving as `trial`)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -120,9 +120,11 @@ clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved
   --adapter run-07/adapter --out serve/merged-run07 --chat-template <stock>`), copy stock `processor_config.json`
   into `merged-run07`, serve `erp-seat-base-ara` for floors → on brokkr's swap cue serve `erp-tune-v7` (run-5 flags,
   :8098). Runbook `docs/runbooks/gx10-run-07.md`; run-6 choreography in `gx10-run-06.md`.
-- **⏳ ana-ml2 pool actions — NEXT SESSION, operator-approved in principle:** (1) `zpool scrub tank` + `zpool clear`
-  on a clean pass; (2) `apt install nvme-cli` + read nvme7 SMART; (3) docker image/builder prune to pull zroot back
-  from 91%. Detail + order: `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`.
+- **✅ ana-ml2 pool actions DONE 2026-09-09 02:02 PT** (`playbooks/ana-ml2-pool-health.yaml`): tank scrub clean +
+  cleared, nvme-cli in, zroot 73%. ⚠ Open follow-ups, operator's call: (a) ZFS pool-health ALERTING — tank sat
+  DEGRADED 04-23→09-05 with nvme7 physically absent and nobody knew (ZED mails `root`, no MTA); (b) nvme7 / slot 0-5
+  keep-vs-replace — `media_errors` 2084 lifetime, 0 growth over a full scrub, watch it each visit; (c) boot-time
+  import race (vdevs UNAVAIL→ONLINE + `no_replicas` every boot). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`.
 - **`trial` (LiteLLM) → `erp-tune-v6-nvfp4a16` on ana-ml2 :8021** (stack `stacks/erp-seat`, vLLM nightly
   `311b3513`, no gate by operator ruling). Forced tool_choice is prompt-driven (6/9); `response_format: json_schema`
   is deterministic. ⚠ `stacks/gemma4-charrp` lacks `--exclude-tools-when-tool-choice-none` (same empty-turn trap);
@@ -144,6 +146,7 @@ clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved
 
 ## Recent decisions
 
+- `[2026-09-09]` **ana-ml2 pool actions LANDED (scrub 0 errors in 1h33 → `zpool clear`; nvme-cli + full-drive SMART table; zroot 91→73% via dangling-image + builder prune, tagged rollback seats kept) — and the missed-scrub mystery SOLVED: nvme7 (slot 0-5, `S47VNY0K600221`) was absent from every boot 04-23→09-05, tank was raidz2-DEGRADED for 4½ months, Debian's scrub/trim cron only touches `ONLINE` pools, and ZED's alert went to a root mailbox with no MTA.** nvme7's 2084 media errors did not move across the scrub → historical, keep + watch. Playbook `playbooks/ana-ml2-pool-health.yaml`; inventory in `servers/ana-ml2/README.md`. → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`
 - `[2026-09-09]` **ana-ml2 `tank`: 2 CKSUM errors on nvme7n1 after a boot-time resilver, NO scrub since 04-12, zroot 91% — three actions DEFERRED to a clean-context session** (scrub → nvme-cli SMART → docker prune), operator ruling "we'll do all 3 on clean context"; tracked at commit `3e18a04` + the post-clear handoff. ESH 10G links measured clean (fiber run live on UDM SFP+2 ↔ USW-Pro-XG Media). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`
 - `[2026-09-08]` **ana-ml2 mesh return routes PERSISTED** as `/etc/network/if-up.d/mesh-routes` (Debian 13 ifupdown, no netplan) via `playbooks/ana-ml2-mesh-routes.yaml` (elway, verified) — operator: "persist the routes". Hook not yet exercised by a real reboot. `f923d6a`.
 - `[2026-09-08]` **ERP run 7 LAUNCHED on pfi-gx10 23:06 PT** under `operator-2026-09-08-rnd-run7` — opening-split slot + mask union; free check passed with two explained deltas; first launch died on a missing recipe (zsh quoting). → `persistent-memory.d/2026-09-08-erp-run7-launched.md`

playbooks/ana-ml2-pool-health.yaml

@@ -0,0 +1,44 @@
+# ana-ml2 pool-health actions (operator-approved 2026-09-09, "all 3 on clean context").
+#
+#   1. kick a scrub of `tank` (2 CKSUM on nvme7n1 after the 09-05 boot resilver,
+#      no scrub since 04-12) — polled separately; `zpool clear` only after a
+#      clean pass, by hand, never in this playbook.
+#   2. install nvme-cli so nvme7's SMART counters can be read.
+#   3. reclaim zroot (91%): dangling images + build cache ONLY. Never `prune -a`
+#      here — tagged idle images include rollback seats.
+#
+# Rerunnable: a second run shows the scrub step skipped (in progress / done
+# within the hour is not detected — the guard is "not already scrubbing"),
+# nvme-cli ok, prunes ok with 0 B reclaimed.
+
+steps:
+  - name: Start scrub of tank (skip if one is already running)
+    shell: zpool scrub tank
+    sudo: true
+    when: "! zpool status tank | grep -q 'scrub in progress'"
+
+  - name: Install nvme-cli
+    shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -q nvme-cli
+    sudo: true
+    creates: /usr/sbin/nvme
+
+  - name: Prune dangling docker images (NOT -a)
+    shell: docker image prune -f | tail -1
+    sudo: true
+
+  - name: Prune docker build cache
+    shell: docker builder prune -f | tail -1
+    sudo: true
+
+verify:
+  - name: Scrub is in progress or already completed today
+    shell: "zpool status tank | grep -Eq 'scrub in progress|scrub repaired .* on .*$(date +%b\\ *%e)'"
+    changed_when: "false"
+
+  - name: nvme binary present
+    shell: test -x /usr/sbin/nvme
+    changed_when: "false"
+
+  - name: zroot usage after prune
+    shell: df -h / | tail -1
+    changed_when: "false"

servers/ana-ml2/README.md

@@ -16,10 +16,37 @@ Primary AI inference host for PFI.
 - **CPU:** AMD EPYC 9254 24-core (96 threads)
 - **RAM:** 566 GB
 - **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
-- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
+- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
 - **OS:** Debian 13 (trixie), kernel 6.12.x
 - **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
 
+## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`
+
+All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
+a Broadcom PEX switch. Two provenance batches: the `S5CU…` six (fw 1.2.2) and the
+`S47V…` pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
+
+| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
+|---|---|---|---|---|---|---|---|---|
+| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
+| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
+| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
+| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
+| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
+| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
+| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
+| **nvme7** | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | **2084** | 2% |
+
+⚠ **nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05** (kernel enumerated 7
+NVMes per boot; PCIe downstream port `02:04.0` had nothing on bus 07). It reappeared at
+the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
+at import. While it was missing `tank` was DEGRADED, and Debian's `zfsutils-linux` cron
+(`/usr/lib/zfs-linux/{scrub,trim}`) only touches pools whose health is `ONLINE`, so tank
+got **no scrub and no trim from 04-12 to 09-06**. ZED's `ZED_EMAIL_ADDR=root` has no
+MTA behind it, so the 4½-month degradation alerted nobody. `media_errors=2084` on
+nvme7 is a lifetime counter; the 2026-09-09 scrub is the first fresh measurement
+(baseline 2084 at 00:32 PT — compare after any future event, growth = replace).
+
 ## Key paths
 
 | Path | Purpose |