ops(ana-ml2): pool-health actions landed — tank scrubbed clean + cleared, nvme-cli SMART inventory, zroot 91→73%; root cause of missed scrubs = nvme7 absent 04-23→09-05 (pool DEGRADED, ZED mail unrouted)

- playbooks/ana-ml2-pool-health.yaml: rerunnable elway play (scrub-if-idle, nvme-cli, dangling-image + builder prune; never prune -a) - servers/ana-ml2/README.md: 8-drive PM1725b inventory with SMART counters, nvme7 absence + alerting-gap note - persistent-memory: deferred entry closed, outcome + follow-ups (pool-health alerting, nvme7/slot 0-5 watch, boot import race)
2026-09-09 02:03:11 -07:00
parent 5ad948bf31
commit 5de5583762
4 changed files with 122 additions and 5 deletions
@@ -0,0 +1,43 @@
+# ana-ml2 pool health — the three deferred actions LANDED, and the root cause of the missed scrubs (2026-09-09)
+
+Executed 2026-09-09 00:29–02:02 PT via `playbooks/ana-ml2-pool-health.yaml` (elway, rerunnable) plus
+hand polling. Operator pre-approval: "we'll do all 3 on clean context" (00:30 PT ruling; findings `3e18a04`).
+
+## What was done
+
+1. **`zpool scrub tank`** started 00:29:28 PT, 6.84 TiB at ~1.4 GB/s. **Finished 02:02:12 PT: `scrub repaired 0B in 01:32:44 with 0 errors`.** `zpool clear tank` applied 02:02:38; nvme7n1 CKSUM 2 → 0, pool ONLINE, no known data errors. Seats kept serving throughout (1.25–1.43 GB/s issue rate).
+2. **`nvme-cli` installed** (2.13-2) and every drive's SMART read. nvme7 = Dell PM1725b 1.6 TB, sn
+   `S47VNY0K600221`, fw 1.2.0, `critical_warning 0`, `percentage_used 2%`, spare 100%, **`media_errors 2084`
+   (= `num_err_log_entries`)**, 3083 unsafe shutdowns / 3093 power cycles vs ~200 on the six `S5CU…` siblings.
+   The error-log entries all decode as "Successful Completion" (status 0) — the drive logs them but they carry no
+   LBA. Baseline `media_errors` 2084 at 00:32 PT; **still 2084 at 02:02 PT after the scrub read the whole 6.84 TiB — zero growth**, so the counter is prior-life history, not an active fault; the 2 CKSUM were the stale-block artefact of the 09-05 late resilver, not media. Full table in `servers/ana-ml2/README.md`.
+3. **zroot 91% → 73%**: `docker image prune` (dangling only) reclaimed 80.19 GB, `docker builder prune` 36.54 GB.
+   62 → ~40 images; the tagged idle set (vLLM v0.22–v0.27.1 + 4 nightlies, sglang, aeon-trainer, vibevoice 55 GB,
+   kokoro ×2, infinity, parakeet) was deliberately left — several are rollback seats. Two dangling volumes
+   (`parakeet_parakeet_cache`, `searxng_searxng-data`, 5 GB) also left; not asked for.
+
+## Why tank had no scrub since 04-12 — nvme7 was physically missing for 4½ months
+
+`journalctl --list-boots` + per-boot kernel logs: **boots on 04-23 (×2), 05-06, 06-12 and 08-26 each enumerated
+SEVEN NVMe controllers; only the 09-05 14:26 boot found eight.** PCIe downstream port `0000:02:04.0` (slot 0-5)
+bridged bus 07 every time but nothing answered on it. So from ≤04-23 to 09-05 `tank` was raidz2 **DEGRADED with
+nvme7n1 absent** — one parity margin left, unnoticed. Debian's `/usr/lib/zfs-linux/scrub` and `…/trim` filter
+`zpool list -o health` for `ONLINE` only, which is exactly why the second-Sunday cron scrubbed `zroot` (08-09) and
+skipped `tank`, and why the first-Sunday trim also skipped tank May–Aug and resumed 09-06. The 09-05 cold boot
+(clean systemd shutdown 14:02, power back 14:26) re-seated the drive; ZFS resilvered 638 GB (≈4 months of
+writes) in 1h10 and logged 2 CKSUM on nvme7n1 at import (stale blocks read before the resilver caught up).
+**Nobody was told:** `ZED_EMAIL_ADDR=root`, and there is no MTA on the box (`/var/mail` empty, no
+exim/postfix/msmtp).
+
+## Follow-ups (surfaced, NOT done — operator's call)
+
+- **Pool-health alerting.** A DEGRADED pool for 4½ months is a monitoring hole. Cheapest fix: a Beszel/Uptime-Kuma
+  style check on `zpool status -x` or `zpool list -H -o health` across the ZFS hosts, or point ZED at a real notifier.
+- **nvme7 replace-vs-keep.** Slot 0-5 dropped a drive for months and the drive carries 2084 lifetime media errors
+  and a flapping-grade power-cycle count. If the scrub was clean, keep it but watch `media_errors` at every
+  visit; growth ⇒ replace (any PM1725b 1.6 TB or larger; `zpool replace tank nvme7n1 <new>`). Also worth a reseat
+  / cable check on slot 0-5 at the next hands-on visit — the fault may be the bay, not the drive.
+- **Boot-time import race.** Every boot logs several vdevs UNAVAIL then ONLINE within the same second plus a
+  `vdev.no_replicas` event — the import fires before all NVMes settle. Harmless so far (raidz2 absorbs it) but it
+  is the same pattern that produced the 09-05 checksum errors. A `zfs-import` `After=`/udev-settle tweak would
+  close it.