esh-pfi-infrastructure/persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md

ana-ml2 pool health — the three deferred actions LANDED, and the root cause of the missed scrubs (2026-09-09)

Executed 2026-09-09 00:29–02:02 PT via playbooks/ana-ml2-pool-health.yaml (elway, rerunnable) plus hand polling. Operator pre-approval: "we'll do all 3 on clean context" (00:30 PT ruling; findings 3e18a04).

What was done

1. zpool scrub tank started 00:29:28 PT, 6.84 TiB at ~1.4 GB/s. Finished 02:02:12 PT: scrub repaired 0B in 01:32:44 with 0 errors. zpool clear tank applied 02:02:38; nvme7n1 CKSUM 2 → 0, pool ONLINE, no known data errors. Seats kept serving throughout (1.25–1.43 GB/s issue rate).
2. nvme-cli installed (2.13-2) and every drive's SMART read. nvme7 = Dell PM1725b 1.6 TB, sn S47VNY0K600221, fw 1.2.0, critical_warning 0, percentage_used 2%, spare 100%, media_errors 2084 (= num_err_log_entries), 3083 unsafe shutdowns / 3093 power cycles vs ~200 on the six S5CU… siblings. The error-log entries all decode as "Successful Completion" (status 0) — the drive logs them but they carry no LBA. Baseline media_errors 2084 at 00:32 PT; still 2084 at 02:02 PT after the scrub read the whole 6.84 TiB — zero growth, so the counter is prior-life history, not an active fault; the 2 CKSUM were the stale-block artefact of the 09-05 late resilver, not media. Full table in servers/ana-ml2/README.md.
3. zroot 91% → 73%: docker image prune (dangling only) reclaimed 80.19 GB, docker builder prune 36.54 GB. 62 → ~40 images; the tagged idle set (vLLM v0.22–v0.27.1 + 4 nightlies, sglang, aeon-trainer, vibevoice 55 GB, kokoro ×2, infinity, parakeet) was deliberately left — several are rollback seats. Two dangling volumes (parakeet_parakeet_cache, searxng_searxng-data, 5 GB) also left; not asked for.

Why tank had no scrub since 04-12 — nvme7 was physically missing for 4½ months

journalctl --list-boots + per-boot kernel logs: boots on 04-23 (×2), 05-06, 06-12 and 08-26 each enumerated SEVEN NVMe controllers; only the 09-05 14:26 boot found eight. PCIe downstream port 0000:02:04.0 (slot 0-5) bridged bus 07 every time but nothing answered on it. So from ≤04-23 to 09-05 tank was raidz2 DEGRADED with nvme7n1 absent — one parity margin left, unnoticed. Debian's /usr/lib/zfs-linux/scrub and …/trim filter zpool list -o health for ONLINE only, which is exactly why the second-Sunday cron scrubbed zroot (08-09) and skipped tank, and why the first-Sunday trim also skipped tank May–Aug and resumed 09-06. The 09-05 cold boot (clean systemd shutdown 14:02, power back 14:26) re-seated the drive; ZFS resilvered 638 GB (≈4 months of writes) in 1h10 and logged 2 CKSUM on nvme7n1 at import (stale blocks read before the resilver caught up). Nobody was told: ZED_EMAIL_ADDR=root, and there is no MTA on the box (/var/mail empty, no exim/postfix/msmtp).

Follow-ups (surfaced, NOT done — operator's call)

• Pool-health alerting. A DEGRADED pool for 4½ months is a monitoring hole. Cheapest fix: a Beszel/Uptime-Kuma style check on zpool status -x or zpool list -H -o health across the ZFS hosts, or point ZED at a real notifier.
• nvme7 replace-vs-keep. Slot 0-5 dropped a drive for months and the drive carries 2084 lifetime media errors and a flapping-grade power-cycle count. If the scrub was clean, keep it but watch media_errors at every visit; growth ⇒ replace (any PM1725b 1.6 TB or larger; zpool replace tank nvme7n1 <new>). Also worth a reseat / cable check on slot 0-5 at the next hands-on visit — the fault may be the bay, not the drive.
• Boot-time import race. Every boot logs several vdevs UNAVAIL then ONLINE within the same second plus a vdev.no_replicas event — the import fires before all NVMes settle. Harmless so far (raidz2 absorbs it) but it is the same pattern that produced the 09-05 checksum errors. A zfs-import After=/udev-settle tweak would close it.