# ana-ml2 pool health — the three deferred actions LANDED, and the root cause of the missed scrubs (2026-09-09)

Executed 2026-09-09 00:29–02:02 PT via `playbooks/ana-ml2-pool-health.yaml` (elway, rerunnable) plus
hand polling. Operator pre-approval: "we'll do all 3 on clean context" (00:30 PT ruling; findings `3e18a04`).

## What was done

1. **`zpool scrub tank`** started 00:29:28 PT, 6.84 TiB at ~1.4 GB/s. **Finished 02:02:12 PT: `scrub repaired 0B in 01:32:44 with 0 errors`.** `zpool clear tank` applied 02:02:38; nvme7n1 CKSUM 2 → 0, pool ONLINE, no known data errors. Seats kept serving throughout (1.25–1.43 GB/s issue rate).
2. **`nvme-cli` installed** (2.13-2) and every drive's SMART read. nvme7 = Dell PM1725b 1.6 TB, sn
   `S47VNY0K600221`, fw 1.2.0, `critical_warning 0`, `percentage_used 2%`, spare 100%, **`media_errors 2084`
   (= `num_err_log_entries`)**, 3083 unsafe shutdowns / 3093 power cycles vs ~200 on the six `S5CU…` siblings.
   The error-log entries all decode as "Successful Completion" (status 0) — the drive logs them but they carry no
   LBA. Baseline `media_errors` 2084 at 00:32 PT; **still 2084 at 02:02 PT after the scrub read the whole 6.84 TiB — zero growth**, so the counter is prior-life history, not an active fault; the 2 CKSUM were the stale-block artefact of the 09-05 late resilver, not media. Full table in `servers/ana-ml2/README.md`.
3. **zroot 91% → 73%**: `docker image prune` (dangling only) reclaimed 80.19 GB, `docker builder prune` 36.54 GB.
   62 → ~40 images; the tagged idle set (vLLM v0.22–v0.27.1 + 4 nightlies, sglang, aeon-trainer, vibevoice 55 GB,
   kokoro ×2, infinity, parakeet) was deliberately left — several are rollback seats. Two dangling volumes
   (`parakeet_parakeet_cache`, `searxng_searxng-data`, 5 GB) also left; not asked for.

## Why tank had no scrub since 04-12 — nvme7 was physically missing for 4½ months

`journalctl --list-boots` + per-boot kernel logs: **boots on 04-23 (×2), 05-06, 06-12 and 08-26 each enumerated
SEVEN NVMe controllers; only the 09-05 14:26 boot found eight.** PCIe downstream port `0000:02:04.0` (slot 0-5)
bridged bus 07 every time but nothing answered on it. So from ≤04-23 to 09-05 `tank` was raidz2 **DEGRADED with
nvme7n1 absent** — one parity margin left, unnoticed. Debian's `/usr/lib/zfs-linux/scrub` and `…/trim` filter
`zpool list -o health` for `ONLINE` only, which is exactly why the second-Sunday cron scrubbed `zroot` (08-09) and
skipped `tank`, and why the first-Sunday trim also skipped tank May–Aug and resumed 09-06. The 09-05 cold boot
(clean systemd shutdown 14:02, power back 14:26) re-seated the drive; ZFS resilvered 638 GB (≈4 months of
writes) in 1h10 and logged 2 CKSUM on nvme7n1 at import (stale blocks read before the resilver caught up).
**Nobody was told:** `ZED_EMAIL_ADDR=root`, and there is no MTA on the box (`/var/mail` empty, no
exim/postfix/msmtp).

## Follow-ups (surfaced, NOT done — operator's call)

- **Pool-health alerting.** A DEGRADED pool for 4½ months is a monitoring hole. Cheapest fix: a Beszel/Uptime-Kuma
  style check on `zpool status -x` or `zpool list -H -o health` across the ZFS hosts, or point ZED at a real notifier.
- **nvme7 replace-vs-keep.** Slot 0-5 dropped a drive for months and the drive carries 2084 lifetime media errors
  and a flapping-grade power-cycle count. If the scrub was clean, keep it but watch `media_errors` at every
  visit; growth ⇒ replace (any PM1725b 1.6 TB or larger; `zpool replace tank nvme7n1 <new>`). Also worth a reseat
  / cable check on slot 0-5 at the next hands-on visit — the fault may be the bay, not the drive.
- **Boot-time import race.** Every boot logs several vdevs UNAVAIL then ONLINE within the same second plus a
  `vdev.no_replicas` event — the import fires before all NVMes settle. Harmless so far (raidz2 absorbs it) but it
  is the same pattern that produced the 09-05 checksum errors. A `zfs-import` `After=`/udev-settle tweak would
  close it.