vh
5de5583762
- playbooks/ana-ml2-pool-health.yaml: rerunnable elway play (scrub-if-idle, nvme-cli, dangling-image + builder prune; never prune -a) - servers/ana-ml2/README.md: 8-drive PM1725b inventory with SMART counters, nvme7 absence + alerting-gap note - persistent-memory: deferred entry closed, outcome + follow-ups (pool-health alerting, nvme7/slot 0-5 watch, boot import race)
44 lines
3.9 KiB
Markdown
44 lines
3.9 KiB
Markdown
# ana-ml2 pool health — the three deferred actions LANDED, and the root cause of the missed scrubs (2026-09-09)
|
||
|
||
Executed 2026-09-09 00:29–02:02 PT via `playbooks/ana-ml2-pool-health.yaml` (elway, rerunnable) plus
|
||
hand polling. Operator pre-approval: "we'll do all 3 on clean context" (00:30 PT ruling; findings `3e18a04`).
|
||
|
||
## What was done
|
||
|
||
1. **`zpool scrub tank`** started 00:29:28 PT, 6.84 TiB at ~1.4 GB/s. **Finished 02:02:12 PT: `scrub repaired 0B in 01:32:44 with 0 errors`.** `zpool clear tank` applied 02:02:38; nvme7n1 CKSUM 2 → 0, pool ONLINE, no known data errors. Seats kept serving throughout (1.25–1.43 GB/s issue rate).
|
||
2. **`nvme-cli` installed** (2.13-2) and every drive's SMART read. nvme7 = Dell PM1725b 1.6 TB, sn
|
||
`S47VNY0K600221`, fw 1.2.0, `critical_warning 0`, `percentage_used 2%`, spare 100%, **`media_errors 2084`
|
||
(= `num_err_log_entries`)**, 3083 unsafe shutdowns / 3093 power cycles vs ~200 on the six `S5CU…` siblings.
|
||
The error-log entries all decode as "Successful Completion" (status 0) — the drive logs them but they carry no
|
||
LBA. Baseline `media_errors` 2084 at 00:32 PT; **still 2084 at 02:02 PT after the scrub read the whole 6.84 TiB — zero growth**, so the counter is prior-life history, not an active fault; the 2 CKSUM were the stale-block artefact of the 09-05 late resilver, not media. Full table in `servers/ana-ml2/README.md`.
|
||
3. **zroot 91% → 73%**: `docker image prune` (dangling only) reclaimed 80.19 GB, `docker builder prune` 36.54 GB.
|
||
62 → ~40 images; the tagged idle set (vLLM v0.22–v0.27.1 + 4 nightlies, sglang, aeon-trainer, vibevoice 55 GB,
|
||
kokoro ×2, infinity, parakeet) was deliberately left — several are rollback seats. Two dangling volumes
|
||
(`parakeet_parakeet_cache`, `searxng_searxng-data`, 5 GB) also left; not asked for.
|
||
|
||
## Why tank had no scrub since 04-12 — nvme7 was physically missing for 4½ months
|
||
|
||
`journalctl --list-boots` + per-boot kernel logs: **boots on 04-23 (×2), 05-06, 06-12 and 08-26 each enumerated
|
||
SEVEN NVMe controllers; only the 09-05 14:26 boot found eight.** PCIe downstream port `0000:02:04.0` (slot 0-5)
|
||
bridged bus 07 every time but nothing answered on it. So from ≤04-23 to 09-05 `tank` was raidz2 **DEGRADED with
|
||
nvme7n1 absent** — one parity margin left, unnoticed. Debian's `/usr/lib/zfs-linux/scrub` and `…/trim` filter
|
||
`zpool list -o health` for `ONLINE` only, which is exactly why the second-Sunday cron scrubbed `zroot` (08-09) and
|
||
skipped `tank`, and why the first-Sunday trim also skipped tank May–Aug and resumed 09-06. The 09-05 cold boot
|
||
(clean systemd shutdown 14:02, power back 14:26) re-seated the drive; ZFS resilvered 638 GB (≈4 months of
|
||
writes) in 1h10 and logged 2 CKSUM on nvme7n1 at import (stale blocks read before the resilver caught up).
|
||
**Nobody was told:** `ZED_EMAIL_ADDR=root`, and there is no MTA on the box (`/var/mail` empty, no
|
||
exim/postfix/msmtp).
|
||
|
||
## Follow-ups (surfaced, NOT done — operator's call)
|
||
|
||
- **Pool-health alerting.** A DEGRADED pool for 4½ months is a monitoring hole. Cheapest fix: a Beszel/Uptime-Kuma
|
||
style check on `zpool status -x` or `zpool list -H -o health` across the ZFS hosts, or point ZED at a real notifier.
|
||
- **nvme7 replace-vs-keep.** Slot 0-5 dropped a drive for months and the drive carries 2084 lifetime media errors
|
||
and a flapping-grade power-cycle count. If the scrub was clean, keep it but watch `media_errors` at every
|
||
visit; growth ⇒ replace (any PM1725b 1.6 TB or larger; `zpool replace tank nvme7n1 <new>`). Also worth a reseat
|
||
/ cable check on slot 0-5 at the next hands-on visit — the fault may be the bay, not the drive.
|
||
- **Boot-time import race.** Every boot logs several vdevs UNAVAIL then ONLINE within the same second plus a
|
||
`vdev.no_replicas` event — the import fires before all NVMes settle. Harmless so far (raidz2 absorbs it) but it
|
||
is the same pattern that produced the 09-05 checksum errors. A `zfs-import` `After=`/udev-settle tweak would
|
||
close it.
|