ops(ana-ml2): pool-health actions landed — tank scrubbed clean + cleared, nvme-cli SMART inventory, zroot 91→73%; root cause of missed scrubs = nvme7 absent 04-23→09-05 (pool DEGRADED, ZED mail unrouted)

- playbooks/ana-ml2-pool-health.yaml: rerunnable elway play (scrub-if-idle, nvme-cli, dangling-image + builder prune; never prune -a) - servers/ana-ml2/README.md: 8-drive PM1725b inventory with SMART counters, nvme7 absence + alerting-gap note - persistent-memory: deferred entry closed, outcome + follow-ups (pool-health alerting, nvme7/slot 0-5 watch, boot import race)
2026-09-09 02:03:11 -07:00
parent 5ad948bf31
commit 5de5583762
4 changed files with 122 additions and 5 deletions
@@ -0,0 +1,44 @@
+# ana-ml2 pool-health actions (operator-approved 2026-09-09, "all 3 on clean context").
+#
+#   1. kick a scrub of `tank` (2 CKSUM on nvme7n1 after the 09-05 boot resilver,
+#      no scrub since 04-12) — polled separately; `zpool clear` only after a
+#      clean pass, by hand, never in this playbook.
+#   2. install nvme-cli so nvme7's SMART counters can be read.
+#   3. reclaim zroot (91%): dangling images + build cache ONLY. Never `prune -a`
+#      here — tagged idle images include rollback seats.
+#
+# Rerunnable: a second run shows the scrub step skipped (in progress / done
+# within the hour is not detected — the guard is "not already scrubbing"),
+# nvme-cli ok, prunes ok with 0 B reclaimed.
+
+steps:
+  - name: Start scrub of tank (skip if one is already running)
+    shell: zpool scrub tank
+    sudo: true
+    when: "! zpool status tank | grep -q 'scrub in progress'"
+
+  - name: Install nvme-cli
+    shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -q nvme-cli
+    sudo: true
+    creates: /usr/sbin/nvme
+
+  - name: Prune dangling docker images (NOT -a)
+    shell: docker image prune -f | tail -1
+    sudo: true
+
+  - name: Prune docker build cache
+    shell: docker builder prune -f | tail -1
+    sudo: true
+
+verify:
+  - name: Scrub is in progress or already completed today
+    shell: "zpool status tank | grep -Eq 'scrub in progress|scrub repaired .* on .*$(date +%b\\ *%e)'"
+    changed_when: "false"
+
+  - name: nvme binary present
+    shell: test -x /usr/sbin/nvme
+    changed_when: "false"
+
+  - name: zroot usage after prune
+    shell: df -h / | tail -1
+    changed_when: "false"