vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

ops(ana-ml2): pool-health actions landed — tank scrubbed clean + cleared, nvme-cli SMART inventory, zroot 91→73%; root cause of missed scrubs = nvme7 absent 04-23→09-05 (pool DEGRADED, ZED mail unrouted)

Browse Source 
- playbooks/ana-ml2-pool-health.yaml: rerunnable elway play (scrub-if-idle, nvme-cli, dangling-image + builder prune; never prune -a)
- servers/ana-ml2/README.md: 8-drive PM1725b inventory with SMART counters, nvme7 absence + alerting-gap note
- persistent-memory: deferred entry closed, outcome + follow-ups (pool-health alerting, nvme7/slot 0-5 watch, boot import race)
This commit is contained in:
vh
2026-09-09 02:03:11 -07:00
parent 5ad948bf31
commit 5de5583762
4 changed files with 122 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/ana-ml2/README.md
+28 -1
View File
@@ -16,10 +16,37 @@ Primary AI inference host for PFI.
- **CPU:** AMD EPYC 9254 24-core (96 threads)
- **RAM:** 566 GB
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
- **OS:** Debian 13 (trixie), kernel 6.12.x
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the `S5CU…` six (fw 1.2.2) and the
`S47V…` pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| **nvme7** | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | **2084** | 2% |
⚠ **nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05** (kernel enumerated 7
NVMes per boot; PCIe downstream port `02:04.0` had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing `tank` was DEGRADED, and Debian's `zfsutils-linux` cron
(`/usr/lib/zfs-linux/{scrub,trim}`) only touches pools whose health is `ONLINE`, so tank
got **no scrub and no trim from 04-12 to 09-06**. ZED's `ZED_EMAIL_ADDR=root` has no
MTA behind it, so the 4½-month degradation alerted nobody. `media_errors=2084` on
nvme7 is a lifetime counter; the 2026-09-09 scrub is the first fresh measurement
(baseline 2084 at 00:32 PT — compare after any future event, growth = replace).
## Key paths
| Path | Purpose |