runbook(park 100): pfi-tacticalrmm patched; reboot list now 8 hosts

2026-10-05 10:55:27 -07:00
parent aa86c37954
commit a808b349b8
2 changed files with 6 additions and 4 deletions
@@ -106,14 +106,16 @@ bouncer failing closed on slow LAPI answers. Self-cleared.
 - ~~pfi-pteradactyl~~ **FIXED + patched 2026-10-05 ~1045 (Prime):** "apache2 failed / panel down" was a leftover apache2
  losing :80 to nginx; the panel (nginx + php-fpm) was up all along (the curl that read 000 used https or the dead
  `ptero.phasefinal.com`). apache2 masked; 163 → 0; Docker engine held (live game server); reboot deferred.
- pfi-tacticalrmm: certbot renewal failed 2026-10-04 1831 (web + MeshCentral fine). 164 pending.
+- ~~pfi-tacticalrmm~~ **cert fixed (infra-ops) + patched 2026-10-05 ~1055 (infra-hermes):** 164 → 0, postgres restarted,
+  RMM/daphne/nats/meshcentral healthy, MeshCentral never restarted (AMT CIRA :4433 untouched), `certbot renew --dry-run`
+  passes after the patch. Kernel 6.1.0-53 needs a reboot (deferred).
 - pfi-gx10: wants a `dgx-release` 7.5 → 7.6 upgrade, a new bootloader package (`nvidia-spark-grub-kho`) and kernel
  7.0.0-1019-nvidia. 212 pending.
 - ana-nas (NFS SPOF) and irv-ml1 (Prime: off-limits): out of scope.

 **Deferred to a daytime window:**
- reboots to activate new kernels: ana-docker, nh3-dev, nh3-extdev, corviduo-dev, vm-esh-nas, esh-vm-db. None
-  builds NVIDIA with DKMS.
+- reboots to activate new kernels: ana-docker, nh3-dev, nh3-extdev, corviduo-dev, vm-esh-nas, esh-vm-db, plus
+  pfi-tacticalrmm and pfi-pteradactyl (added later on 10-05). None builds NVIDIA with DKMS.
 - Docker engine bumps held on ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-gx10.
 - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it.
 - CrowdSec bouncer `updateMaxFailure` (0 = block on the first failed LAPI pull): infra-ops leans toward ~3.
@@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca

 ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT)

- **Waiting on Prime (2026-10-05 0355):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-gx10 (dgx-release) parked; pfi-pteradactyl fixed + patched; pfi-tacticalrmm patching via infra-hermes; CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause).
+- **Waiting on Prime (2026-10-05, updated 1100):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db/pfi-tacticalrmm/pfi-pteradactyl; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-gx10 (dgx-release) parked; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause).
 - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%.
 - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
 - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`