From 258a3eea68dc762013d8215ec988e0f2188da784 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 11:34:00 -0700 Subject: [PATCH] docs: pfi-gx10 update recorded; held-engine list and Waiting-on-Prime refreshed --- docs/runbooks/fleet-guest-patch-2026-10-05.md | 9 ++++++--- persistent-memory.md | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index 72b5bf2..44c1087 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -109,8 +109,10 @@ bouncer failing closed on slow LAPI answers. Self-cleared. - ~~pfi-tacticalrmm~~ **cert fixed (infra-ops) + patched 2026-10-05 ~1055 (infra-hermes):** 164 → 0, postgres restarted, RMM/daphne/nats/meshcentral healthy, MeshCentral never restarted (AMT CIRA :4433 untouched), `certbot renew --dry-run` passes after the patch. Kernel 6.1.0-53 needs a reboot (deferred). -- pfi-gx10: wants a `dgx-release` 7.5 → 7.6 upgrade, a new bootloader package (`nvidia-spark-grub-kho`) and kernel - 7.0.0-1019-nvidia. 212 pending. +- ~~pfi-gx10~~ **updated + rebooted 2026-10-05 ~1120 (Prime):** DGX OTA 7.5.0 → 7.6.0, kernel 6.17.0-1032 → 7.0.0-1019-nvidia + (its prebuilt `linux-modules-nvidia-580-open-7.0.0-1019` came with it, so this was not fv-ml1's DKMS failure), driver + 580.173 → 580.178.04, Docker 29.2 → 29.6. Post: nvidia-smi OK, CDI lists the GPU, 0 failed units. nvidia-cdi-refresh + fails before the reboot (driver/library mismatch) and clears itself after. - ana-nas (NFS SPOF) and irv-ml1 (Prime: off-limits): out of scope. **Deferred to a daytime window:** @@ -121,7 +123,8 @@ bouncer failing closed on slow LAPI answers. Self-cleared. ⚠ ana-docker's Docker apt source points at `bullseye` on a bookworm host (it installs bullseye builds). It works, but repoint it at bookworm deliberately some day. Still pending: nh3-dev (reboot + engine). None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer `/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself. -- Docker engine bumps held on ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-gx10. +- Docker engine bumps held on nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-pteradactyl + (ana-docker → 29.8.0 and pfi-gx10 → 29.6 done 10-05). - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it. - CrowdSec bouncer `updateMaxFailure` (0 = block on the first failed LAPI pull): infra-ops leans toward ~3. diff --git a/persistent-memory.md b/persistent-memory.md index 1f6fce7..04144bf 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) -- **Waiting on Prime (2026-10-05, updated 1100):** reboots for ana-docker/nh3-dev/vm-esh-nas (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl rebooted clean 10-05 ~1100; pfi-tacticalrmm by Prime + TRMM 1.5.2); a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-gx10 (dgx-release) parked; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause). +- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`