diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index ef54997..fdf4e87 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -125,8 +125,10 @@ bouncer failing closed on slow LAPI answers. Self-cleared. `/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself. - ⚠ For every held engine bump: if `/etc/docker/daemon.json` has `live-restore: true`, an engine upgrade without a reboot leaves containers that bind-mount `/var/run/docker.sock` on a stale socket. Restart them afterwards (see servers/ana-docker/README.md). -- Docker engine bumps held on nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-pteradactyl - (ana-docker → 29.8.0 and pfi-gx10 → 29.6 done 10-05). +- Docker engine bumps still held on nh3-docker (29.4.1; the bus + NH3 DNS), esh-docker-vm (29.4.1; ESH DNS + homepage + wedge risk) and pfi-pteradactyl (docker.io; the live game server). Done 10-05: ana-docker → 29.8.2 (bookworm repo), + pfi-gx10 → 29.6.2, and ~1410–1415 esh-ml1, nh3-ml1, nh3-dev, corviduo-dev → 29.8.2, each verified (same containers, + health endpoints, the embedding routes back on their primaries). NVIDIA/CUDA holds on the ml1s were kept. - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it. - CrowdSec bouncer `updateMaxFailure` (0 = block on the first failed LAPI pull): infra-ops leans toward ~3. diff --git a/persistent-memory.md b/persistent-memory.md index 3aed3ae..88545cd 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) -- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). +- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm, pfi-pteradactyl (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev done 10-05 ~1415); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`