From 3429b5c55d961ecb3bed14c702f3e0c338543e18 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 14:18:57 -0700 Subject: [PATCH] ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs --- docs/runbooks/fleet-guest-patch-2026-10-05.md | 4 ++-- persistent-memory.md | 2 +- servers/pfi-pteradactyl/README.md | 6 ++++-- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index fdf4e87..f68a926 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -125,8 +125,8 @@ bouncer failing closed on slow LAPI answers. Self-cleared. `/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself. - ⚠ For every held engine bump: if `/etc/docker/daemon.json` has `live-restore: true`, an engine upgrade without a reboot leaves containers that bind-mount `/var/run/docker.sock` on a stale socket. Restart them afterwards (see servers/ana-docker/README.md). -- Docker engine bumps still held on nh3-docker (29.4.1; the bus + NH3 DNS), esh-docker-vm (29.4.1; ESH DNS + homepage - wedge risk) and pfi-pteradactyl (docker.io; the live game server). Done 10-05: ana-docker → 29.8.2 (bookworm repo), +- Docker engine bumps still held on nh3-docker (29.4.1; the bus + NH3 DNS) and esh-docker-vm (29.4.1; ESH DNS + homepage + wedge risk). pfi-pteradactyl's docker.io upgraded 10-05 1418 (needed a manual dockerd restart plus a wings restart). Done 10-05: ana-docker → 29.8.2 (bookworm repo), pfi-gx10 → 29.6.2, and ~1410–1415 esh-ml1, nh3-ml1, nh3-dev, corviduo-dev → 29.8.2, each verified (same containers, health endpoints, the embedding routes back on their primaries). NVIDIA/CUDA holds on the ml1s were kept. - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it. diff --git a/persistent-memory.md b/persistent-memory.md index 88545cd..572a1c3 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) -- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm, pfi-pteradactyl (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev done 10-05 ~1415); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). +- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev, pfi-pteradactyl done 10-05 ~1415–1418); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed, patched, engine done; pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05). - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` diff --git a/servers/pfi-pteradactyl/README.md b/servers/pfi-pteradactyl/README.md index 03422c5..21ef087 100644 --- a/servers/pfi-pteradactyl/README.md +++ b/servers/pfi-pteradactyl/README.md @@ -20,8 +20,10 @@ Pterodactyl game-server panel at the Anaheim colo. internal). Access is LAN-only by IP, which is fine. Links the panel generates (emails, for example) carry the dead name. - **Patched 2026-10-05** (163 packages, MariaDB 11.7 → 11.8.9). Pre-patch DB dump: `/root/panel-db-pre-patch-20261005.sql.gz`. - **`docker.io`, `containerd` and `runc` are HELD**: an engine upgrade restarts dockerd, which would stop the running - game server (wings does not restart it). Bump them when no one is playing. Kernel 6.1.0-53 needs a reboot (deferred). + Kernel 6.1.0-53 is active after the 10-05 ~1100 reboot. Docker (Debian `docker.io` 20.10.24 +b6, containerd deb12u3) was + upgraded 10-05 1418 on Prime's go. ⚠ **Debian's `docker.io` postinst does NOT restart dockerd**: the old binary keeps + running (`/proc//exe` shows `(deleted)`) until `systemctl restart docker`. That restart stops the game server; + `systemctl restart wings` brings it back (wings restores the servers that were running, as it does at boot). ## Infrastructure