ops(pfi-pteradactyl): docker.io/containerd upgraded (manual dockerd + wings restart); docs

2026-10-05 14:18:57 -07:00
parent 58bd802872
commit 3429b5c55d
3 changed files with 7 additions and 5 deletions
@@ -125,8 +125,8 @@ bouncer failing closed on slow LAPI answers. Self-cleared.
  `/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself.
 - ⚠ For every held engine bump: if `/etc/docker/daemon.json` has `live-restore: true`, an engine upgrade without a reboot leaves
  containers that bind-mount `/var/run/docker.sock` on a stale socket. Restart them afterwards (see servers/ana-docker/README.md).
- Docker engine bumps still held on nh3-docker (29.4.1; the bus + NH3 DNS), esh-docker-vm (29.4.1; ESH DNS + homepage
+- Docker engine bumps still held on nh3-docker (29.4.1; the bus + NH3 DNS) and esh-docker-vm (29.4.1; ESH DNS + homepage
-  wedge risk) and pfi-pteradactyl (docker.io; the live game server). Done 10-05: ana-docker → 29.8.2 (bookworm repo),
+  wedge risk). pfi-pteradactyl's docker.io upgraded 10-05 1418 (needed a manual dockerd restart plus a wings restart). Done 10-05: ana-docker → 29.8.2 (bookworm repo),
  pfi-gx10 → 29.6.2, and ~1410–1415 esh-ml1, nh3-ml1, nh3-dev, corviduo-dev → 29.8.2, each verified (same containers,
  health endpoints, the embedding routes back on their primaries). NVIDIA/CUDA holds on the ml1s were kept.
 - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it.
@@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT)
- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm, pfi-pteradactyl (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev done 10-05 ~1415); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05).
+- **Waiting on Prime (2026-10-05, updated 1100):** reboot + engine bump for nh3-dev (all other kernel reboots done 10-05 ~1100–1133, incl. ana-docker with Docker 29.8.0; pfi-gx10 updated to DGX OTA 7.6 / kernel 7.0); Docker engine bumps still held on nh3-docker, esh-docker-vm (esh-ml1, nh3-ml1, nh3-dev, corviduo-dev, pfi-pteradactyl done 10-05 ~1415–1418); fv-ml1's held ZFS/toolkit/kernel; pfi-pteradactyl fixed, patched, engine done; pfi-tacticalrmm patched; CrowdSec bouncer updateMaxFailure set to 3 (Prime, 10-05).
 - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%.
 - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
 - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
@@ -20,8 +20,10 @@ Pterodactyl game-server panel at the Anaheim colo.
  internal). Access is LAN-only by IP, which is fine. Links the panel generates (emails, for example) carry the
  dead name.
 - **Patched 2026-10-05** (163 packages, MariaDB 11.7 → 11.8.9). Pre-patch DB dump: `/root/panel-db-pre-patch-20261005.sql.gz`.
-  **`docker.io`, `containerd` and `runc` are HELD**: an engine upgrade restarts dockerd, which would stop the running
+  Kernel 6.1.0-53 is active after the 10-05 ~1100 reboot. Docker (Debian `docker.io` 20.10.24 +b6, containerd deb12u3) was
-  game server (wings does not restart it). Bump them when no one is playing. Kernel 6.1.0-53 needs a reboot (deferred).
+  upgraded 10-05 1418 on Prime's go. ⚠ **Debian's `docker.io` postinst does NOT restart dockerd**: the old binary keeps
  running (`/proc/<pid>/exe` shows `(deleted)`) until `systemctl restart docker`. That restart stops the game server;
  `systemctl restart wings` brings it back (wings restores the servers that were running, as it does at boot).
 ## Infrastructure