diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index 44c1087..ef54997 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -123,6 +123,8 @@ bouncer failing closed on slow LAPI answers. Self-cleared. ⚠ ana-docker's Docker apt source points at `bullseye` on a bookworm host (it installs bullseye builds). It works, but repoint it at bookworm deliberately some day. Still pending: nh3-dev (reboot + engine). None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer `/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself. +- ⚠ For every held engine bump: if `/etc/docker/daemon.json` has `live-restore: true`, an engine upgrade without a reboot leaves + containers that bind-mount `/var/run/docker.sock` on a stale socket. Restart them afterwards (see servers/ana-docker/README.md). - Docker engine bumps held on nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-pteradactyl (ana-docker → 29.8.0 and pfi-gx10 → 29.6 done 10-05). - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it. diff --git a/persistent-memory.md b/persistent-memory.md index 04144bf..3aed3ae 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -302,6 +302,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md` +- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md` - `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md` - `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md` - `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result diff --git a/servers/ana-docker/README.md b/servers/ana-docker/README.md index 86e1e16..403ed28 100644 --- a/servers/ana-docker/README.md +++ b/servers/ana-docker/README.md @@ -7,6 +7,14 @@ General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0 - **LAN IP:** 10.250.50.70 - **FQDN:** `ana-docker.phasefinal.com` - **SSH:** standard port 22 +- **Docker CE from the bookworm repo since 2026-10-05** (29.8.2). Until then `/etc/apt/sources.list.d/docker.list` said + `bullseye` on this bookworm host. Pre-change copy: `/root/docker.list.pre-bookworm-20261005`. +- ⚠ **`live-restore: true` + bind-mounted `/var/run/docker.sock` = stale sockets after any engine upgrade WITHOUT a + reboot.** Containers keep running through the dockerd restart, but dockerd recreates the socket, so every container that + bind-mounts the socket file still points at the old, deleted inode ("Cannot connect to the Docker daemon"). After an + engine bump, restart them: crowdsec FIRST, then traefik, dozzle, beszel-agent, dockge, gitea-runner (find them with + `docker inspect` Mounts Source=/var/run/docker.sock). Found 2026-10-05 when dozzle went unhealthy and traefik's + docker provider stopped seeing changes. - **Traefik entrypoints:** 80/443 terminate here; cert resolver `anaprod` (Let's Encrypt) - ⚠ **`readTimeout: 10m` on both entrypoints since 2026-10-05.** Traefik v3 defaults it to 60s, which cut every request body slower than 60 s. Measured: skaldsong's 2.5 GB registry blob PUTs died at exactly 60.0 s (504, then 499),