ops(ana-docker): docker-ce from bookworm repo (29.8.2); live-restore stale-socket lesson; docs

2026-10-05 13:31:30 -07:00
parent 258a3eea68
commit 5301ede9d2
3 changed files with 11 additions and 0 deletions
@@ -123,6 +123,8 @@ bouncer failing closed on slow LAPI answers. Self-cleared.
  ⚠ ana-docker's Docker apt source points at `bullseye` on a bookworm host (it installs bullseye builds). It works, but
  repoint it at bookworm deliberately some day. Still pending: nh3-dev (reboot + engine). None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer
  `/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself.
 - ⚠ For every held engine bump: if `/etc/docker/daemon.json` has `live-restore: true`, an engine upgrade without a reboot leaves
  containers that bind-mount `/var/run/docker.sock` on a stale socket. Restart them afterwards (see servers/ana-docker/README.md).
 - Docker engine bumps held on nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-pteradactyl
  (ana-docker → 29.8.0 and pfi-gx10 → 29.6 done 10-05).
 - fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it.
@@ -302,6 +302,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 ## Recent decisions
 - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
 - `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
 - `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
 - `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md`
 - `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result
@@ -7,6 +7,14 @@ General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0
 - **LAN IP:** 10.250.50.70
 - **FQDN:** `ana-docker.phasefinal.com`
 - **SSH:** standard port 22
 - **Docker CE from the bookworm repo since 2026-10-05** (29.8.2). Until then `/etc/apt/sources.list.d/docker.list` said
  `bullseye` on this bookworm host. Pre-change copy: `/root/docker.list.pre-bookworm-20261005`.
 - ⚠ **`live-restore: true` + bind-mounted `/var/run/docker.sock` = stale sockets after any engine upgrade WITHOUT a
  reboot.** Containers keep running through the dockerd restart, but dockerd recreates the socket, so every container that
  bind-mounts the socket file still points at the old, deleted inode ("Cannot connect to the Docker daemon"). After an
  engine bump, restart them: crowdsec FIRST, then traefik, dozzle, beszel-agent, dockge, gitea-runner (find them with
  `docker inspect` Mounts Source=/var/run/docker.sock). Found 2026-10-05 when dozzle went unhealthy and traefik's
  docker provider stopped seeing changes.
 - **Traefik entrypoints:** 80/443 terminate here; cert resolver `anaprod` (Let's Encrypt)
  - ⚠ **`readTimeout: 10m` on both entrypoints since 2026-10-05.** Traefik v3 defaults it to 60s, which cut every
    request body slower than 60 s. Measured: skaldsong's 2.5 GB registry blob PUTs died at exactly 60.0 s (504, then 499),