Fleet guest patch campaign, 2026-10-05 (henge park 100)

Ordered by Prime, 2026-10-05 ~0145: bring every host current. Low-blast-radius hosts go first. irv-ml1 is OFF-LIMITS: do not patch it, do not touch apt on it. Reboots may wait. Verify every host before and after. Anything that looks dicey is parked for Prime, not pushed through.

Executor: infra-hermes. Owner of the plan / escalation: infra-ops. fv-ml1 was patched by infra-ops in the same window as its Docker storage move (its holds are listed at the bottom).

Rules for every host

Claim, then record. scripts/ops-log claim <host> apt --why "park 100 patch", then scripts/ops-log record --host <host> --action patch --target apt --detail "<before→after, holds, reboot y/n>" and release.
Pre-check, and save the output so it can be compared after: systemctl --failed --no-legend, docker ps --format '{{.Names}} {{.Status}}' (where Docker exists), the host's own health check from the table, df -h / (need ≥ 2 GB free), and dpkg --audit (must be empty). If the host is already broken before you start, do not patch. Park it.
apt-get update must be clean. A dead third-party repo is fixable (comment out its sources entry, and note it). Any other error means park the host.
Read the pending list before upgrading (apt-get -s upgrade --with-new-pkgs | grep ^Inst), and apply the holds below with apt-mark hold. Park the host instead of upgrading if the list contains any of: a Debian release change, proxmox-kernel*/pve-kernel* on a hypervisor, the ZFS stack on a root-on-ZFS host, grub*/shim*/systemd-boot* on an unusual boot setup, or anything you cannot explain.

Never run apt in-band through qm guest exec, and never in a plain ssh session. If ssh drops, dpkg is left half-configured; that happened to both PBS VMs on 10-04. Run it as a transient unit and poll it:

sudo systemd-run --unit=park100-patch --collect -E DEBIAN_FRONTEND=noninteractive -E NEEDRESTART_MODE=l \
  apt-get -q -y -o Dpkg::Options::=--force-confold upgrade --with-new-pkgs
sudo journalctl -u park100-patch -f      # or: systemctl is-active park100-patch

NEEDRESTART_MODE=l means needrestart lists services but restarts nothing.

Post-check: the same commands as step 2, and they must match. Same failed units (ideally none), same containers running and healthy, health check passing, dpkg --audit empty, apt list --upgradable showing only held packages. Note /var/run/reboot-required. Do NOT reboot: reboots are deferred to Prime.
If something breaks and is not fixed within 10 minutes: stop, leave it in the safest state, park the host with exactly what happened, and tell infra-ops. If the break is user-facing (DNS, the bus, the gateway, gitea), also tell Miranda, per the standing order.
One host at a time. Never the two DNS hosts (nh3-docker, ana-docker) in the same hour.

Holds

Docker engine (docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras) is held on every host whose containers matter: ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-gx10. An engine upgrade restarts dockerd at once, which bounces every container (the bus, DNS, the gateway, Worldtree, the embed seats). Those bumps are parked for one daytime window. Elsewhere the engine may upgrade.
NVIDIA/CUDA user-space (nvidia-* libnvidia-* cuda-* libcuda*) is held on the GPU LXCs (esh-ml1, nh3-ml1) and pfi-gx10. In an LXC it must match the host's kernel driver.
⚠ Kernels on any host that builds NVIDIA with DKMS (hold linux-image-amd64 linux-headers-amd64, or the -generic/-nvidia metas on Ubuntu/DGX): --with-new-pkgs PULLS A NEW KERNEL. On fv-ml1 tonight, 6.12.111 came in, NVIDIA 580.65.06 DKMS FAILED to build for it, and dpkg was left half-configured. Had it booted, there would be no GPU driver and every seat would be down. infra-ops purged it. Check dkms status for nvidia before patching; if it is present, hold the kernel metas and park the kernel bump.
Proxmox hosts: hold proxmox-kernel-* pve-kernel-* and the ZFS stack. If those are the only pending packages, the host counts as current; park the kernel/ZFS bump.

Order (low → high blast radius)

#	host	address	health check	notes
1	ana-filebot	10.250.50.53	`systemctl --failed`	LXC
2	pfi-pteradactyl	10.250.50.55	panel `curl -sk https://localhost` → 200/302	game panel
3	nh3-extdev	10.100.50.42	`postbox status` as infra-ops	althing wheel install lives here; do not reinstall it
4	vm-esh-nas	10.0.50.154	its containers + NFS mounts (`findmnt -t nfs,nfs4`)
5	nh3-albok	10.100.50.44	`curl -s localhost:8392/health` → `"status":"ok"`	built 10-04, likely nearly current; engine may upgrade here (single-tenant, albok restarts in ~15 s)
6	esh-matter	10.0.90.20	`systemctl --failed`; matter server port 5580 listening	Home Assistant's Matter fabric
7	pfi-tacticalrmm	10.250.50.57	RMM web UI 200; MeshCentral up	the AMT/CIRA path to the hypervisors runs through it
8	esh-vm-db	10.0.50.60	`pg_isready`; mongo port 27017	⚠ apt lists never refreshed since 2024-08. If the release is EOL or `apt update` is not clean, PARK
9	nh3-ml1	10.100.50.80	`curl -s localhost:8001/health`, `:8013/health`	engine + NVIDIA held. Second embed backend (failover only now)
10	esh-ml1	10.0.50.80	same, plus `:8003` reward seat	fleet embedding primary: engine + NVIDIA held, so no TEI restart
11	corviduo-dev	10.250.50.152	the three Worldtree `/health` (8080/8081/8082)	engine held. Finish before 0700 (the 0730 Worldtree gate run)
12	pbs-nh3, pbs-ana	10.100.50.90, 10.250.50.90	`proxmox-backup-manager datastore list`; no running task	only if something is pending (both were upgraded 10-04). Never during the 0630 sync
13	hypervisors: nh3-pve-2, nh3-pve, esh-pve, esh-nas-pve, pfi-pve		`pvesh get /cluster/resources --type vm` all expected guests running	kernel/ZFS held per above; everything else patched; no reboot
14	esh-docker-vm	10.0.50.45	AdGuard (ESH DNS) answers; Homepage `/api/services`	engine held. ⚠ comment out the dead Google Coral apt repo first (it makes `apt update` exit 100). ⚠ Do not restart homepage (it wedges: incident_homepage_mmap_lock_wedge)
15	ana-docker	10.250.50.70	gateway `:4000/health/liveliness`, gitea `/api/v1/version`, traefik up	engine held. Not in the same hour as nh3-docker
16	nh3-docker	10.100.50.40	AdGuard answers `dig @10.100.50.40 nh3-docker.nh3.internal`; post office `:8390` reachable; searxng	engine held. The bus lives here, so a broken post office cuts your own comms: check it first and last
17	nh3-dev	10.100.10.50	svos `:8770`, hermes gateway, Booth `:8090`, `postbox status`	engine held. Live agent sessions run here; needrestart in list mode only
18	pfi-gx10	10.100.50.60	`nvidia-smi`; its seats	engine + NVIDIA/CUDA held. DGX OS (Ubuntu, arm64): if it wants its own `dgx` release upgrade, PARK

Out of scope / parked up front

irv-ml1: Prime, explicitly. Do not patch.
ana-nas (CT 109): parked. It is the NFS SPOF for PBS-ANA's datastore and the rest-servers. A nfs-kernel-server or samba restart under a running backup is the failure mode. That is Prime's call.
No infra-ops access, so out of reach: pfi-postgres, ana-wg, the mesh routers + nh3-headscale, nh3-nas, the Windows VMs, PFI-Mongo, SlaveBot.
SureFire tenant hosts (sf-*, sfsrv-ana): tenant property, coordinate first. Not tonight.
fv-ml1 (done by infra-ops): patched in the Docker-move window, with the ZFS stack 2.3.2 → 2.3.9 (root is on ZFS) and nvidia-container-toolkit 1.19 → 1.20 HELD and parked for Prime.

Report

When done, or at 0700 at the latest, send infra-ops ONE message with a table: host | pending before → after | holds | reboot-required | parked + reason. Keep the pre/post outputs; they are the evidence.

Result (2026-10-05 0355, infra-hermes + infra-ops)

Patched clean (15): ana-filebot, nh3-extdev, nh3-albok (already current), esh-matter, nh3-ml1, esh-ml1, corviduo-dev, pbs-nh3, pbs-ana, the five hypervisors (already current), esh-docker-vm (dead Coral repo commented out), ana-docker, nh3-docker, nh3-dev, vm-esh-nas (engine 28 → 29, NFS mounts intact), esh-vm-db (current bookworm despite lists stale since 2024), plus fv-ml1 (infra-ops, in the Docker-move window). Pre/post checks matched everywhere. No reboots. User-facing impact: two Gitea 403 blips (0310–0314, 0342) while ana-docker's apt loaded the host. Cause: the CrowdSec bouncer failing closed on slow LAPI answers. Self-cleared.

Parked for Prime:

~~pfi-pteradactyl~~ FIXED + patched 2026-10-05 ~1045 (Prime): "apache2 failed / panel down" was a leftover apache2 losing :80 to nginx; the panel (nginx + php-fpm) was up all along (the curl that read 000 used https or the dead ptero.phasefinal.com). apache2 masked; 163 → 0; Docker engine held (live game server); reboot deferred.
~~pfi-tacticalrmm~~ cert fixed (infra-ops) + patched 2026-10-05 ~1055 (infra-hermes): 164 → 0, postgres restarted, RMM/daphne/nats/meshcentral healthy, MeshCentral never restarted (AMT CIRA :4433 untouched), certbot renew --dry-run passes after the patch. Kernel 6.1.0-53 needs a reboot (deferred).
~~pfi-gx10~~ updated + rebooted 2026-10-05 ~1120 (Prime): DGX OTA 7.5.0 → 7.6.0, kernel 6.17.0-1032 → 7.0.0-1019-nvidia (its prebuilt linux-modules-nvidia-580-open-7.0.0-1019 came with it, so this was not fv-ml1's DKMS failure), driver 580.173 → 580.178.04, Docker 29.2 → 29.6. Post: nvidia-smi OK, CDI lists the GPU, 0 failed units. nvidia-cdi-refresh fails before the reboot (driver/library mismatch) and clears itself after.
ana-nas (NFS SPOF) and irv-ml1 (Prime: off-limits): out of scope.

Deferred to a daytime window:

reboots to activate new kernels: ~~nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl~~ (rebooted 2026-10-05 1055–1109 on Prime's word; all came back clean, boot ids changed, 0 failed units). pfi-tacticalrmm rebooted + TRMM updated to 1.5.2 by Prime ~1110 (verified: 0 failed units, cert intact). vm-esh-nas rebooted ~1115 (NFS mounts + containers back). ana-docker: Docker engine 29.4.1 → 29.8.0 + reboot ~1123–1133 (Prime), 44/44 containers identical to the pre-snapshot, gateway/gitea/CrowdSec OK. ⚠ ana-docker's Docker apt source points at bullseye on a bookworm host (it installs bullseye builds). It works, but repoint it at bookworm deliberately some day. Still pending: nh3-dev (reboot + engine). None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer /health after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself.
Docker engine bumps held on nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-pteradactyl (ana-docker → 29.8.0 and pfi-gx10 → 29.6 done 10-05).
fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it.
CrowdSec bouncer updateMaxFailure (0 = block on the first failed LAPI pull): infra-ops leans toward ~3.

Lesson: four hosts were first parked as "no NOPASSWD sudo" because the ssh aliases log in as lkraven. As infra-ops@ all four had it (auto-memory feedback_ssh_alias_defaults_to_lkraven).

11 KiB Raw Blame History Unescape Escape