Fleet guest patch campaign, 2026-10-05 (henge park 100)

Ordered by Prime, 2026-10-05 ~0145: bring every host current. Low-blast-radius hosts go first. irv-ml1 is OFF-LIMITS: do not patch it, do not touch apt on it. Reboots may wait. Verify every host before and after. Anything that looks dicey is parked for Prime, not pushed through.

Executor: infra-hermes. Owner of the plan / escalation: infra-ops. fv-ml1 was patched by infra-ops in the same window as its Docker storage move (its holds are listed at the bottom).

Rules for every host

Claim, then record. scripts/ops-log claim <host> apt --why "park 100 patch", then scripts/ops-log record --host <host> --action patch --target apt --detail "<before→after, holds, reboot y/n>" and release.
Pre-check, and save the output so it can be compared after: systemctl --failed --no-legend, docker ps --format '{{.Names}} {{.Status}}' (where Docker exists), the host's own health check from the table, df -h / (need ≥ 2 GB free), and dpkg --audit (must be empty). If the host is already broken before you start, do not patch. Park it.
apt-get update must be clean. A dead third-party repo is fixable (comment out its sources entry, and note it). Any other error means park the host.
Read the pending list before upgrading (apt-get -s upgrade --with-new-pkgs | grep ^Inst), and apply the holds below with apt-mark hold. Park the host instead of upgrading if the list contains any of: a Debian release change, proxmox-kernel*/pve-kernel* on a hypervisor, the ZFS stack on a root-on-ZFS host, grub*/shim*/systemd-boot* on an unusual boot setup, or anything you cannot explain.

Never run apt in-band through qm guest exec, and never in a plain ssh session. If ssh drops, dpkg is left half-configured; that happened to both PBS VMs on 10-04. Run it as a transient unit and poll it:

sudo systemd-run --unit=park100-patch --collect -E DEBIAN_FRONTEND=noninteractive -E NEEDRESTART_MODE=l \
  apt-get -q -y -o Dpkg::Options::=--force-confold upgrade --with-new-pkgs
sudo journalctl -u park100-patch -f      # or: systemctl is-active park100-patch

NEEDRESTART_MODE=l means needrestart lists services but restarts nothing.

Post-check: the same commands as step 2, and they must match. Same failed units (ideally none), same containers running and healthy, health check passing, dpkg --audit empty, apt list --upgradable showing only held packages. Note /var/run/reboot-required. Do NOT reboot: reboots are deferred to Prime.
If something breaks and is not fixed within 10 minutes: stop, leave it in the safest state, park the host with exactly what happened, and tell infra-ops. If the break is user-facing (DNS, the bus, the gateway, gitea), also tell Miranda, per the standing order.
One host at a time. Never the two DNS hosts (nh3-docker, ana-docker) in the same hour.

Holds

Docker engine (docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras) is held on every host whose containers matter: ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-gx10. An engine upgrade restarts dockerd at once, which bounces every container (the bus, DNS, the gateway, Worldtree, the embed seats). Those bumps are parked for one daytime window. Elsewhere the engine may upgrade.
NVIDIA/CUDA user-space (nvidia-* libnvidia-* cuda-* libcuda*) is held on the GPU LXCs (esh-ml1, nh3-ml1) and pfi-gx10. In an LXC it must match the host's kernel driver.
Proxmox hosts: hold proxmox-kernel-* pve-kernel-* and the ZFS stack. If those are the only pending packages, the host counts as current; park the kernel/ZFS bump.

Order (low → high blast radius)

#	host	address	health check	notes
1	ana-filebot	10.250.50.53	`systemctl --failed`	LXC
2	pfi-pteradactyl	10.250.50.55	panel `curl -sk https://localhost` → 200/302	game panel
3	nh3-extdev	10.100.50.42	`postbox status` as infra-ops	althing wheel install lives here; do not reinstall it
4	vm-esh-nas	10.0.50.154	its containers + NFS mounts (`findmnt -t nfs,nfs4`)
5	nh3-albok	10.100.50.44	`curl -s localhost:8392/health` → `"status":"ok"`	built 10-04, likely nearly current; engine may upgrade here (single-tenant, albok restarts in ~15 s)
6	esh-matter	10.0.90.20	`systemctl --failed`; matter server port 5580 listening	Home Assistant's Matter fabric
7	pfi-tacticalrmm	10.250.50.57	RMM web UI 200; MeshCentral up	the AMT/CIRA path to the hypervisors runs through it
8	esh-vm-db	10.0.50.60	`pg_isready`; mongo port 27017	⚠ apt lists never refreshed since 2024-08. If the release is EOL or `apt update` is not clean, PARK
9	nh3-ml1	10.100.50.80	`curl -s localhost:8001/health`, `:8013/health`	engine + NVIDIA held. Second embed backend (failover only now)
10	esh-ml1	10.0.50.80	same, plus `:8003` reward seat	fleet embedding primary: engine + NVIDIA held, so no TEI restart
11	corviduo-dev	10.250.50.152	the three Worldtree `/health` (8080/8081/8082)	engine held. Finish before 0700 (the 0730 Worldtree gate run)
12	pbs-nh3, pbs-ana	10.100.50.90, 10.250.50.90	`proxmox-backup-manager datastore list`; no running task	only if something is pending (both were upgraded 10-04). Never during the 0630 sync
13	hypervisors: nh3-pve-2, nh3-pve, esh-pve, esh-nas-pve, pfi-pve		`pvesh get /cluster/resources --type vm` all expected guests running	kernel/ZFS held per above; everything else patched; no reboot
14	esh-docker-vm	10.0.50.45	AdGuard (ESH DNS) answers; Homepage `/api/services`	engine held. ⚠ comment out the dead Google Coral apt repo first (it makes `apt update` exit 100). ⚠ Do not restart homepage (it wedges: incident_homepage_mmap_lock_wedge)
15	ana-docker	10.250.50.70	gateway `:4000/health/liveliness`, gitea `/api/v1/version`, traefik up	engine held. Not in the same hour as nh3-docker
16	nh3-docker	10.100.50.40	AdGuard answers `dig @10.100.50.40 nh3-docker.nh3.internal`; post office `:8390` reachable; searxng	engine held. The bus lives here, so a broken post office cuts your own comms: check it first and last
17	nh3-dev	10.100.10.50	svos `:8770`, hermes gateway, Booth `:8090`, `postbox status`	engine held. Live agent sessions run here; needrestart in list mode only
18	pfi-gx10	10.100.50.60	`nvidia-smi`; its seats	engine + NVIDIA/CUDA held. DGX OS (Ubuntu, arm64): if it wants its own `dgx` release upgrade, PARK

Out of scope / parked up front

irv-ml1: Prime, explicitly. Do not patch.
ana-nas (CT 109): parked. It is the NFS SPOF for PBS-ANA's datastore and the rest-servers. A nfs-kernel-server or samba restart under a running backup is the failure mode. That is Prime's call.
No infra-ops access, so out of reach: pfi-postgres, ana-wg, the mesh routers + nh3-headscale, nh3-nas, the Windows VMs, PFI-Mongo, SlaveBot.
SureFire tenant hosts (sf-*, sfsrv-ana): tenant property, coordinate first. Not tonight.
fv-ml1 (done by infra-ops): patched in the Docker-move window, with the ZFS stack 2.3.2 → 2.3.9 (root is on ZFS) and nvidia-container-toolkit 1.19 → 1.20 HELD and parked for Prime.

Report

When done, or at 0700 at the latest, send infra-ops ONE message with a table: host | pending before → after | holds | reboot-required | parked + reason. Keep the pre/post outputs; they are the evidence.

7.6 KiB Raw Blame History