docs(runbooks): fleet guest patch campaign 2026-10-05 (park 100); fv-ml1 docker-to-tank one-off

docs/runbooks/fleet-guest-patch-2026-10-05.md

@@ -0,0 +1,89 @@
+# Fleet guest patch campaign, 2026-10-05 (henge park 100)
+
+**Ordered by Prime, 2026-10-05 ~0145:** bring every host current. Low-blast-radius hosts go first.
+**irv-ml1 is OFF-LIMITS: do not patch it, do not touch apt on it.** Reboots may wait. Verify every host before and
+after. Anything that looks dicey is parked for Prime, not pushed through.
+
+**Executor:** infra-hermes. **Owner of the plan / escalation:** infra-ops. fv-ml1 was patched by infra-ops in the
+same window as its Docker storage move (its holds are listed at the bottom).
+
+## Rules for every host
+
+1. **Claim, then record.** `scripts/ops-log claim <host> apt --why "park 100 patch"`, then
+   `scripts/ops-log record --host <host> --action patch --target apt --detail "<before→after, holds, reboot y/n>"` and release.
+2. **Pre-check, and save the output so it can be compared after:**
+   `systemctl --failed --no-legend`, `docker ps --format '{{.Names}} {{.Status}}'` (where Docker exists), the host's own
+   health check from the table, `df -h /` (need ≥ 2 GB free), and `dpkg --audit` (must be empty).
+   If the host is already broken before you start, do not patch. Park it.
+3. **`apt-get update` must be clean.** A dead third-party repo is fixable (comment out its sources entry, and note it).
+   Any other error means park the host.
+4. **Read the pending list before upgrading** (`apt-get -s upgrade --with-new-pkgs | grep ^Inst`), and apply the holds
+   below with `apt-mark hold`. **Park the host instead of upgrading** if the list contains any of:
+   a Debian release change, `proxmox-kernel*`/`pve-kernel*` on a hypervisor, the ZFS stack on a root-on-ZFS host,
+   `grub*`/`shim*`/`systemd-boot*` on an unusual boot setup, or anything you cannot explain.
+5. **Never run apt in-band through `qm guest exec`**, and never in a plain ssh session. If ssh drops, dpkg is left
+   half-configured; that happened to both PBS VMs on 10-04. Run it as a transient unit and poll it:
+   ```bash
+   sudo systemd-run --unit=park100-patch --collect -E DEBIAN_FRONTEND=noninteractive -E NEEDRESTART_MODE=l \
+     apt-get -q -y -o Dpkg::Options::=--force-confold upgrade --with-new-pkgs
+   sudo journalctl -u park100-patch -f      # or: systemctl is-active park100-patch
+   ```
+   `NEEDRESTART_MODE=l` means needrestart lists services but restarts nothing.
+6. **Post-check: the same commands as step 2, and they must match.** Same failed units (ideally none), same containers
+   running and healthy, health check passing, `dpkg --audit` empty, `apt list --upgradable` showing only held packages.
+   Note `/var/run/reboot-required`. **Do NOT reboot**: reboots are deferred to Prime.
+7. **If something breaks** and is not fixed within 10 minutes: stop, leave it in the safest state, park the host with
+   exactly what happened, and tell infra-ops. If the break is user-facing (DNS, the bus, the gateway, gitea), also tell
+   Miranda, per the standing order.
+8. **One host at a time.** Never the two DNS hosts (nh3-docker, ana-docker) in the same hour.
+
+## Holds
+
+- **Docker engine (`docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras`)
+  is held on every host whose containers matter**: ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev,
+  esh-ml1, nh3-ml1, pfi-gx10. An engine upgrade restarts dockerd at once, which bounces every container (the bus,
+  DNS, the gateway, Worldtree, the embed seats). Those bumps are parked for one daytime window. Elsewhere the engine
+  may upgrade.
+- **NVIDIA/CUDA user-space (`nvidia-* libnvidia-* cuda-* libcuda*`) is held on the GPU LXCs (esh-ml1, nh3-ml1) and
+  pfi-gx10.** In an LXC it must match the host's kernel driver.
+- **Proxmox hosts:** hold `proxmox-kernel-* pve-kernel-*` and the ZFS stack. If those are the only pending packages,
+  the host counts as current; park the kernel/ZFS bump.
+
+## Order (low → high blast radius)
+
+| # | host | address | health check | notes |
+|---|---|---|---|---|
+| 1 | ana-filebot | 10.250.50.53 | `systemctl --failed` | LXC |
+| 2 | pfi-pteradactyl | 10.250.50.55 | panel `curl -sk https://localhost` → 200/302 | game panel |
+| 3 | nh3-extdev | 10.100.50.42 | `postbox status` as infra-ops | althing wheel install lives here; do not reinstall it |
+| 4 | vm-esh-nas | 10.0.50.154 | its containers + NFS mounts (`findmnt -t nfs,nfs4`) | |
+| 5 | nh3-albok | 10.100.50.44 | `curl -s localhost:8392/health` → `"status":"ok"` | built 10-04, likely nearly current; engine may upgrade here (single-tenant, albok restarts in ~15 s) |
+| 6 | esh-matter | 10.0.90.20 | `systemctl --failed`; matter server port 5580 listening | Home Assistant's Matter fabric |
+| 7 | pfi-tacticalrmm | 10.250.50.57 | RMM web UI 200; MeshCentral up | the AMT/CIRA path to the hypervisors runs through it |
+| 8 | esh-vm-db | 10.0.50.60 | `pg_isready`; mongo port 27017 | ⚠ apt lists never refreshed since 2024-08. If the release is EOL or `apt update` is not clean, PARK |
+| 9 | nh3-ml1 | 10.100.50.80 | `curl -s localhost:8001/health`, `:8013/health` | engine + NVIDIA held. Second embed backend (failover only now) |
+| 10 | esh-ml1 | 10.0.50.80 | same, plus `:8003` reward seat | **fleet embedding primary**: engine + NVIDIA held, so no TEI restart |
+| 11 | corviduo-dev | 10.250.50.152 | the three Worldtree `/health` (8080/8081/8082) | engine held. Finish before 0700 (the 0730 Worldtree gate run) |
+| 12 | pbs-nh3, pbs-ana | 10.100.50.90, 10.250.50.90 | `proxmox-backup-manager datastore list`; no running task | only if something is pending (both were upgraded 10-04). Never during the 0630 sync |
+| 13 | hypervisors: nh3-pve-2, nh3-pve, esh-pve, esh-nas-pve, pfi-pve | | `pvesh get /cluster/resources --type vm` all expected guests running | kernel/ZFS held per above; everything else patched; no reboot |
+| 14 | esh-docker-vm | 10.0.50.45 | AdGuard (ESH DNS) answers; Homepage `/api/services` | engine held. ⚠ comment out the dead Google Coral apt repo first (it makes `apt update` exit 100). ⚠ Do not restart homepage (it wedges: incident_homepage_mmap_lock_wedge) |
+| 15 | ana-docker | 10.250.50.70 | gateway `:4000/health/liveliness`, gitea `/api/v1/version`, traefik up | engine held. Not in the same hour as nh3-docker |
+| 16 | nh3-docker | 10.100.50.40 | AdGuard answers `dig @10.100.50.40 nh3-docker.nh3.internal`; post office `:8390` reachable; searxng | engine held. The bus lives here, so a broken post office cuts your own comms: check it first and last |
+| 17 | nh3-dev | 10.100.10.50 | svos `:8770`, hermes gateway, Booth `:8090`, `postbox status` | engine held. Live agent sessions run here; needrestart in list mode only |
+| 18 | pfi-gx10 | 10.100.50.60 | `nvidia-smi`; its seats | engine + NVIDIA/CUDA held. DGX OS (Ubuntu, arm64): if it wants its own `dgx` release upgrade, PARK |
+
+## Out of scope / parked up front
+
+- **irv-ml1**: Prime, explicitly. Do not patch.
+- **ana-nas** (CT 109): parked. It is the NFS SPOF for PBS-ANA's datastore and the rest-servers. A `nfs-kernel-server`
+  or `samba` restart under a running backup is the failure mode. That is Prime's call.
+- **No infra-ops access, so out of reach:** pfi-postgres, ana-wg, the mesh routers + nh3-headscale, nh3-nas, the
+  Windows VMs, PFI-Mongo, SlaveBot.
+- **SureFire tenant hosts** (sf-*, sfsrv-ana): tenant property, coordinate first. Not tonight.
+- **fv-ml1** (done by infra-ops): patched in the Docker-move window, with the ZFS stack 2.3.2 → 2.3.9 (root is on
+  ZFS) and nvidia-container-toolkit 1.19 → 1.20 HELD and parked for Prime.
+
+## Report
+
+When done, or at 0700 at the latest, send infra-ops ONE message with a table: host | pending before → after | holds |
+reboot-required | parked + reason. Keep the pre/post outputs; they are the evidence.

scripts/oneoff/fv-ml1-docker-to-tank-20261005.sh

@@ -0,0 +1,90 @@
+#!/bin/bash
+# One-off, 2026-10-05 (Prime, park 99): move fv-ml1's Docker data off the root SSD mirror (zroot) onto tank,
+# and patch the host in the same window so the model seats bounce once, not twice.
+#
+#   scp this to fv-ml1, then: sudo systemd-run --unit=docker-move --collect /root/fv-ml1-docker-to-tank.sh
+#   progress: /var/log/docker-move-20261005.log
+#
+# Shape: a sparse 1 TiB zvol tank/docker, ext4, mounted AT /var/lib/docker (fstab, nofail, after
+# zfs-volume-wait). The path does not change, so nothing that names /var/lib/docker (restic, volumes, compose)
+# needs editing. Why a zvol and not a dataset: Docker's overlay2 driver refuses a ZFS backing filesystem, and an
+# unpinned daemon would silently pick the zfs graph driver and lose every image. ext4 under overlay2 is the
+# canonical setup. docker.service gets RequiresMountsFor=/var/lib/docker, so if the zvol is ever missing at boot
+# docker stays down instead of starting fresh on an empty root directory.
+#
+# Patching: everything except the packages that could take the box or its GPUs down at the next boot, which are
+# held and parked for Prime: the ZFS stack (2.3.2 -> 2.3.9 rebuilds the module root is mounted with) and
+# nvidia-container-toolkit (the GPU hook every seat starts through). No reboot.
+#
+# The old tree stays at /var/lib/docker.old-zroot until the seats are verified; deleting it is a separate,
+# literal-path step.
+set -euo pipefail
+exec > >(tee -a /var/log/docker-move-20261005.log) 2>&1
+step() { echo; echo "=== $(date +%H:%M:%S) $*"; }
+
+step "pre-state"
+docker ps --format '{{.Names}}' | sort > /root/docker-move-running-before.txt
+cat /root/docker-move-running-before.txt | tr '\n' ' '; echo
+df -h / /tank | cat
+
+step "zvol tank/docker (sparse 1T) + ext4"
+zfs list tank/docker >/dev/null 2>&1 || zfs create -s -V 1T tank/docker
+udevadm settle
+for i in $(seq 1 30); do [ -e /dev/zvol/tank/docker ] && break; sleep 1; done
+test -e /dev/zvol/tank/docker
+blkid /dev/zvol/tank/docker >/dev/null 2>&1 || mkfs.ext4 -q -L docker -E lazy_itable_init=1,lazy_journal_init=1 /dev/zvol/tank/docker
+mkdir -p /mnt/docker-new
+mountpoint -q /mnt/docker-new || mount /dev/zvol/tank/docker /mnt/docker-new
+
+step "pass 1 (live copy)"
+rsync -aHAXS --numeric-ids /var/lib/docker/ /mnt/docker-new/
+
+step "stop docker"
+systemctl stop docker.socket docker.service
+sleep 2
+if pgrep -x dockerd >/dev/null; then echo 'dockerd still running, abort before the final pass'; exit 1; fi
+
+step "pass 2 (final, --delete)"
+rsync -aHAXS --numeric-ids --delete /var/lib/docker/ /mnt/docker-new/
+a=$(find /var/lib/docker -xdev | wc -l); b=$(find /mnt/docker-new -xdev | wc -l)
+echo "entries: old=$a new=$b"
+# lost+found exists only on the new ext4, so new = old + 1
+test "$((b - a))" -eq 1 || { echo 'entry count mismatch, abort (docker is stopped, old tree untouched)'; exit 1; }
+
+step "swap: old tree aside, zvol mounted at /var/lib/docker"
+umount /mnt/docker-new
+mv /var/lib/docker /var/lib/docker.old-zroot
+install -d -m 710 /var/lib/docker
+grep -q '^/dev/zvol/tank/docker ' /etc/fstab || \
+  echo '/dev/zvol/tank/docker /var/lib/docker ext4 defaults,nofail,x-systemd.requires=zfs-volume-wait.service,x-systemd.after=zfs-volume-wait.service 0 2' >> /etc/fstab
+install -d /etc/systemd/system/docker.service.d
+printf '[Unit]\n# /var/lib/docker is the tank/docker zvol (2026-10-05). Never start on an empty root dir.\nRequiresMountsFor=/var/lib/docker\n' \
+  > /etc/systemd/system/docker.service.d/10-requires-docker-mount.conf
+systemctl daemon-reload
+mount /var/lib/docker
+findmnt -no SOURCE,FSTYPE /var/lib/docker
+
+step "patch (holds: ZFS stack, nvidia-container-toolkit)"
+apt-mark hold zfs-dkms zfsutils-linux zfs-zed zfs-initramfs libzfs6linux libzpool6linux libnvpair3linux libuutil3linux \
+  nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1 2>&1 | tail -3 || true
+DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get -q -y -o Dpkg::Options::=--force-confold upgrade --with-new-pkgs
+echo "still upgradable (held/kept back):"; apt list --upgradable 2>/dev/null | tail -n +2 | cut -d/ -f1 | tr '\n' ' '; echo
+test -f /var/run/reboot-required && echo "REBOOT REQUIRED (deferred): $(cat /var/run/reboot-required.pkgs 2>/dev/null | tr '\n' ' ')"
+
+step "start docker"
+systemctl start docker.service
+docker info --format 'root={{.DockerRootDir}} driver={{.Driver}} ver={{.ServerVersion}} images={{.Images}}'
+
+step "wait for the same containers to be running + healthy (up to 25 min)"
+for i in $(seq 1 150); do
+  docker ps --format '{{.Names}}' | sort > /root/docker-move-running-after.txt
+  missing=$(comm -23 /root/docker-move-running-before.txt /root/docker-move-running-after.txt | tr '\n' ' ')
+  # `|| true`: grep finding nothing is the SUCCESS case here, and set -e would kill the script on it
+  starting=$(docker ps --format '{{.Names}} {{.Status}}' | grep -E 'health: starting|unhealthy' | cut -d' ' -f1 | tr '\n' ' ' || true)
+  [ -z "$missing" ] && [ -z "$starting" ] && break
+  sleep 10
+done
+echo "missing: ${missing:-none}"; echo "starting/unhealthy: ${starting:-none}"
+docker ps --format '{{.Names}}|{{.Status}}' | cat
+df -h / /var/lib/docker | cat
+step "done"