135 lines
12 KiB
Markdown
135 lines
12 KiB
Markdown
# Fleet guest patch campaign, 2026-10-05 (henge park 100)
|
||
|
||
**Ordered by Prime, 2026-10-05 ~0145:** bring every host current. Low-blast-radius hosts go first.
|
||
**irv-ml1 is OFF-LIMITS: do not patch it, do not touch apt on it.** Reboots may wait. Verify every host before and
|
||
after. Anything that looks dicey is parked for Prime, not pushed through.
|
||
|
||
**Executor:** infra-hermes. **Owner of the plan / escalation:** infra-ops. fv-ml1 was patched by infra-ops in the
|
||
same window as its Docker storage move (its holds are listed at the bottom).
|
||
|
||
## Rules for every host
|
||
|
||
1. **Claim, then record.** `scripts/ops-log claim <host> apt --why "park 100 patch"`, then
|
||
`scripts/ops-log record --host <host> --action patch --target apt --detail "<before→after, holds, reboot y/n>"` and release.
|
||
2. **Pre-check, and save the output so it can be compared after:**
|
||
`systemctl --failed --no-legend`, `docker ps --format '{{.Names}} {{.Status}}'` (where Docker exists), the host's own
|
||
health check from the table, `df -h /` (need ≥ 2 GB free), and `dpkg --audit` (must be empty).
|
||
If the host is already broken before you start, do not patch. Park it.
|
||
3. **`apt-get update` must be clean.** A dead third-party repo is fixable (comment out its sources entry, and note it).
|
||
Any other error means park the host.
|
||
4. **Read the pending list before upgrading** (`apt-get -s upgrade --with-new-pkgs | grep ^Inst`), and apply the holds
|
||
below with `apt-mark hold`. **Park the host instead of upgrading** if the list contains any of:
|
||
a Debian release change, `proxmox-kernel*`/`pve-kernel*` on a hypervisor, the ZFS stack on a root-on-ZFS host,
|
||
`grub*`/`shim*`/`systemd-boot*` on an unusual boot setup, or anything you cannot explain.
|
||
5. **Never run apt in-band through `qm guest exec`**, and never in a plain ssh session. If ssh drops, dpkg is left
|
||
half-configured; that happened to both PBS VMs on 10-04. Run it as a transient unit and poll it:
|
||
```bash
|
||
sudo systemd-run --unit=park100-patch --collect -E DEBIAN_FRONTEND=noninteractive -E NEEDRESTART_MODE=l \
|
||
apt-get -q -y -o Dpkg::Options::=--force-confold upgrade --with-new-pkgs
|
||
sudo journalctl -u park100-patch -f # or: systemctl is-active park100-patch
|
||
```
|
||
`NEEDRESTART_MODE=l` means needrestart lists services but restarts nothing.
|
||
6. **Post-check: the same commands as step 2, and they must match.** Same failed units (ideally none), same containers
|
||
running and healthy, health check passing, `dpkg --audit` empty, `apt list --upgradable` showing only held packages.
|
||
Note `/var/run/reboot-required`. **Do NOT reboot**: reboots are deferred to Prime.
|
||
7. **If something breaks** and is not fixed within 10 minutes: stop, leave it in the safest state, park the host with
|
||
exactly what happened, and tell infra-ops. If the break is user-facing (DNS, the bus, the gateway, gitea), also tell
|
||
Miranda, per the standing order.
|
||
8. **One host at a time.** Never the two DNS hosts (nh3-docker, ana-docker) in the same hour.
|
||
|
||
## Holds
|
||
|
||
- **Docker engine (`docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras`)
|
||
is held on every host whose containers matter**: ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev,
|
||
esh-ml1, nh3-ml1, pfi-gx10. An engine upgrade restarts dockerd at once, which bounces every container (the bus,
|
||
DNS, the gateway, Worldtree, the embed seats). Those bumps are parked for one daytime window. Elsewhere the engine
|
||
may upgrade.
|
||
- **NVIDIA/CUDA user-space (`nvidia-* libnvidia-* cuda-* libcuda*`) is held on the GPU LXCs (esh-ml1, nh3-ml1) and
|
||
pfi-gx10.** In an LXC it must match the host's kernel driver.
|
||
- ⚠ **Kernels on any host that builds NVIDIA with DKMS** (hold `linux-image-amd64 linux-headers-amd64`, or the
|
||
`-generic`/`-nvidia` metas on Ubuntu/DGX): `--with-new-pkgs` PULLS A NEW KERNEL. On fv-ml1 tonight, 6.12.111 came in,
|
||
NVIDIA 580.65.06 DKMS FAILED to build for it, and dpkg was left half-configured. Had it booted, there would be no
|
||
GPU driver and every seat would be down. infra-ops purged it. Check `dkms status` for `nvidia` before patching; if it
|
||
is present, hold the kernel metas and park the kernel bump.
|
||
- **Proxmox hosts:** hold `proxmox-kernel-* pve-kernel-*` and the ZFS stack. If those are the only pending packages,
|
||
the host counts as current; park the kernel/ZFS bump.
|
||
|
||
## Order (low → high blast radius)
|
||
|
||
| # | host | address | health check | notes |
|
||
|---|---|---|---|---|
|
||
| 1 | ana-filebot | 10.250.50.53 | `systemctl --failed` | LXC |
|
||
| 2 | pfi-pteradactyl | 10.250.50.55 | panel `curl -sk https://localhost` → 200/302 | game panel |
|
||
| 3 | nh3-extdev | 10.100.50.42 | `postbox status` as infra-ops | althing wheel install lives here; do not reinstall it |
|
||
| 4 | vm-esh-nas | 10.0.50.154 | its containers + NFS mounts (`findmnt -t nfs,nfs4`) | |
|
||
| 5 | nh3-albok | 10.100.50.44 | `curl -s localhost:8392/health` → `"status":"ok"` | built 10-04, likely nearly current; engine may upgrade here (single-tenant, albok restarts in ~15 s) |
|
||
| 6 | esh-matter | 10.0.90.20 | `systemctl --failed`; matter server port 5580 listening | Home Assistant's Matter fabric |
|
||
| 7 | pfi-tacticalrmm | 10.250.50.57 | RMM web UI 200; MeshCentral up | the AMT/CIRA path to the hypervisors runs through it |
|
||
| 8 | esh-vm-db | 10.0.50.60 | `pg_isready`; mongo port 27017 | ⚠ apt lists never refreshed since 2024-08. If the release is EOL or `apt update` is not clean, PARK |
|
||
| 9 | nh3-ml1 | 10.100.50.80 | `curl -s localhost:8001/health`, `:8013/health` | engine + NVIDIA held. Second embed backend (failover only now) |
|
||
| 10 | esh-ml1 | 10.0.50.80 | same, plus `:8003` reward seat | **fleet embedding primary**: engine + NVIDIA held, so no TEI restart |
|
||
| 11 | corviduo-dev | 10.250.50.152 | the three Worldtree `/health` (8080/8081/8082) | engine held. Finish before 0700 (the 0730 Worldtree gate run) |
|
||
| 12 | pbs-nh3, pbs-ana | 10.100.50.90, 10.250.50.90 | `proxmox-backup-manager datastore list`; no running task | only if something is pending (both were upgraded 10-04). Never during the 0630 sync |
|
||
| 13 | hypervisors: nh3-pve-2, nh3-pve, esh-pve, esh-nas-pve, pfi-pve | | `pvesh get /cluster/resources --type vm` all expected guests running | kernel/ZFS held per above; everything else patched; no reboot |
|
||
| 14 | esh-docker-vm | 10.0.50.45 | AdGuard (ESH DNS) answers; Homepage `/api/services` | engine held. ⚠ comment out the dead Google Coral apt repo first (it makes `apt update` exit 100). ⚠ Do not restart homepage (it wedges: incident_homepage_mmap_lock_wedge) |
|
||
| 15 | ana-docker | 10.250.50.70 | gateway `:4000/health/liveliness`, gitea `/api/v1/version`, traefik up | engine held. Not in the same hour as nh3-docker |
|
||
| 16 | nh3-docker | 10.100.50.40 | AdGuard answers `dig @10.100.50.40 nh3-docker.nh3.internal`; post office `:8390` reachable; searxng | engine held. The bus lives here, so a broken post office cuts your own comms: check it first and last |
|
||
| 17 | nh3-dev | 10.100.10.50 | svos `:8770`, hermes gateway, Booth `:8090`, `postbox status` | engine held. Live agent sessions run here; needrestart in list mode only |
|
||
| 18 | pfi-gx10 | 10.100.50.60 | `nvidia-smi`; its seats | engine + NVIDIA/CUDA held. DGX OS (Ubuntu, arm64): if it wants its own `dgx` release upgrade, PARK |
|
||
|
||
## Out of scope / parked up front
|
||
|
||
- **irv-ml1**: Prime, explicitly. Do not patch.
|
||
- **ana-nas** (CT 109): parked. It is the NFS SPOF for PBS-ANA's datastore and the rest-servers. A `nfs-kernel-server`
|
||
or `samba` restart under a running backup is the failure mode. That is Prime's call.
|
||
- **No infra-ops access, so out of reach:** pfi-postgres, ana-wg, the mesh routers + nh3-headscale, nh3-nas, the
|
||
Windows VMs, PFI-Mongo, SlaveBot.
|
||
- **SureFire tenant hosts** (sf-*, sfsrv-ana): tenant property, coordinate first. Not tonight.
|
||
- **fv-ml1** (done by infra-ops): patched in the Docker-move window, with the ZFS stack 2.3.2 → 2.3.9 (root is on
|
||
ZFS) and nvidia-container-toolkit 1.19 → 1.20 HELD and parked for Prime.
|
||
|
||
## Report
|
||
|
||
When done, or at 0700 at the latest, send infra-ops ONE message with a table: host | pending before → after | holds |
|
||
reboot-required | parked + reason. Keep the pre/post outputs; they are the evidence.
|
||
|
||
## Result (2026-10-05 0355, infra-hermes + infra-ops)
|
||
|
||
**Patched clean (15):** ana-filebot, nh3-extdev, nh3-albok (already current), esh-matter, nh3-ml1, esh-ml1, corviduo-dev,
|
||
pbs-nh3, pbs-ana, the five hypervisors (already current), esh-docker-vm (dead Coral repo commented out), ana-docker,
|
||
nh3-docker, nh3-dev, vm-esh-nas (engine 28 → 29, NFS mounts intact), esh-vm-db (current bookworm despite lists stale
|
||
since 2024), plus fv-ml1 (infra-ops, in the Docker-move window). Pre/post checks matched everywhere. No reboots.
|
||
User-facing impact: two Gitea 403 blips (0310–0314, 0342) while ana-docker's apt loaded the host. Cause: the CrowdSec
|
||
bouncer failing closed on slow LAPI answers. Self-cleared.
|
||
|
||
**Parked for Prime:**
|
||
- ~~pfi-pteradactyl~~ **FIXED + patched 2026-10-05 ~1045 (Prime):** "apache2 failed / panel down" was a leftover apache2
|
||
losing :80 to nginx; the panel (nginx + php-fpm) was up all along (the curl that read 000 used https or the dead
|
||
`ptero.phasefinal.com`). apache2 masked; 163 → 0; Docker engine held (live game server); reboot deferred.
|
||
- ~~pfi-tacticalrmm~~ **cert fixed (infra-ops) + patched 2026-10-05 ~1055 (infra-hermes):** 164 → 0, postgres restarted,
|
||
RMM/daphne/nats/meshcentral healthy, MeshCentral never restarted (AMT CIRA :4433 untouched), `certbot renew --dry-run`
|
||
passes after the patch. Kernel 6.1.0-53 needs a reboot (deferred).
|
||
- ~~pfi-gx10~~ **updated + rebooted 2026-10-05 ~1120 (Prime):** DGX OTA 7.5.0 → 7.6.0, kernel 6.17.0-1032 → 7.0.0-1019-nvidia
|
||
(its prebuilt `linux-modules-nvidia-580-open-7.0.0-1019` came with it, so this was not fv-ml1's DKMS failure), driver
|
||
580.173 → 580.178.04, Docker 29.2 → 29.6. Post: nvidia-smi OK, CDI lists the GPU, 0 failed units. nvidia-cdi-refresh
|
||
fails before the reboot (driver/library mismatch) and clears itself after.
|
||
- ana-nas (NFS SPOF) and irv-ml1 (Prime: off-limits): out of scope.
|
||
|
||
**Deferred to a daytime window:**
|
||
- reboots to activate new kernels: ~~nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl~~ (rebooted 2026-10-05
|
||
1055–1109 on Prime's word; all came back clean, boot ids changed, 0 failed units). pfi-tacticalrmm rebooted + TRMM updated to 1.5.2
|
||
by Prime ~1110 (verified: 0 failed units, cert intact). vm-esh-nas rebooted ~1115 (NFS mounts + containers back). ana-docker: Docker engine
|
||
29.4.1 → 29.8.0 + reboot ~1123–1133 (Prime), 44/44 containers identical to the pre-snapshot, gateway/gitea/CrowdSec OK.
|
||
⚠ ana-docker's Docker apt source points at `bullseye` on a bookworm host (it installs bullseye builds). It works, but
|
||
repoint it at bookworm deliberately some day. Still pending: nh3-dev (reboot + engine). None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer
|
||
`/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself.
|
||
- ⚠ For every held engine bump: if `/etc/docker/daemon.json` has `live-restore: true`, an engine upgrade without a reboot leaves
|
||
containers that bind-mount `/var/run/docker.sock` on a stale socket. Restart them afterwards (see servers/ana-docker/README.md).
|
||
- Docker engine bumps held on nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-pteradactyl
|
||
(ana-docker → 29.8.0 and pfi-gx10 → 29.6 done 10-05).
|
||
- fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it.
|
||
- CrowdSec bouncer `updateMaxFailure` (0 = block on the first failed LAPI pull): infra-ops leans toward ~3.
|
||
|
||
**Lesson:** four hosts were first parked as "no NOPASSWD sudo" because the ssh aliases log in as lkraven. As
|
||
infra-ops@<ip> all four had it (auto-memory `feedback_ssh_alias_defaults_to_lkraven`).
|