From 241b14bc66ecd73a9aef5f2614ae1e3519ced745 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 03:55:58 -0700 Subject: [PATCH] =?UTF-8?q?runbook(park=20100):=20results=20=E2=80=94=2015?= =?UTF-8?q?=20patched,=20parks=20and=20deferrals;=20memory?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/runbooks/fleet-guest-patch-2026-10-05.md | 26 +++++++++++++++++++ persistent-memory.md | 3 ++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index ec4181f..684b1b5 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -92,3 +92,29 @@ same window as its Docker storage move (its holds are listed at the bottom). When done, or at 0700 at the latest, send infra-ops ONE message with a table: host | pending before → after | holds | reboot-required | parked + reason. Keep the pre/post outputs; they are the evidence. + +## Result (2026-10-05 0355, infra-hermes + infra-ops) + +**Patched clean (15):** ana-filebot, nh3-extdev, nh3-albok (already current), esh-matter, nh3-ml1, esh-ml1, corviduo-dev, +pbs-nh3, pbs-ana, the five hypervisors (already current), esh-docker-vm (dead Coral repo commented out), ana-docker, +nh3-docker, nh3-dev, vm-esh-nas (engine 28 → 29, NFS mounts intact), esh-vm-db (current bookworm despite lists stale +since 2024), plus fv-ml1 (infra-ops, in the Docker-move window). Pre/post checks matched everywhere. No reboots. +User-facing impact: two Gitea 403 blips (0310–0314, 0342) while ana-docker's apt loaded the host. Cause: the CrowdSec +bouncer failing closed on slow LAPI answers. Self-cleared. + +**Parked for Prime:** +- pfi-pteradactyl: already broken before the run (apache2 failed, panel down). 164 pending. +- pfi-tacticalrmm: certbot renewal failed 2026-10-04 1831 (web + MeshCentral fine). 164 pending. +- pfi-gx10: wants a `dgx-release` 7.5 → 7.6 upgrade, a new bootloader package (`nvidia-spark-grub-kho`) and kernel + 7.0.0-1019-nvidia. 212 pending. +- ana-nas (NFS SPOF) and irv-ml1 (Prime: off-limits): out of scope. + +**Deferred to a daytime window:** +- reboots to activate new kernels: ana-docker, nh3-dev, nh3-extdev, corviduo-dev, vm-esh-nas, esh-vm-db. None + builds NVIDIA with DKMS. +- Docker engine bumps held on ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-gx10. +- fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it. +- CrowdSec bouncer `updateMaxFailure` (0 = block on the first failed LAPI pull): infra-ops leans toward ~3. + +**Lesson:** four hosts were first parked as "no NOPASSWD sudo" because the ssh aliases log in as lkraven. As +infra-ops@ all four had it (auto-memory `feedback_ssh_alias_defaults_to_lkraven`). diff --git a/persistent-memory.md b/persistent-memory.md index 01d9225..fe16551 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,6 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) +- **Waiting on Prime (2026-10-05 0355):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; parked pfi-pteradactyl, pfi-tacticalrmm (certbot), pfi-gx10 (dgx-release); CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean). - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` @@ -301,7 +302,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions - `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md` -- `[2026-10-05]` **Fleet guest patching (park 100) handed to infra-hermes 0243** — plan + holds in `docs/runbooks/fleet-guest-patch-2026-10-05.md`; irv-ml1 OFF-LIMITS (Prime); ana-nas parked; report due by 0700. +- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result - `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md` - `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md` - `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`