vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

runbook(park 100): pfi-tacticalrmm rebooted + TRMM 1.5.2 (Prime)

Browse Source
This commit is contained in:
vh
2026-10-05 11:13:34 -07:00
parent 912d932917
commit f6fe6f67f8
2 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/fleet-guest-patch-2026-10-05.md
+2 -2
View File
@@ -115,8 +115,8 @@ bouncer failing closed on slow LAPI answers. Self-cleared.
**Deferred to a daytime window:**
- reboots to activate new kernels: ~~nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl~~ (rebooted 2026-10-05
1055–1109 on Prime's word; all came back clean, boot ids changed, 0 failed units). Still pending: ana-docker, nh3-dev,
vm-esh-nas, pfi-tacticalrmm. None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer
1055–1109 on Prime's word; all came back clean, boot ids changed, 0 failed units). pfi-tacticalrmm rebooted + TRMM updated to 1.5.2
by Prime ~1110 (verified: 0 failed units, cert intact). Still pending: ana-docker, nh3-dev, vm-esh-nas. None builds NVIDIA with DKMS. ⚠ corviduo-dev's Worldtree APIs take ~3 min to answer
`/health` after boot (startup, not a fault); wings on pfi-pteradactyl restores the running game server by itself.
- Docker engine bumps held on ana-docker, nh3-docker, esh-docker-vm, corviduo-dev, nh3-dev, esh-ml1, nh3-ml1, pfi-gx10.
- fv-ml1: ZFS 2.3.9, nvidia-container-toolkit 1.20, and a kernel with an NVIDIA driver that builds on it.
persistent-memory.md
+1 -1
View File
@@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
### 2026-10-04: live now (as of 2026-10-04 ~2300 PT)
- **Waiting on Prime (2026-10-05, updated 1100):** reboots for ana-docker/nh3-dev/vm-esh-nas/pfi-tacticalrmm (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl rebooted clean 10-05 ~1100); a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-gx10 (dgx-release) parked; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause).
- **Waiting on Prime (2026-10-05, updated 1100):** reboots for ana-docker/nh3-dev/vm-esh-nas (nh3-extdev, corviduo-dev, esh-vm-db, pfi-pteradactyl rebooted clean 10-05 ~1100; pfi-tacticalrmm by Prime + TRMM 1.5.2); a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-gx10 (dgx-release) parked; pfi-pteradactyl fixed + patched (engine held: live game server); pfi-tacticalrmm patched; CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause).
- **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%.
- **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
- **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`