diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index 684b1b5..df2bc37 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -103,7 +103,9 @@ User-facing impact: two Gitea 403 blips (0310–0314, 0342) while ana-docker's a bouncer failing closed on slow LAPI answers. Self-cleared. **Parked for Prime:** -- pfi-pteradactyl: already broken before the run (apache2 failed, panel down). 164 pending. +- ~~pfi-pteradactyl~~ **FIXED + patched 2026-10-05 ~1045 (Prime):** "apache2 failed / panel down" was a leftover apache2 + losing :80 to nginx; the panel (nginx + php-fpm) was up all along (the curl that read 000 used https or the dead + `ptero.phasefinal.com`). apache2 masked; 163 → 0; Docker engine held (live game server); reboot deferred. - pfi-tacticalrmm: certbot renewal failed 2026-10-04 1831 (web + MeshCentral fine). 164 pending. - pfi-gx10: wants a `dgx-release` 7.5 → 7.6 upgrade, a new bootloader package (`nvidia-spark-grub-kho`) and kernel 7.0.0-1019-nvidia. 212 pending. diff --git a/persistent-memory.md b/persistent-memory.md index 9773d45..5b17aaf 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,7 +119,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ### 2026-10-04: live now (as of 2026-10-04 ~2300 PT) -- **Waiting on Prime (2026-10-05 0355):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; parked pfi-pteradactyl, pfi-tacticalrmm (cert now fixed, so it can be patched), pfi-gx10 (dgx-release); CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause). +- **Waiting on Prime (2026-10-05 0355):** reboots for ana-docker/nh3-dev/nh3-extdev/corviduo-dev/vm-esh-nas/esh-vm-db; a daytime window for the held Docker engine bumps; fv-ml1's held ZFS/toolkit/kernel; pfi-gx10 (dgx-release) parked; pfi-pteradactyl fixed + patched; pfi-tacticalrmm patching via infra-hermes; CrowdSec bouncer `updateMaxFailure` 0 → ~3 (lean; less needed now that WAL fixed the root cause). - **irv-ml1 `/storetank` ~88% is planned, not a leak (2026-10-05 0035 Beszel 85% alert).** A Foley weights stage (`/tmp/comfy-foley-stage-20261005/stage.py`, lkraven, arbo models, 40 GB over 97 files) has a built-in 200 GiB free-space floor and refuses below it. The alert stays tripped at ~88%. - **albok-service now on nh3-albok (VM 110, nh3-pve-2, 10.100.50.44) since 2332; DNS moved. CLOSED 2026-10-05 0021:** albok-dev acceptance passed 0018, the old `/srv/albok` on nh3-docker was deleted on Prime's approval, and the first VM 110 backup is on pbs-ana. The compose tombstone stays on nh3-docker. → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` diff --git a/servers/pfi-pteradactyl/README.md b/servers/pfi-pteradactyl/README.md index 2e95a7d..03422c5 100644 --- a/servers/pfi-pteradactyl/README.md +++ b/servers/pfi-pteradactyl/README.md @@ -8,6 +8,21 @@ Pterodactyl game-server panel at the Anaheim colo. - **Web UI:** http://10.250.50.55/ (listed on homepage under **Games**) - **SSH:** `lkraven@pfi-pteradactyl` +## State and gotchas (2026-10-05) + +- **The panel is nginx + php8.3-fpm**, not Apache: `/etc/nginx/sites-enabled/pterodactyl.conf`, root + `/var/www/pterodactyl/public`, plain HTTP on :80, with MariaDB, redis, pteroq and wings (:8080). Wings reaches the panel + at `http://10.250.50.55` (`/etc/pterodactyl/config.yml` `remote:`). +- **apache2 is MASKED** (2026-10-05). It is only installed because `libapache2-mod-php8.3` pulls it in. It fought nginx + for :80 and showed as a failed unit after every boot, which is what made the host look broken during park 100. + The panel itself was up the whole time. +- `APP_URL` in `/var/www/pterodactyl/.env` is `http://ptero.phasefinal.com`, a name with **no DNS record** (Cloudflare or + internal). Access is LAN-only by IP, which is fine. Links the panel generates (emails, for example) carry the + dead name. +- **Patched 2026-10-05** (163 packages, MariaDB 11.7 → 11.8.9). Pre-patch DB dump: `/root/panel-db-pre-patch-20261005.sql.gz`. + **`docker.io`, `containerd` and `runc` are HELD**: an engine upgrade restarts dockerd, which would stop the running + game server (wings does not restart it). Bump them when no one is playing. Kernel 6.1.0-53 needs a reboot (deferred). + ## Infrastructure - **Hypervisor:** `pfi-pve` (VMID **107**)