feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2

Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
This commit is contained in:
vh
2026-10-05 20:22:16 -07:00
parent e369b1c769
commit 1210b4a0d5
6 changed files with 210 additions and 46 deletions
+1 -1
View File
@@ -301,7 +301,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
## Recent decisions
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"). Offline shell assets served before auth; rerun `svos-front-inventory` after every SVOS release. → `servers/nh3-svos/README.md`.
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`