feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2

Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
This commit is contained in:
vh
2026-10-05 20:22:16 -07:00
parent e369b1c769
commit 1210b4a0d5
6 changed files with 210 additions and 46 deletions
+5
View File
@@ -32,6 +32,8 @@
# 1 backups STALE/DOWN, and the alert was delivered
# 2 the ALERT PATH ITSELF FAILED (investigate this before the backups)
# 3 ERRORED-JOBS — every body fresh, a job errored recently; alert delivered
# 4 DEGRADED — no job errored, but an OK task logged an ERROR (fs-freeze refused =
# crash-consistent image only); alert delivered
#
# 3 is deliberately not 1. A nightly job that errored is worth a look; a stale
# backup body is worth a page. Reporting both as "STALE" over a fleet whose
@@ -83,6 +85,9 @@ printf '%s\n' "$out"
if [ "$rc" -eq 3 ]; then
subject="🟡 Backup jobs errored ($(date '+%Y-%m-%d')) — all bodies fresh"
lead="A vzdump job errored recently. Every backup body on the fleet is FRESH; nothing is stale."
elif [ "$rc" -eq 4 ]; then
subject="🟠 Backup DEGRADED ($(date '+%Y-%m-%d')) — an OK job logged an ERROR"
lead="Every backup body is FRESH and no job failed, but a backup logged an ERROR inside a task PVE calls OK. An fs-freeze failure there means that image is crash-consistent only, not application-consistent."
else
subject="🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))"
lead="Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet."