feat(svos-front,backups): SVOS hosting bounds and previous-release assets; backup check flags fs-freeze failures and covers nh3-pve-2
Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
This commit is contained in:
@@ -32,6 +32,8 @@
|
||||
# 1 backups STALE/DOWN, and the alert was delivered
|
||||
# 2 the ALERT PATH ITSELF FAILED (investigate this before the backups)
|
||||
# 3 ERRORED-JOBS — every body fresh, a job errored recently; alert delivered
|
||||
# 4 DEGRADED — no job errored, but an OK task logged an ERROR (fs-freeze refused =
|
||||
# crash-consistent image only); alert delivered
|
||||
#
|
||||
# 3 is deliberately not 1. A nightly job that errored is worth a look; a stale
|
||||
# backup body is worth a page. Reporting both as "STALE" over a fleet whose
|
||||
@@ -83,6 +85,9 @@ printf '%s\n' "$out"
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
subject="🟡 Backup jobs errored ($(date '+%Y-%m-%d')) — all bodies fresh"
|
||||
lead="A vzdump job errored recently. Every backup body on the fleet is FRESH; nothing is stale."
|
||||
elif [ "$rc" -eq 4 ]; then
|
||||
subject="🟠 Backup DEGRADED ($(date '+%Y-%m-%d')) — an OK job logged an ERROR"
|
||||
lead="Every backup body is FRESH and no job failed, but a backup logged an ERROR inside a task PVE calls OK. An fs-freeze failure there means that image is crash-consistent only, not application-consistent."
|
||||
else
|
||||
subject="🔴 Backup freshness ALERT ($(date '+%Y-%m-%d'))"
|
||||
lead="Automated daily backup-freshness check found STALE or DOWN backup layer(s) on the PFI fleet."
|
||||
|
||||
Reference in New Issue
Block a user