Files
esh-pfi-infrastructure/scripts
vh 7fe4102458 fix(backups): stop saying STALE over a fleet whose every backup is fresh
The check collapsed two different findings into one verdict. On 2026-09-20 it
printed "RESULT: STALE" while reporting 37 FRESH layers and zero stale ones --
every backup body provably current, the three ❌ rows all yesterday's pre-fix
runs aging out of the 36h window. infra-hermes caught it in triage: a reader,
or a forwarder, could page someone over a state where nothing is stale.

STALE is a claim about backup AGE. A job that ran and errored is a different
claim with different urgency. They now have different words and different exit
codes:

  0  all backups fresh
  1  STALE          -- a body past the threshold, or an endpoint down
  3  ERRORED-JOBS   -- every body fresh, a vzdump job errored recently

The alert wrapper mirrors the code and matches its own wording to the finding:
🟡 "Backup jobs errored — all bodies fresh" instead of 🔴 "Backup freshness
ALERT", and it now exits with the check's code rather than flattening
everything to 1, so `systemctl status` distinguishes the states too.

This is the same defect class the rest of this script was built to fix, one
level up: not an instrument that fails to look, but one that looks correctly
and then reports the wrong word for what it saw. An alarm that cries outage
over a healthy fleet earns being ignored exactly as fast as one that stays
silent over a broken one.

Verified all three states by forcing each: BACKUP_JOB_WINDOW_HOURS=1 -> exit 0,
default -> exit 3, BACKUP_MAX_AGE_HOURS=1 -> exit 1.
2026-09-20 08:03:05 -07:00
..