Snapshot age is structurally blind to a backup job that executes every night
and fails every night. Nothing new is written, so the group simply ages, and
the fault only surfaces once it crosses the 48h threshold -- days after the
first failure, with the evidence sitting in a task log nobody reads.
Two live cases, both found today and both invisible for a week by this exact
mechanism:
* esh-nas-pve CT 107 (vm-jellyfin): a backup run died around 09-06 and left
a stale `backup` lock, so every nightly since failed instantly with "CT is
locked (backup)". Age named it on ~09-12. Task status would have named it
on 09-07.
* esh-pve VM 102 (esh-vm-workstation): failing nightly since ~09-06 with
"timeout waiting on systemd". Same six-day gap.
PVE already records every task result in /var/log/pve/tasks/index. This reads
it on all four non-tenant PVE nodes and reports any vzdump in the last 36h
whose status is not OK, as its own section that sets the exit code.
It found a third case on its first run: esh-nas-pve's job had been reporting
`job errors` nightly while every guest on that node read 0-1h fresh, so no
age-based check could ever have flagged it.
Window is BACKUP_JOB_WINDOW_HOURS (default 36 -- longer than a daily cycle so
one missed run does not hide a failure). A node whose task log cannot be read
is reported, never assumed healthy.
ana-scale (CT 114) is a subnet-router LXC, excluded from vzdump on 2026-09-07
after a backup lock on its ESH counterpart blackholed that entire site. The
freshness check knew nothing about that and reported it 🔴 STALE every single
morning, which is how an alarm teaches you to ignore it.
Such guests now get their own section: printed every run, never hidden, and
not counted as a fault.
The subtlety is in how coverage is computed, and the obvious implementation is
wrong twice over:
* Reading one job's `exclude` list gets ana CT 109 (ana-nas) exactly
backwards. It IS excluded from the 03:00 all-guests job AND it has its own
dedicated 22:00 job. Suppressing on the exclude list would have stopped
alarming on a guest that is genuinely backed up -- trading a noisy alarm
for a blind one.
* ESH's job uses an explicit `vmid 100..107` INCLUDE list, so esh-scale 108
is excluded by OMISSION and appears in no exclude list at all.
So coverage is a union across every enabled job on the cluster, and a guest is
"intentionally not backed up" only when none of them covers it.
If coverage cannot be read, nothing is suppressed and the gap is reported: an
unreachable PVE node means we do not know, and a backup alarm must fail loud.
The SureFire namespace is never consulted (tenant property), so its guests can
never be suppressed either.
Verified against the live fleet on all four paths: CT 114 suppressed; CT 109
NOT suppressed despite being in an exclude list; esh-vm-workstation 102, which
a job really does cover and which really is failing, still reports STALE; and
with a PVE node made unreachable, 114 returns to STALE with the gap named.
vh