Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.
The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).
The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
"a repo exists that nobody watches" -> impossible now, it is discovered
"a repo we expected is gone" -> EXPECTED REPO NOT FOUND
Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.
Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.
Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.
⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.
THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.
THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
- CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
present AND a restore returns non-zero bytes. "A snapshot exists containing
X" is satisfied by a three-month-old one; "the latest is recent" is
satisfied by an empty one. Age alone was the old problem; content alone is
the same problem rotated.
- IT RESTORES. Metadata is what survives the failures worth fearing -- a
pruned or partially-written repo can list a path whose blobs are gone.
~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
a repo that returns one file will return others.
- IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
so the probe hops through infra-ops@localhost. /etc/restic is root-only by
design.
⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.
TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.
The check collapsed two different findings into one verdict. On 2026-09-20 it
printed "RESULT: STALE" while reporting 37 FRESH layers and zero stale ones --
every backup body provably current, the three ❌ rows all yesterday's pre-fix
runs aging out of the 36h window. infra-hermes caught it in triage: a reader,
or a forwarder, could page someone over a state where nothing is stale.
STALE is a claim about backup AGE. A job that ran and errored is a different
claim with different urgency. They now have different words and different exit
codes:
0 all backups fresh
1 STALE -- a body past the threshold, or an endpoint down
3 ERRORED-JOBS -- every body fresh, a vzdump job errored recently
The alert wrapper mirrors the code and matches its own wording to the finding:
🟡 "Backup jobs errored — all bodies fresh" instead of 🔴 "Backup freshness
ALERT", and it now exits with the check's code rather than flattening
everything to 1, so `systemctl status` distinguishes the states too.
This is the same defect class the rest of this script was built to fix, one
level up: not an instrument that fails to look, but one that looks correctly
and then reports the wrong word for what it saw. An alarm that cries outage
over a healthy fleet earns being ignored exactly as fast as one that stays
silent over a broken one.
Verified all three states by forcing each: BACKUP_JOB_WINDOW_HOURS=1 -> exit 0,
default -> exit 3, BACKUP_MAX_AGE_HOURS=1 -> exit 1.
Snapshot age is structurally blind to a backup job that executes every night
and fails every night. Nothing new is written, so the group simply ages, and
the fault only surfaces once it crosses the 48h threshold -- days after the
first failure, with the evidence sitting in a task log nobody reads.
Two live cases, both found today and both invisible for a week by this exact
mechanism:
* esh-nas-pve CT 107 (vm-jellyfin): a backup run died around 09-06 and left
a stale `backup` lock, so every nightly since failed instantly with "CT is
locked (backup)". Age named it on ~09-12. Task status would have named it
on 09-07.
* esh-pve VM 102 (esh-vm-workstation): failing nightly since ~09-06 with
"timeout waiting on systemd". Same six-day gap.
PVE already records every task result in /var/log/pve/tasks/index. This reads
it on all four non-tenant PVE nodes and reports any vzdump in the last 36h
whose status is not OK, as its own section that sets the exit code.
It found a third case on its first run: esh-nas-pve's job had been reporting
`job errors` nightly while every guest on that node read 0-1h fresh, so no
age-based check could ever have flagged it.
Window is BACKUP_JOB_WINDOW_HOURS (default 36 -- longer than a daily cycle so
one missed run does not hide a failure). A node whose task log cannot be read
is reported, never assumed healthy.
ana-scale (CT 114) is a subnet-router LXC, excluded from vzdump on 2026-09-07
after a backup lock on its ESH counterpart blackholed that entire site. The
freshness check knew nothing about that and reported it 🔴 STALE every single
morning, which is how an alarm teaches you to ignore it.
Such guests now get their own section: printed every run, never hidden, and
not counted as a fault.
The subtlety is in how coverage is computed, and the obvious implementation is
wrong twice over:
* Reading one job's `exclude` list gets ana CT 109 (ana-nas) exactly
backwards. It IS excluded from the 03:00 all-guests job AND it has its own
dedicated 22:00 job. Suppressing on the exclude list would have stopped
alarming on a guest that is genuinely backed up -- trading a noisy alarm
for a blind one.
* ESH's job uses an explicit `vmid 100..107` INCLUDE list, so esh-scale 108
is excluded by OMISSION and appears in no exclude list at all.
So coverage is a union across every enabled job on the cluster, and a guest is
"intentionally not backed up" only when none of them covers it.
If coverage cannot be read, nothing is suppressed and the gap is reported: an
unreachable PVE node means we do not know, and a backup alarm must fail loud.
The SureFire namespace is never consulted (tenant property), so its guests can
never be suppressed either.
Verified against the live fleet on all four paths: CT 114 suppressed; CT 109
NOT suppressed despite being in an exclude list; esh-vm-workstation 102, which
a job really does cover and which really is failing, still reports STALE; and
with a PVE node made unreachable, 114 returns to STALE with the gap named.
vh