Files
esh-pfi-infrastructure/docs/runbooks
vh 4979869731 feat(backups): discover restic repos instead of enumerating them
Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.

The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).

The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
    "a repo exists that nobody watches"  -> impossible now, it is discovered
    "a repo we expected is gone"         -> EXPECTED REPO NOT FOUND

Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.

Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.

Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
2026-09-22 13:36:57 -07:00
..