feat(backups): discover restic repos instead of enumerating them

Adding nh3-dev to the host list fixed the instance. This fixes the class, on
svos-dev's framing: a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means the next repo added is
unwatched BY DEFAULT and nothing says so. The list of what to check can
silently disagree with the set of what exists -- the same shape as every other
instrument fault found this day, only slower-acting.

The check now asks each NAS. A directory is a repository when it has a
snapshots/ child, which cleanly separates real repos from container dirs
(/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of
its own -- verified rather than assumed before building discovery on the
layout).

The hand-written list survives DEMOTED to an EXPECTED set, used only to report
a repo that has VANISHED. Two facts that would otherwise both read as silence
stay distinct:
    "a repo exists that nobody watches"  -> impossible now, it is discovered
    "a repo we expected is gone"         -> EXPECTED REPO NOT FOUND

Preventive, not corrective: all 8 repos currently discovered are already in the
expected sets, so this found no live gap. It removes the possibility of the
next one.

Controls run, since a check only ever seen passing is untested: a bogus
expected repo reports EXPECTED REPO NOT FOUND and turns the verdict STALE;
unchanged expectations still report all-fresh; all 8 repos report their age.

Observed while testing, not a fault: restic/ana/esh-docker-vm is 36h old
against 12h for every other repo. Inside the 48h threshold so correctly green,
but it is a day behind the fleet and worth a look.
This commit is contained in:
vh
2026-09-22 13:36:57 -07:00
parent ba60fda16c
commit 4979869731
2 changed files with 83 additions and 13 deletions
+26
View File
@@ -209,6 +209,32 @@ reachability):
hops through `infra-ops@localhost`, which has it. The credentials under
`/etc/restic` are deliberately root-only.
### Repos are DISCOVERED, not enumerated
The host list is no longer hand-written. `check-backup-freshness.sh` asks each
NAS which repositories exist — a directory counts as one when it has a
`snapshots/` child, which distinguishes a real repo from a container dir
(`/volume1/Backup/restic/repo/` holds `ana|esh|nh3` namespaces and no snapshots
of its own).
**Why (svos-dev's framing, 2026-09-22):** adding nh3-dev fixed the *instance*;
the *class* was that a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means **the next repo added is
unwatched by default and nothing says so.** The list of what to check can
silently disagree with the set of what exists.
The hand-written list survives **demoted**, as `EXPECTED_RESTIC_ANA` /
`EXPECTED_RESTIC_NH3`, used only to report a repo that has *vanished*. That
keeps two facts distinct that would otherwise both read as silence:
| fact | now |
|---|---|
| a repo exists that nobody watches | **impossible** — it is discovered |
| a repo we expected is gone | reported as `EXPECTED REPO NOT FOUND` |
Currently 8 repos: 5 on ana-nas (`repo/ana/*`), 3 on nh3-nas. The `esh` and
`nh3` namespaces under `repo/` on ana-nas are empty.
⚠ **`restic ls` always prints a header line**, matched or not — a path absent
from the repo returns 1 line, a real one returned 6. Counting with `grep -c .`
read the header as a hit and made the ABSENT case fall through to *"blobs