feat(backups): freshness check + daily alert timer; record rest-server-ana recovery, fstab hardening, esh-pve-nas gap, worldtree admin-key provisioning

docs/runbooks/backups.md

@@ -28,8 +28,8 @@ layer has failed.
 | **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** |
 | nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
 | esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
-| esh-vm-db | VM (esh-pve-nas?) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only (a DB!) |
-| vm-esh-nas | VM (esh-pve-nas) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only |
+| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
+| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
 | other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
 | SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
 
@@ -150,19 +150,27 @@ the stop→umount→rm-ghost→remount→start variant.
 
 ## Known gaps / TODO
 
-- [ ] **No backup-freshness alerting** — the systemic miss; the ana
-      outage went unnoticed ~6.5 weeks. Add a check (e.g. a daily job that
-      alerts if any repo's newest snapshot is > 48 h old, or wire
-      resticprofile's check/`--json` output to a notifier).
-- [ ] **fstab hardening** — change ana-docker's `/mnt/backup` entry from
-      `defaults` to `_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30,retry`
-      so it self-heals on reboot/NAS-blip instead of getting stuck.
-- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer.
-      Acceptable IF restic stays healthy + alerted; bulk `/tank` models are
-      re-downloadable, but bespoke quants/configs/scripts are not.
-- [ ] **Verify esh-pve-nas coverage** — vm-esh-nas + **esh-vm-db (a
-      postgres+mongo DB VM)** have no PBS namespace. Confirm whether they
-      get a PBS image anywhere; if not, restic is their sole net.
-- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth
-      creds were exposed during the 2026-06-20 diagnosis (Backrest config
-      dump). Add to the disclosed-keys rotation queue.
+- [x] **Backup-freshness alerting — DONE (2026-06-20).**
+      `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
+      stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
+      (`scripts/install-backup-freshness-timer.sh`) → `backup-freshness-alert.sh`
+      posts an **althing alert to infra-ops** on any stale/down layer. Run the
+      check by hand anytime. (Channel is althing for now — swap in email/ntfy if
+      you want a louder one.)
+- [x] **fstab hardening — DONE (2026-06-20).** ana-docker `/mnt/backup` →
+      `noauto,x-systemd.automount,x-systemd.mount-timeout=30` (autofs self-heals
+      on a NAS blip instead of getting stuck `failed`; activates on next reboot).
+      `/etc/fstab.bak-pre-harden` saved. **NOTE:** `/mnt/compose` is still bare
+      `defaults` (same latent risk) — harden it the same way when convenient.
+- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer; now
+      healthy + alerted. Bulk `/tank` models are re-downloadable; bespoke
+      quants/configs/scripts are the real loss-risk.
+- [x] **esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered.** No
+      `esh-pve-nas` namespace exists on PBS-ANA, so **esh-vm-db (postgres+mongo)
+      + vm-esh-nas are restic-only.** For the DB VM, restic-with-dumps is the
+      *preferred* method (vs a VM image) **IF** the resticprofile includes
+      `pg_dump`/`mongodump` — confirm that. Optionally add esh-pve-nas as a PBS
+      source. ESH is home-lab (no SLA).
+- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth creds
+      were exposed during the 2026-06-20 diagnosis. **BELAYED** — operator
+      handling offline.