docs(backups): add backup architecture + freshness runbook; record rest-server-ana recovery + correct ana-docker sudo path

persistent-memory.md

@@ -98,12 +98,14 @@ no longer deployed sidecars here. See Recent decisions.)
   playbooks run fully non-interactive over key SSH.
 
 - **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
-  on most PFI Linux boxes (corviduo-dev included since 2026-06-15) — BUT
-  **on ana-docker the session lands as `lkraven` (docker-group, NO
-  passwordless sudo)**. So on ana-docker: docker commands work directly;
-  root-owned-file edits need the docker-as-root trick or infra-ops; but
-  the litellm config + most stack compose/conf are `lkraven`-owned →
-  plain `cp`/edit, no sudo. (Learned 2026-06-20 — see Tried-and-abandoned.)
+  on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
+  ana-docker there are TWO identities: the **default `ssh ana-docker` =
+  `lkraven`** (docker-group, NO passwordless sudo — docker works, root-file
+  edits don't); **but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (verified
+  2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files,
+  service control), use `ssh infra-ops@ana-docker`, NOT the default session.**
+  lkraven-owned files (litellm config, most stack compose/conf) still take
+  plain `cp`/edit under either identity.
 
 ## Current state / in-flight
 
@@ -190,6 +192,16 @@ _As of 2026-06-20:_
   **⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5
   per-repo rest-server basic-auth passwords into the session transcript →
   rotate after recovery (add to the disclosed-keys rotation queue).
+  **✅ RECOVERED (2026-06-20):** via `ssh infra-ops@ana-docker` (NOT the default
+  lkraven session) — `sudo mount -a` + `reset-failed mnt-backup.mount` re-attached
+  the NFS, then `docker compose up -d --force-recreate` rebound rest-server onto
+  the real `/data` (now `Loaded htpasswd file`, 401, healthy). Verified end-to-end:
+  triggered backups on the two no-PBS hosts' priority — **ana-ml2 snapshot landed
+  22:53Z** (total 19→20); ana-docker's larger catch-up was still running at hand-off
+  (will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00
+  timers (or trigger manually). **Full topology + 2-min freshness-check now documented
+  in `docs/runbooks/backups.md`.** STILL OPEN (prevention): backup-freshness alert +
+  fstab `_netdev`/retry hardening + verify esh-pve-nas PBS gap + rotate the 5 creds.
 
 - **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated
   `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs
@@ -377,11 +389,13 @@ _101 older entries archived to archival-memory.md._
   state). Prune them with `docker buildx prune --builder <name> -af`, or
   `docker volume rm` orphaned ones (builder already removed).
 
-- `[2026-06-20]` **On ana-docker I am `lkraven` (no NOPASSWD sudo), not
-  infra-ops.** A `sudo cp` of the litellm config silently failed (password
-  prompt) and I restarted the gateway on an unchanged config (one wasted
-  bounce). The config + most stack files are `lkraven`-owned → plain `cp`,
-  no sudo. Verify the SSH user/ownership before reaching for sudo.
+- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) —
+  but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day).
+  Early on a `sudo cp` as lkraven silently failed (password prompt) → one
+  wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and
+  nearly punted the rest-server-ana recovery to the operator. The real rule:
+  reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm
+  config, most stack compose/conf) take plain `cp` under either identity.
 
 - `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.