docs: document rest-server-ana data layout + repo-recreate flow

rest-server-ana README now describes the /mnt/backup/restic/repo/ top-level NFS mount and its three per-site subdirs: ana/ — live data served by this rest-server (per-host repos + .htpasswd) — what DATA_DIR points at esh/ — mirror destination for ESH-site backups (pending) nh3/ — mirror destination for NH3 Synology's tree (pending) ana-ml2 README gains a proper "Recreating the repo" section with the correct /mnt/backup/restic/repo/ana/ana-ml2/ path for wiping the old repo after a lost passphrase, and two paths for regenerating keys: - interactive: type a user-generated passphrase at restic's init prompt, then install it into /etc/restic/password via `cat > file` + Ctrl-D (no shell history or transcript exposure) - scripted: openssl rand -base64 48, passphrase prints once and must be captured into the password manager immediately Cross-site replication snippet in rest-server-ana README updated to use the unified /mnt/backup/restic/repo/{esh,nh3}/ destinations instead of the earlier restic-mirror-*/ staging paths.
2026-04-20 23:26:17 -07:00
parent f5703550fb
commit f5cc60bcfc
2 changed files with 102 additions and 17 deletions
@@ -87,27 +87,91 @@ manager or rotate (see "Recreating the repo" below).
 **Do not run `restic init`** — the repo is already initialized; init
 against an existing repo errors out deliberately.

-### If no repo exists yet (fresh setup)
+### Recreating the repo (lost passphrase / starting fresh)

-Only applicable if you're starting from zero — otherwise skip.
+If `restic snapshots` returns `wrong password or no key found` and the
+passphrase can't be recovered, the old repo's data is unrecoverable and
+the cleanest path is to wipe and reinit. If you're truly starting from
+zero (no prior htpasswd entry or repo at all), skip the
+`htpasswd` step — only the init is needed.
+
+The rest-server's data root is `/mnt/backup/restic/repo/ana/` on
+ana-docker (NFS-mounted from the Debian NAS at 10.250.50.50). Per-host
+repos are direct subdirs — for `ana-ml2` that's
+`/mnt/backup/restic/repo/ana/ana-ml2/`. See
+`stacks/rest-server-ana/README.md` for the full sibling layout
+(`ana/`, `esh/`, `nh3/`).

 ```bash
+# 1. Wipe the old repo content on ana-docker. `--append-only` blocks
+#    restic's own delete ops but not direct filesystem removal — this
+#    is intentional for exactly this case.
 ssh -t ana-docker '
-  sudo htpasswd -B /path/to/rest-server/data/.htpasswd ana-ml2 &&  # prompts for password
-  docker restart rest-server
+  sudo rm -rf /mnt/backup/restic/repo/ana/ana-ml2 &&
+  sudo docker restart rest-server
 '

-ssh -t ana-ml2 '
-  # Write /etc/restic/restic.env and /etc/restic/password as above, using
-  # the fresh htpasswd password and a freshly generated passphrase:
-  openssl rand -base64 48  # save this in your password manager IMMEDIATELY
+# 2. Optional: rotate the htpasswd password for ana-ml2 at the same time.
+#    Produces a new hash; replace in place. (Skip if the existing
+#    htpasswd password is still trusted.)
+ssh ana-docker 'docker run --rm httpd:2.4-alpine htpasswd -nbB ana-ml2 "<new-pw>"' \
+  | ssh -t ana-docker 'sudo sed -i "/^ana-ml2:/d" /mnt/backup/restic/repo/ana/.htpasswd && sudo tee -a /mnt/backup/restic/repo/ana/.htpasswd >/dev/null'

-  sudo env $(cat /etc/restic/restic.env) \
-       RESTIC_PASSWORD_FILE=/etc/restic/password \
-       restic init
+# 3a. Init the repo — interactively prompt for a passphrase you generated
+#     in your password manager ahead of time. restic asks twice (init +
+#     confirm). This keeps the passphrase out of shell history / transcripts.
+ssh -t ana-ml2 '
+  sudo bash -c "
+    set -a
+    . /etc/restic/restic.env
+    set +a
+    restic init
+  "
+'
+
+# 3b. Install the same passphrase into /etc/restic/password so timers can
+#     run unattended. `cat > file` + Ctrl-D avoids the passphrase ever
+#     landing in your shell history or the ssh command line.
+ssh -t ana-ml2 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
+#   Terminal waits for input:
+#   - paste the same passphrase
+#   - press Enter
+#   - press Ctrl-D on an empty line
+#   (restic strips the trailing newline when reading the file.)
+
+# 3c. Verify the file-based passphrase agrees with what init registered.
+ssh -t ana-ml2 '
+  sudo bash -c "
+    set -a
+    . /etc/restic/restic.env
+    set +a
+    RESTIC_PASSWORD_FILE=/etc/restic/password restic snapshots
+  "
+'
+#   Expect: "no snapshots found" — repo is initialized and both paths
+#   (interactive and file-based) decrypt it.
+```
+
+**Alternative (random machine-generated passphrase):** if you want restic
+to generate the passphrase rather than supplying one from your password
+manager, replace 3a/3b with:
+
+```bash
+ssh -t ana-ml2 '
+  sudo bash -c "
+    openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
+    chmod 600 /etc/restic/password
+    cat /etc/restic/password  # copy this to your password manager NOW
+    echo
+    set -a; . /etc/restic/restic.env; set +a
+    RESTIC_PASSWORD_FILE=/etc/restic/password restic init
+  "
 '
 ```

+The passphrase prints to the terminal exactly once — don't close the
+window before copying it. Losing it = losing this repo.
+
 ### 3. Install resticprofile on ana-ml2

 The host has `restic` but not `resticprofile`. From the resticprofile
@@ -4,7 +4,21 @@ Anaheim-site restic backup endpoint. Replaces the older `restic` stack on ana-do

 **Server:** ana-docker (`10.250.50.70`)
 **Port:** `http://10.250.50.70:8000`
-**Data:** `/mnt/backup/restic/repo/ana/` (NFS mount on the host, served by the Debian file server at `10.250.50.50`)
+**Data root:** `/mnt/backup/restic/repo/` (NFS mount on the host, served by the Debian file server at `10.250.50.50`)
+
+## Data layout
+
+The NFS export at `/mnt/backup/restic/repo/` is partitioned by site into three sibling subdirs:
+
+| Path | Role |
+|---|---|
+| `/mnt/backup/restic/repo/ana/` | **Live data served by this rest-server.** Each Anaheim-side host writes its repo under here (e.g. `ana/ana-docker/`, `ana/ana-ml2/`). `.htpasswd` also lives here. |
+| `/mnt/backup/restic/repo/esh/` | Destination for mirroring ESH-site backups to Anaheim (cross-site redundancy; not yet wired up). |
+| `/mnt/backup/restic/repo/nh3/` | Destination for mirroring NH3-site backups (from the Synology `rest-server-nh3`) into Anaheim. Same — pending. |
+
+The rest-server container is configured with `/mnt/backup/restic/repo/ana` as its data root (see compose's `DATA_DIR` env), so clients use URLs of the form `rest:http://user:pw@10.250.50.70:8000/<host>/` — the `<host>` path is relative to `ana/`.
+
+The two mirror dirs (`esh/`, `nh3/`) aren't served by restic at all; they're rsync destinations.

 Paired with:
 - **`rest-server-nh3`** on the Synology (`10.100.50.50:8000`, data on Btrfs).
@@ -131,13 +145,20 @@ If you go the second-endpoint route, copy this stack to `stacks/rest-server-ana-

 ## Off-site replication

-Scheduled on ana-docker (to be written):
+Scheduled on ana-docker (to be written). Mirror destinations live in the
+sibling subdirs described under "Data layout" above:

 ```bash
-# Pull NH3's tree down to this side
-rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic-mirror-nh3/
-# Push our tree to NH3
+# Pull NH3's tree into ana-docker's /mnt/backup/restic/repo/nh3/
+rsync -avz --delete admin@10.100.50.50:/volume1/Backup/restic/ /mnt/backup/restic/repo/nh3/
+
+# Pull the ESH rest-server's tree into /mnt/backup/restic/repo/esh/
+# (once an ESH rest-server is set up — currently ESH hosts write directly
+# to this Anaheim rest-server at ana/esh-*)
+
+# Push our tree (ana/) to NH3 for redundancy in the other direction
 rsync -avz --delete /mnt/backup/restic/repo/ana/ admin@10.100.50.50:/volume1/Backup/restic-mirror-ana/
 ```

-Two unidirectional syncs, each running in the direction its data flows. Prune runs only at the origin so the mirror shrinks correctly.
+Each sync unidirectional, running in the direction its data flows. Prune
+runs only at the origin so the mirror shrinks correctly.