diff --git a/configs/restic/esh-docker-vm/profiles.yaml b/configs/restic/esh-docker-vm/profiles.yaml
index c43520f..7f3ad5d 100644
--- a/configs/restic/esh-docker-vm/profiles.yaml
+++ b/configs/restic/esh-docker-vm/profiles.yaml
@@ -52,8 +52,8 @@ default:
       # Docker internals
       - /var/lib/docker/volumes/backingFsBlockDev
       - /var/lib/docker/volumes/metadata.db
-      # Offen sidecar's buffer / tmp (if any left behind)
-      - /var/lib/docker/volumes/*_offen_backup_data
+      # Offen `*_offen_backup_data` exclude removed 2026-04-23 — offen
+      # sidecars retired fleet-wide; no more offen-scratch volumes exist.
       # Ephemeral / regenerable
       - /opt/docker/compose/*/logs
       - "**/*.log"
diff --git a/configs/restic/esh-vm-db/README.md b/configs/restic/esh-vm-db/README.md
new file mode 100644
index 0000000..345928a
--- /dev/null
+++ b/configs/restic/esh-vm-db/README.md
@@ -0,0 +1,164 @@
+# restic / esh-vm-db
+
+**Two-database host** at the ESH site (PostgreSQL 15 + MongoDB). Covered
+at the VM-image layer by PBS-ANA via esh-pve (or whichever ESH
+hypervisor owns this VM — confirm on next inventory pass). This restic
+profile adds DB-level granularity via pre-backup dumps.
+
+## What's backed up
+
+| Path | Purpose |
+|---|---|
+| `/etc` | Host config — systemd, ssh, chrony, apt, pg_hba.conf, mongod.conf |
+| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
+| `/home` | User homes (lkraven + any DB-admin locals) |
+| `/var/lib/restic/stage` | **pg_dumpall.sql.gz** + **mongodump/** produced by pre-backup.sh |
+
+## What's **not** backed up (by design)
+
+- **`/var/lib/postgresql`** — raw PGDATA. Live-capture risk;
+  `pg_dumpall` in pre-backup covers it consistently.
+- **`/var/lib/mongodb`** — raw mongo dbPath. Same reasoning;
+  `mongodump` covers it.
+- NFS mount `/mnt/backup` (from esh-nas — not ours to mirror).
+
+## Pre-backup hook
+
+`pre-backup.sh` runs as root before restic. It:
+
+1. Checks `pg_isready` on :5432 — if OK, runs `pg_dumpall` piped
+   through gzip to `$STAGE/pg_dumpall.sql.gz`
+2. Checks mongo ping via `mongosh` — if OK, runs `mongodump` into
+   `$STAGE/mongodump/`
+
+Both dumps are atomic (write to `.tmp`, then rename). If either DB is
+unreachable, the script logs a WARN and continues — a failed DB dump
+doesn't abort the whole restic run, and restic falls back to whatever
+stage content is left over from the prior successful dump.
+
+## Deploy (one-time)
+
+### 1. Create rest-server-ana htpasswd entry
+
+**Do NOT use `sudo` for .htpasswd writes on ana-docker.** The file is
+NFS-mounted from ana-nas and owned by uid 1000 (the rest-server user,
+which equals lkraven). Sudo-root on the client gets squashed to
+nobody on the NFS server and can't read/write the file. lkraven
+writes it natively, using the docker group for the bcrypt helper.
+
+```bash
+# Pick password in password manager first
+HTPW='<new-pw-saved-to-pw-manager>'
+
+ssh -t ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB esh-vm-db '$HTPW' | \
+                   tee /tmp/htline.txt > /dev/null && \
+                   sed -i '/^esh-vm-db:/d' /mnt/backup/restic/repo/ana/.htpasswd && \
+                   cat /tmp/htline.txt >> /mnt/backup/restic/repo/ana/.htpasswd && \
+                   rm /tmp/htline.txt && \
+                   grep ^esh-vm-db: /mnt/backup/restic/repo/ana/.htpasswd && \
+                   docker restart rest-server"
+unset HTPW
+```
+
+### 2. Install secrets on esh-vm-db
+
+```bash
+ssh -t esh-vm-db 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic /var/lib/restic/stage'
+
+# restic.env — URL-encode the password if it has special chars
+ssh -t esh-vm-db "sudo bash -c '
+  read -sp \"htpasswd pw for rest-server-ana: \" HTPW; echo
+  cat > /etc/restic/restic.env <<EOF
+RESTIC_REPOSITORY=rest:http://esh-vm-db:\$HTPW@10.250.50.70:8000/esh-vm-db/
+EOF
+  chmod 600 /etc/restic/restic.env
+'"
+
+# Repo passphrase (prints once — save to password manager)
+ssh -t esh-vm-db 'sudo bash -c "
+  openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
+  chmod 600 /etc/restic/password
+  echo === SAVE THIS TO PASSWORD MANAGER NOW ===
+  cat /etc/restic/password
+  echo
+"'
+```
+
+### 3. Initialize the repo
+
+```bash
+ssh -t esh-vm-db 'sudo bash -c "
+  set -a; . /etc/restic/restic.env; set +a
+  RESTIC_PASSWORD_FILE=/etc/restic/password restic init
+"'
+```
+
+### 4. Install prerequisites (restic, resticprofile, mongosh client)
+
+```bash
+ssh -t esh-vm-db 'which restic || sudo apt-get install -y restic; \
+                  which mongosh || echo "NOTE: mongosh not found; pre-backup mongo ping will fail safely — install via MongoDB APT repo if needed"; \
+                  curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin; \
+                  /usr/local/bin/resticprofile --version'
+```
+
+### 5. Deploy profile + hook
+
+```bash
+scp configs/restic/esh-vm-db/profiles.yaml esh-vm-db:/tmp/
+scp configs/restic/esh-vm-db/pre-backup.sh esh-vm-db:/tmp/
+
+ssh -t esh-vm-db 'sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml && \
+                  sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh && \
+                  rm /tmp/profiles.yaml /tmp/pre-backup.sh'
+```
+
+### 6. Schedule + verify
+
+```bash
+ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml schedule --all && \
+                  systemctl list-timers "resticprofile*" --no-pager'
+
+# First manual run
+ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
+```
+
+Expect first run to land ~50-200 MB (mostly the mongodump directory + pg_dumpall).
+Cross-check from Backrest UI on ana-docker.
+
+## Restore
+
+### Full host config
+
+```bash
+ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /etc'
+```
+
+### Just the PG dump
+
+```bash
+ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /var/lib/restic/stage/pg_dumpall.sql.gz'
+# Then: gunzip + psql < pg_dumpall.sql
+```
+
+### Just a mongo DB
+
+```bash
+ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /var/lib/restic/stage/mongodump'
+# Then: mongorestore /tmp/restore/var/lib/restic/stage/mongodump/
+```
+
+## Gotchas
+
+- **mongosh must be installed** or the mongo pre-backup step silently
+  skips (logged as WARN). Install from the MongoDB APT repo if not
+  already present — the stock Debian `mongodb-clients` package is
+  out of date and doesn't include `mongosh`.
+- **Mongo authentication** — if mongod ever gets auth enabled (it's
+  currently open to 0.0.0.0 with no auth, which is its own concern),
+  `mongodump` will need `--username/--password` flags. Reference in
+  pre-backup.sh when that change happens.
+- **pg_hba.conf** — `pg_dumpall` requires local postgres superuser
+  access. Currently works via `sudo -u postgres` + peer auth on the
+  local socket. If `pg_hba.conf` ever changes peer → md5 for local,
+  the hook needs a `~postgres/.pgpass` entry.
diff --git a/configs/restic/esh-vm-db/pre-backup.sh b/configs/restic/esh-vm-db/pre-backup.sh
new file mode 100644
index 0000000..1c46806
--- /dev/null
+++ b/configs/restic/esh-vm-db/pre-backup.sh
@@ -0,0 +1,59 @@
+#!/bin/bash
+# pre-backup.sh — esh-vm-db.
+# Runs as root from resticprofile's `run-before`.
+#
+# Produces consistent DB dumps in /var/lib/restic/stage/. Two DBs here:
+#   - Postgres 15 (port 5432, local) — pg_dumpall all databases
+#   - MongoDB (port 27017, local) — mongodump all databases
+#
+# Peer-data DBs (paperless-ng on Postgres) are the primary consumers;
+# raw volume capture isn't in the restic source list so these dumps
+# are the ONLY way restic sees DB data.
+#
+# Errors in individual blocks log a WARN; whole script doesn't abort.
+
+set -euo pipefail
+
+STAGE=/var/lib/restic/stage
+install -d -o root -g root -m 0700 "$STAGE"
+
+log() { printf '%s pre-backup(esh-vm-db): %s\n' "$(date -Is)" "$*"; }
+warn() { printf '%s pre-backup(esh-vm-db): WARN: %s\n' "$(date -Is)" "$*" >&2; }
+
+# ---- Postgres ----------------------------------------------------------
+PG_DUMP="$STAGE/pg_dumpall.sql.gz"
+if sudo -u postgres pg_isready -h localhost -p 5432 > /dev/null 2>&1; then
+  log "pg_dumpall starting → $PG_DUMP"
+  if sudo -u postgres pg_dumpall -h localhost -p 5432 | gzip > "$PG_DUMP.tmp"; then
+    mv "$PG_DUMP.tmp" "$PG_DUMP"
+    log "pg_dumpall done ($(du -h "$PG_DUMP" | cut -f1))"
+  else
+    warn "pg_dumpall failed (exit $?); keeping previous dump if any"
+    rm -f "$PG_DUMP.tmp"
+  fi
+else
+  warn "postgres not ready on :5432 — skipping pg_dumpall"
+fi
+
+# ---- MongoDB -----------------------------------------------------------
+MONGO_DIR="$STAGE/mongodump"
+if mongosh --quiet --eval 'db.adminCommand({ping: 1}).ok' | grep -q '^1$'; then
+  log "mongodump starting → $MONGO_DIR"
+  rm -rf "$MONGO_DIR.tmp"
+  if mongodump --out "$MONGO_DIR.tmp" > /dev/null 2>&1; then
+    rm -rf "$MONGO_DIR"
+    mv "$MONGO_DIR.tmp" "$MONGO_DIR"
+    log "mongodump done ($(du -sh "$MONGO_DIR" | cut -f1))"
+  else
+    warn "mongodump failed (exit $?); keeping previous dump if any"
+    rm -rf "$MONGO_DIR.tmp"
+  fi
+else
+  warn "mongo not reachable via mongosh — skipping mongodump"
+fi
+
+# ---- Retention on stage dir --------------------------------------------
+# restic dedupes identical dumps at the chunk level, so we can safely keep
+# overwriting the same files. No explicit rotation needed here.
+
+log "pre-backup complete"
diff --git a/configs/restic/esh-vm-db/profiles.yaml b/configs/restic/esh-vm-db/profiles.yaml
new file mode 100644
index 0000000..4d88f15
--- /dev/null
+++ b/configs/restic/esh-vm-db/profiles.yaml
@@ -0,0 +1,89 @@
+# resticprofile config for esh-vm-db.
+#
+# Two-database host (Postgres 15 on :5432, MongoDB on :27017). Small
+# VM on an ESH hypervisor; backed up at VM-image level by PBS-ANA via
+# its hypervisor. This restic profile captures:
+#
+#   1. Host config (/etc, /root) — fast config recovery without
+#      waiting for VM-image restore
+#   2. Per-DB dumps via pre-backup hook — pg_dumpall + mongodump,
+#      written to /var/lib/restic/stage/ and included in the restic
+#      snapshot. Gives us DB-level restore granularity alongside the
+#      VM-image restore from PBS.
+#
+# What's DELIBERATELY NOT in source:
+#   - /var/lib/postgresql — raw live PGDATA. Inconsistent if captured
+#     while postgres is running; pg_dumpall above covers this.
+#   - /var/lib/mongodb — same reasoning; mongodump covers it.
+#
+# Writes cross-site to rest-server-ana (10.250.50.70:8000). No local
+# NH3-style rest-server on the ESH side; esh-docker-vm and vm-esh-nas
+# also use rest-server-ana, so this follows fleet pattern.
+
+version: "1"
+
+global:
+  priority: low
+  ionice: true
+  ionice-class: 2
+  ionice-level: 7
+  min-memory: 100
+
+default:
+  env-file: /etc/restic/restic.env
+  env:
+    RESTIC_PASSWORD_FILE: /etc/restic/password
+  initialize: false
+  lock: /var/lock/restic-esh-vm-db.lock
+
+  backup:
+    verbose: 1
+    run-before:
+      - /etc/restic/pre-backup.sh
+    run-after:
+      - date +%s > /var/lib/restic/last-success
+    source:
+      - /etc                   # host config
+      - /root                  # root's scripts, ssh keys, shell history
+      - /home                  # user home dirs
+      - /var/lib/restic/stage  # pg_dumpall + mongodump outputs from pre-backup
+    exclude:
+      # Raw DB data is captured via pre-backup dumps, not volume-level
+      - /var/lib/postgresql
+      - /var/lib/mongodb
+      # NFS mount from esh-nas (backup target for other stacks — not ours to mirror)
+      - /mnt/backup
+      # Ephemeral / regenerable
+      - "**/*.log"
+      - "**/*.log.*"
+      - "**/*.pid"
+      - /root/.cache
+      - /root/.local/share/Trash
+      - /root/.npm
+      - /root/.python_history
+      - /home/*/.cache
+      - /home/*/.local/share/Trash
+      - /home/*/.npm
+    tag:
+      - host:esh-vm-db
+      - site:esh
+      - fleet:home-lab
+    schedule: "*-*-* 01:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-backup.log
+
+  forget:
+    keep-daily: 7
+    keep-weekly: 4
+    keep-monthly: 12
+    keep-yearly: 3
+    tag:
+      - host:esh-vm-db
+    # Schedule removed: forget against --append-only rest-server always
+    # fails (delete ops blocked). Run manually during prune ceremony.
+
+  check:
+    read-data-subset: 10%
+    schedule: "Sun *-*-* 05:00:00"
+    schedule-permission: system
+    schedule-log: /var/log/restic-check.log
diff --git a/configs/rsync/ana-nas-to-nh3/README.md b/configs/rsync/ana-nas-to-nh3/README.md
new file mode 100644
index 0000000..0e2d1cb
--- /dev/null
+++ b/configs/rsync/ana-nas-to-nh3/README.md
@@ -0,0 +1,96 @@
+# ana-nas → nh3-nas restic mirror
+
+Nightly rsync of the Ana-side restic repo to the NH3 Synology, giving
+file-level backups cross-site redundancy independent of the PBS layer.
+
+## What + where
+
+| | |
+|---|---|
+| **Source** | `ana-nas:/mnt/backup/restic/repo/ana/` (ZFS `NASPool/backupStore`, written by rest-server-ana on ana-docker via NFS). |
+| **Target** | `nh3-nas:/volume1/Backup/restic-ana-mirror/` (Btrfs `/volume1`, 16 TB free at setup time). |
+| **Runs on** | `ana-nas` (Debian 12, VMID 100 on pfi-pve). |
+| **Runs as** | `lkraven` (uid 1000) — owns the source data natively; no sudo needed. |
+| **Auth** | Dedicated ed25519 keypair `~lkraven/.ssh/id_mirror_nh3` → `syncuser@nh3-nas`. No passphrase (for unattended systemd runs). |
+| **Schedule** | Daily at 04:00, `Persistent=true`, 300s randomized delay. |
+
+## Scheduling rationale
+
+Window sits between:
+- **01:00–03:00** — ana-side restic clients run their nightly backups
+  against rest-server-ana. We wait until those are clearly done before
+  reading the repo.
+- **06:00** — PBS-ANA → PBS-NH3 sync job pulls fresh snapshots across
+  the WAN. Running the restic mirror earlier avoids competing for WAN
+  bandwidth with the (much larger) PBS sync.
+
+## Files
+
+| File | Install path on ana-nas |
+|---|---|
+| `restic-mirror-to-nh3.service` | `/etc/systemd/system/restic-mirror-to-nh3.service` |
+| `restic-mirror-to-nh3.timer`   | `/etc/systemd/system/restic-mirror-to-nh3.timer` |
+
+These are the canonical copies; the on-host copies mirror them.
+
+## Deploy / redeploy
+
+```bash
+cd configs/rsync/ana-nas-to-nh3
+scp restic-mirror-to-nh3.{service,timer} ana-nas:/tmp/
+ssh -t ana-nas 'sudo install -m 644 /tmp/restic-mirror-to-nh3.service /etc/systemd/system/ && \
+                sudo install -m 644 /tmp/restic-mirror-to-nh3.timer   /etc/systemd/system/ && \
+                sudo systemctl daemon-reload && \
+                sudo systemctl enable --now restic-mirror-to-nh3.timer && \
+                sudo systemctl list-timers restic-mirror-to-nh3.timer'
+```
+
+## Operate
+
+```bash
+# Trigger a run manually (e.g. to smoke-test changes)
+ssh -t ana-nas 'sudo systemctl start restic-mirror-to-nh3.service'
+
+# Follow a running transfer
+ssh ana-nas 'journalctl -u restic-mirror-to-nh3.service -f'
+
+# Last run outcome
+ssh ana-nas 'systemctl status restic-mirror-to-nh3.service --no-pager'
+
+# When the timer fires next
+ssh ana-nas 'systemctl list-timers restic-mirror-to-nh3.timer'
+```
+
+## Restore flow (if ana-nas loses the repo)
+
+The mirror is just files — point a restic client directly at the
+nh3-nas copy. Options:
+
+**Option A — pull repo back to ana-nas and use rest-server-ana as
+before.** rsync in reverse:
+
+```bash
+ssh -t ana-nas 'sudo rsync -a \
+  -e "ssh -i /home/lkraven/.ssh/id_mirror_nh3" \
+  syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/ \
+  /mnt/backup/restic/repo/ana/'
+```
+
+**Option B — restic against the mirror directly.** Expose
+`/volume1/Backup/restic-ana-mirror/` via a temporary rest-server or
+NFS share, point the client at it for an emergency restore. Slow (WAN
+hop) but no dataset copy required.
+
+Target is `--append-only: false` on the mirror side — not a problem
+for restore, but means a compromised mirror side *could* be tampered
+with. Balanced against the operational cost of managing an
+append-only mirror, we accept this for a mirror-of-a-mirror.
+
+## What this does NOT cover
+
+- **PBS snapshots** — those replicate via PBS-ANA → PBS-NH3 (a
+  separate pipeline; see `docs/runbooks/pbs-deployment.md`).
+- **rest-server-nh3's own data** — nh3-docker + nh3-dev backups that
+  land on `/volume1/Backup/restic/<user>/`. Those are already at the
+  NH3 Synology; ANA-side doesn't mirror them currently. Symmetric
+  mirror back to ana-nas is a future-work item.
diff --git a/configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.service b/configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.service
new file mode 100644
index 0000000..aaf5576
--- /dev/null
+++ b/configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.service
@@ -0,0 +1,46 @@
+[Unit]
+Description=Mirror rest-server-ana restic repo to NH3 Synology
+Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
+After=network-online.target
+Wants=network-online.target
+ConditionPathIsDirectory=/mnt/backup/restic/repo/ana
+StartLimitBurst=3
+StartLimitIntervalSec=1h
+
+[Service]
+Type=oneshot
+User=lkraven
+Group=lkraven
+Nice=10
+IOSchedulingClass=idle
+
+# Dedicated key for this job, ed25519, ana-nas → nh3-nas:syncuser.
+# --append-only on rest-server-ana means source files are never
+# rewritten or deleted by clients; --delete here mirrors any explicit
+# prune operations (done out-of-band during the quarterly ceremony).
+# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
+ExecStart=/usr/bin/rsync \
+  --archive \
+  --delete \
+  --partial \
+  --info=stats2 \
+  --timeout=300 \
+  --exclude=tmp/ \
+  --exclude=.lock \
+  -e "ssh -i /home/lkraven/.ssh/id_mirror_nh3 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
+  /mnt/backup/restic/repo/ana/ \
+  syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/
+
+# ServerAlive{Interval,CountMax}=30/3 + rsync --timeout bound below
+# force detection of dead WAN within ~90s; without this, a silent TCP
+# drop holds the service in "activating" for hours on kernel retransmit
+# backoff. TimeoutStartSec=6h caps worst-case initial sync; after
+# that systemd kills the service even if rsync somehow wedges.
+TimeoutStartSec=6h
+
+# Tight retry window — a transient WAN blip shouldn't wedge the timer
+# schedule, but spamming retries on a real outage is noise.
+# StartLimitBurst/Interval live in [Unit] (systemd v230+); the burst
+# cap prevents runaway retries even with the short RestartSec.
+Restart=on-failure
+RestartSec=30s
diff --git a/configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.timer b/configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.timer
new file mode 100644
index 0000000..dba0e25
--- /dev/null
+++ b/configs/rsync/ana-nas-to-nh3/restic-mirror-to-nh3.timer
@@ -0,0 +1,17 @@
+[Unit]
+Description=Daily mirror of rest-server-ana restic repo to NH3
+Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
+
+[Timer]
+# 04:00 daily — sits between ana-side restic clients (01:00 finish
+# window) and the PBS-ANA → PBS-NH3 sync at 06:00, so WAN contention
+# is minimized. Persistent=true catches missed runs if ana-nas was
+# offline; RandomizedDelaySec smears load if this ever scales to
+# multiple mirror jobs on the same host.
+OnCalendar=*-*-* 04:00:00
+Persistent=true
+RandomizedDelaySec=300
+Unit=restic-mirror-to-nh3.service
+
+[Install]
+WantedBy=timers.target
diff --git a/configs/rsync/nh3-nas-to-ana/README.md b/configs/rsync/nh3-nas-to-ana/README.md
new file mode 100644
index 0000000..c7a7994
--- /dev/null
+++ b/configs/rsync/nh3-nas-to-ana/README.md
@@ -0,0 +1,104 @@
+# nh3-nas → ana-nas restic mirror (return leg)
+
+Symmetric counterpart to `configs/rsync/ana-nas-to-nh3/`. Nightly
+rsync of the NH3-side restic repo to the Ana NAS, so file-level
+backups for nh3-docker and nh3-dev have cross-site redundancy.
+
+## What + where
+
+| | |
+|---|---|
+| **Source** | `nh3-nas:/volume1/Backup/restic/` (Btrfs on cachedev_0, written by rest-server-nh3 on nh3-nas). Holds per-client repos: nh3-docker/, nh3-dev/, repo/. |
+| **Target** | `ana-nas:/mnt/backup/restic-nh3-mirror/` (ZFS `NASPool/backupStore`, 20 TB pool with ~20 TB free at setup). |
+| **Runs on** | `nh3-nas` (Synology RS2418+, DSM 7+ with native systemd). |
+| **Runs as** | **`root`** — required because rest-server-nh3's Docker container writes restic files as `admin:users mode 400`, and only root (via CAP_DAC_READ_SEARCH) can read them. Running as `syncuser` was tried first and denied. See gotchas below. |
+| **Auth** | Dedicated ed25519 keypair `/root/.ssh/id_mirror_ana` → `lkraven@ana-nas`. No passphrase. |
+| **Schedule** | Daily at 05:00, `Persistent=true`, 300s randomized delay. |
+
+## Why 05:00
+
+Slots between:
+- 04:00 ana-nas → nh3-nas (the other mirror) — lets that finish before
+  NH3 starts pushing anything back.
+- 06:00 PBS-ANA → PBS-NH3 sync — avoids WAN contention with the much
+  larger PBS replication.
+
+## DSM-specific notes
+
+- DSM 7.x ships a stripped/older systemd. Unit files in
+  `/etc/systemd/system/` persist across DSM updates but are NOT
+  guaranteed to — verify after each DSM major update.
+- **DSM systemd is pre-v247** — it refuses `Restart=on-failure` on
+  `Type=oneshot` services, and does not recognize `StartLimitBurst`
+  or `StartLimitIntervalSec`. This service deliberately omits those;
+  a failed run just means the daily timer picks up again tomorrow.
+  For a mirror used as DR (not primary), that's acceptable.
+- `systemctl --now` flag is also absent. Use `enable` + `start`
+  separately on DSM.
+- `syncuser` has `administrators` group membership, so sudo works
+  for unit installation. `syncuser` itself runs the service (no
+  privilege escalation at runtime).
+- rsync lives at `/usr/bin/rsync` on DSM (same as Debian). Older
+  `/bin/rsync` doesn't exist.
+- `scp` to DSM needs `-O` (legacy protocol) — sftp subsystem is off
+  by default. Either enable SFTP in DSM (**Control Panel → File
+  Services → FTP → SFTP**) or keep using `scp -O`.
+- **rest-server-nh3 writes restic files as `admin:users mode 400`**
+  (via DSM ContainerManager). syncuser cannot read these even with
+  admins-group ACL because the newer DSM doesn't ship `synoacltool`
+  to add per-user ACEs. The service therefore runs as root
+  (CAP_DAC_READ_SEARCH bypasses the 400 mode).
+- **rsync `--archive` broke cross-filesystem perm/owner preservation**
+  from Btrfs-with-syno_acl → ZFS-on-Debian — the dest dir ended up
+  mode `0000` and rsync couldn't recover on subsequent runs. Switched
+  to `-rlt --no-perms --no-owner --no-group --chmod=D755,F644` which
+  copies contents verbatim and forces sensible dest perms. Restic's
+  repo integrity doesn't rely on Unix perms so this is a clean swap.
+
+## Files
+
+| File | Install path on nh3-nas |
+|---|---|
+| `restic-mirror-to-ana.service` | `/etc/systemd/system/restic-mirror-to-ana.service` |
+| `restic-mirror-to-ana.timer`   | `/etc/systemd/system/restic-mirror-to-ana.timer` |
+
+## Prereqs (one-time setup)
+
+1. Generate the keypair on nh3-nas as syncuser:
+   ```bash
+   ssh nh3-nas "test -f ~/.ssh/id_mirror_ana || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_mirror_ana -C 'nh3-nas→ana-nas restic mirror'"
+   ```
+2. Install the pubkey on ana-nas:
+   ```bash
+   ssh nh3-nas 'cat ~/.ssh/id_mirror_ana.pub' | \
+     ssh ana-nas 'cat >> ~/.ssh/authorized_keys'
+   ```
+3. Create the destination dir on ana-nas:
+   ```bash
+   ssh ana-nas 'sudo mkdir -p /mnt/backup/restic-nh3-mirror && \
+                sudo chown lkraven:lkraven /mnt/backup/restic-nh3-mirror'
+   ```
+4. Smoke-test from nh3-nas as syncuser:
+   ```bash
+   ssh nh3-nas 'ssh -i ~/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new \
+                  lkraven@10.250.50.50 "hostname && ls -ld /mnt/backup/restic-nh3-mirror"'
+   ```
+
+## Deploy
+
+```bash
+cd configs/rsync/nh3-nas-to-ana
+scp restic-mirror-to-ana.{service,timer} nh3-nas:/tmp/
+ssh -t nh3-nas 'sudo install -m 644 /tmp/restic-mirror-to-ana.service /etc/systemd/system/ && \
+                sudo install -m 644 /tmp/restic-mirror-to-ana.timer   /etc/systemd/system/ && \
+                sudo systemctl daemon-reload && \
+                sudo systemctl enable --now restic-mirror-to-ana.timer && \
+                sudo systemctl start restic-mirror-to-ana.service && \
+                sudo systemctl list-timers restic-mirror-to-ana.timer'
+```
+
+## Operate
+
+Mirrors the ana-nas side — see that README. Replace
+`restic-mirror-to-nh3` with `restic-mirror-to-ana` and `ana-nas` with
+`nh3-nas` in the commands there.
diff --git a/configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.service b/configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.service
new file mode 100644
index 0000000..6b240b1
--- /dev/null
+++ b/configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.service
@@ -0,0 +1,52 @@
+[Unit]
+Description=Mirror rest-server-nh3 restic repo to ANA NAS
+Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/nh3-nas-to-ana/README.md
+After=network-online.target
+Wants=network-online.target
+ConditionPathIsDirectory=/volume1/Backup/restic
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+Nice=10
+IOSchedulingClass=idle
+
+# Runs as root because rest-server-nh3's Docker container writes the
+# restic repo files as admin:users mode 400 — only root bypasses that
+# via CAP_DAC_READ_SEARCH. Running as syncuser (even with admins
+# group membership) was denied by POSIX mode. Earlier attempt with
+# synoacltool ACL grants failed because this DSM version ships
+# without the tool.
+#
+# Dedicated key for this job, ed25519, nh3-nas:root → ana-nas:lkraven.
+# Does NOT include /volume1/Backup/restic-ana-mirror (that's data
+# ana-nas just sent us; mirroring it back would be a dedupe-less loop).
+# Source is the per-host tree under /volume1/Backup/restic/ only:
+# nh3-docker/, nh3-dev/, repo/.
+# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
+# SSH keepalive + rsync --timeout ensure a dead WAN is detected in
+# ~90s instead of hanging on kernel TCP retransmit backoff.
+#
+# NO Restart=on-failure here: DSM's systemd is pre-v247 and refuses
+# Restart= on Type=oneshot. If a run fails, the daily timer picks
+# up again tomorrow — a single missed mirror is acceptable for DR.
+ExecStart=/usr/bin/rsync \
+  --recursive \
+  --links \
+  --times \
+  --no-perms \
+  --no-owner \
+  --no-group \
+  --chmod=D755,F644 \
+  --delete \
+  --partial \
+  --info=stats2 \
+  --timeout=300 \
+  --exclude=tmp/ \
+  --exclude=.lock \
+  -e "ssh -i /root/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
+  /volume1/Backup/restic/ \
+  lkraven@10.250.50.50:/mnt/backup/restic-nh3-mirror/
+
+TimeoutStartSec=6h
diff --git a/configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.timer b/configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.timer
new file mode 100644
index 0000000..2b88829
--- /dev/null
+++ b/configs/rsync/nh3-nas-to-ana/restic-mirror-to-ana.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Daily mirror of rest-server-nh3 restic repo to ANA
+Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/nh3-nas-to-ana/README.md
+
+[Timer]
+# 05:00 daily — runs after ana→nh3 at 04:00, before PBS sync at
+# 06:00. Same persistence + jitter pattern.
+OnCalendar=*-*-* 05:00:00
+Persistent=true
+RandomizedDelaySec=300
+Unit=restic-mirror-to-ana.service
+
+[Install]
+WantedBy=timers.target
diff --git a/docs/runbooks/disaster-recovery.md b/docs/runbooks/disaster-recovery.md
new file mode 100644
index 0000000..1838e4b
--- /dev/null
+++ b/docs/runbooks/disaster-recovery.md
@@ -0,0 +1,276 @@
+# Fleet disaster-recovery runbook
+
+What breaks when a given host/service goes down, and how to recover.
+Ordered by blast-radius severity — read top-down.
+
+**First action for ANY multi-service outage:** check ana-nas
+reachability. Many fleet-wide "things are broken" events trace back
+to ana-nas's NFS exports going stale. See [ana-nas SPOF memory](../../../.claude/projects/-home-lkraven-development-eshpfi-management/memory/incident_ana_nas_spof.md)
+for details, or just `ping 10.250.50.50` first.
+
+## Governing principle: backups must not risk production
+
+Any backup strategy that could take down the host it's backing up is
+worse than the risk it mitigates. Specifically applies to:
+
+- **NFS-serving hosts (CT 109 / ana-nas):** scheduled vzdump of the
+  container itself is prohibited. The rootfs is trivially rebuildable;
+  the data lives on ospool bind-mounts captured via ZFS snapshot at
+  the host level. If rootfs config is worth preserving, capture via
+  `rsync /etc /root` to a known location — no snapshot, no freeze.
+- **DB servers on NFS storage (VM 105 / pfi-postgres):** prefer
+  `pg_dump` + restic over full vzdump. fs-freeze on NFS-backed
+  PGDATA is a failure mode waiting to happen.
+- **Any host with heavy I/O concurrency during the backup window:**
+  schedule its own backup outside the fleet window OR use `mode=stop`
+  (planned brief downtime beats random-freeze risk).
+
+Incident reference: 2026-04-23 — CT 109, 112, 113 all went offline on
+pfi-pve during morning backup window. Backup suspected as trigger
+(full root cause TBD). See `memory/incident_ana_nas_spof.md`.
+
+---
+
+## Tier 0 — Storage fabric (catastrophic blast radius)
+
+### ana-nas (CT 109 on pfi-pve, 10.250.50.50)
+
+**Blast radius:**
+- ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
+- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas)
+- PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails
+- ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior
+
+**Recovery** (see full procedure in `memory/incident_ana_nas_spof.md`):
+1. `pct start 109` on pfi-pve if stopped; `pct console 109` to see boot if stuck
+2. Once responsive, remount NFS on each consumer and restart services:
+   - pfi-postgres: `umount -lf /mnt/db; mount /mnt/db; systemctl restart postgresql`
+   - ana-docker: `sudo mount -a; sudo docker restart rest-server` — usually enough. `mount -a` re-attempts all fstab entries and bypasses the "failed" state that `mnt-backup.mount` gets stuck in (fstab uses bare `defaults` without auto-retry). If rest-server still errors with `/data/.htpasswd: permission denied`, you've got a **ghost file on the local mount point** — see incident memory for the clean-up procedure (stop rest-server, unmount NFS, rm the ghost, remount, restart).
+   - PBS-ANA: `umount -lf /mnt/pbs-datastore; mount /mnt/pbs-datastore; systemctl restart proxmox-backup proxmox-backup-proxy`
+3. Re-run failed PBS vzdump job for affected VMs (Datacenter → Backup → Run Now)
+4. Restic clients recover automatically on next 01:00 schedule
+
+**Prevention:**
+- Re-schedule ana-nas's own vzdump OUTSIDE the 03:00 fleet window (concurrent NFS load during fleet backup + self-backup is the suspected 2026-04-23 crash root cause)
+- Consider moving pfi-postgres PGDATA to local VM disk (reduces SPOF)
+
+---
+
+### nh3-nas (Synology RS2418+, 10.100.50.50)
+
+**Blast radius:**
+- nh3-docker, nh3-dev restic clients — writes to `rest-server-nh3` fail
+- PBS-NH3 datastore — mirror sync can't write new chunks
+- ana-nas → nh3-nas rsync (04:00 daily) — fails
+
+**Recovery:**
+1. Power-cycle via DSM web UI or physical button if unresponsive
+2. DSM boots 3-5 minutes; NFS exports auto-start
+3. Verify PBS-NH3 datastore recovered:
+   ```
+   ssh pbs-nh3 'ls /mnt/pbs-datastore/.chunks | head; systemctl restart proxmox-backup proxmox-backup-proxy'
+   ```
+4. ana-nas-side rsync retries at next 04:00; no manual action unless you want to trigger now
+
+**Notable:** nh3-nas outage is LESS severe than ana-nas because:
+- No production DB depends on it
+- It's a mirror/DR tier, not primary
+- Ana-side backups keep running independently
+
+---
+
+## Tier 1 — Hypervisors (regional blast radius)
+
+### pfi-pve (Proxmox VE, 10.250.250.31)
+
+**Blast radius — everything hosted on it:**
+- ana-docker (VM), ana-nas (CT 109), pfi-postgres (VM 105), pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm, ana-wg, PBS-ANA (VM), + others
+
+Essentially **all Anaheim primary services** go offline. Because ana-nas lives here too, Tier-0 cascade applies simultaneously.
+
+**Recovery:**
+1. Is the Proxmox host itself up? iDRAC/IPMI console for hardware diagnosis.
+2. If the host is up but VMs/CTs aren't starting: `systemctl status pve-cluster qemu-server pve-container` on the host.
+3. Start VMs/CTs in dependency order:
+   - **First**: CT 109 (ana-nas) — everything else on the host that uses NFS depends on it
+   - **Second**: PBS-ANA VM (resume backup target)
+   - **Third**: VM 105 (pfi-postgres) — many apps wait on this
+   - **Fourth**: ana-docker, then other VMs
+4. After all VMs/CTs up, re-apply the Tier-0 ana-nas recovery steps for any consumers with stale NFS mounts.
+
+**Prevention:** PBS-ANA → PBS-NH3 nightly sync means VM image backups are recoverable at NH3 if pfi-pve is unrecoverable; can restore key VMs to nh3-pve and cut over (cold DR).
+
+---
+
+### nh3-pve (10.100.250.60)
+
+**Blast radius:**
+- nh3-docker (VM) — NH3 site's Docker stacks
+- PBS-NH3 (VM) — DR mirror target
+- Other NH3 VMs
+
+**Recovery:** same pattern as pfi-pve. PBS-NH3 being down breaks DR replication but doesn't affect primary backups (ANA side is authoritative).
+
+---
+
+### esh-pve + esh-pve-nas (10.0.250.35 + 10.0.50.55)
+
+**Blast radius:**
+- All ESH home-lab workloads: esh-docker-vm, vm-esh-nas, media services, home automation stacks
+
+**Recovery:** restore VMs from PBS-ANA (cross-site pull). Not production-critical; no SLA.
+
+---
+
+### sfsrv-ana (Dell R630, 10.250.250.115, iDRAC 10.250.250.110)
+
+**Blast radius:** SureFire tenant workloads.
+- PBS-ANA namespace `sfsrv-pve` has daily snapshots
+- Coordinate with tenant before any recovery action per hosting agreement
+
+**Recovery:** restore from PBS-ANA sfsrv-pve namespace. Tenant may have app-level recovery procedures that take precedence.
+
+---
+
+## Tier 2 — Docker hosts (per-stack blast radius)
+
+### ana-docker (VM 10.250.50.70)
+
+**Blast radius — services hosted on it:**
+- rest-server-ana (fleet restic target; depends on ana-nas NFS)
+- Backrest UI, Beszel hub, Dozzle hub, Traefik (ANA), CrowdSec, Gitea, Vaultwarden, Seafile, Synapse, OpenWebUI, SearXNG, IT Tools, others
+
+**Recovery:**
+1. VM restart on pfi-pve: `qm stop <vmid>; qm start <vmid>`
+2. After boot, stacks auto-start via `docker compose up -d` (compose files at `/opt/docker/compose/*/`)
+3. Check each stack with `docker ps` / dockge
+4. If `/mnt/backup` (NFS) is stale after pfi-pve reboot, force remount
+
+### nh3-docker (VM 10.100.50.40)
+
+**Blast radius:**
+- rest-server-nh3 (if migrated there — currently on nh3-nas Synology directly)
+- NH3-only stacks
+
+**Recovery:** VM restart on nh3-pve. Cross-site Beszel/Dozzle agents on ana-docker keep reporting independently.
+
+### esh-docker-vm + vm-esh-nas
+
+**Blast radius:** ESH home-lab Docker workloads. Paperless, HA, CWA, pgadmin (esh-docker-vm); dockge, filezilla, agents (vm-esh-nas).
+
+**Recovery:** VM restart on respective ESH hypervisors. Not production-critical.
+
+---
+
+## Tier 3 — Workload VMs/LXCs
+
+### pfi-postgres (VM 105 on pfi-pve)
+
+**Blast radius:** vaultwarden, gitea, paperless-ng databases.
+
+**Recovery:** usually it's actually an ana-nas issue (see Tier 0). If postgres itself is the problem:
+1. Check logs: `sudo tail -200 /var/lib/postgresql/*/main/log/postgresql-*.log`
+2. If clean: `sudo systemctl restart postgresql`
+3. If WAL corruption: restore from PBS vzdump (most recent snapshot)
+
+### ana-wg (CT 113 on pfi-pve) — WireGuard VPN
+
+**Blast radius:** remote-access VPN down; remote admin sessions drop but on-prem ops continue.
+
+**Recovery:** `pct start 113` on pfi-pve. Restart wg-quick service if needed.
+
+### pfi-pteradactyl (VM 107 on pfi-pve) — game panel
+
+**Blast radius:** hosted game servers offline.
+
+**Recovery:** VM restart. Not operationally critical.
+
+### pfi-tacticalrmm (VM 111 on pfi-pve) — RMM tooling
+
+**Blast radius:** RMM agent dashboards; agents continue running on endpoints but can't phone home.
+
+**Recovery:** VM restart; TacticalRMM services auto-start.
+
+### ana-filebot (CT 112 on pfi-pve) — file automation
+
+**Blast radius:** scheduled file operations (rename, organize); low-criticality.
+
+**Recovery:** `pct start 112`.
+
+### pfi-ana-webhost (VM on pfi-pve) — web workload
+
+**Blast radius:** hosted website(s) offline.
+
+**Recovery:** VM restart; web server auto-start.
+
+---
+
+## Tier 4 — Specialty workloads
+
+### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)
+
+**Blast radius:** AI inference services (llama-swap, vllm-qwen3). Consumer-facing chat/embedding endpoints fail.
+
+**Recovery:**
+1. Check OS via SSH. If unresponsive, BMC console at <https://10.250.250.50>.
+2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`).
+3. Docker stacks auto-start via compose `restart: unless-stopped`.
+
+**Notable:** no Proxmox hypervisor involved — recovery is pure bare-metal. File-level restic runs from inside the host. No vzdump coverage (it's not a VM). Backups via restic to rest-server-ana.
+
+### SureFire hosts (sfsrv-ana, sf-ana-container, sf-r630)
+
+**Blast radius:** tenant workloads (scoped to SureFire client).
+
+**Recovery:** coordinate with tenant. PFI responsibility is hardware + OS layer; application recovery may need tenant input. See `servers/sfsrv-ana/README.md` for hosting agreement scope.
+
+---
+
+## Tier 5 — Networking
+
+### FortiGate 60F (ESH gateway, 10.0.250.1)
+
+**Blast radius:** ESH site WAN + inter-site VPN to ANA/NH3.
+
+**Recovery:** physical console access; restore config from FortiManager if needed.
+
+### UniFi controllers (ESH 10.0.0.1, PFI 10.100.0.1)
+
+**Blast radius:** UniFi AP/switch management (existing config persists on devices; only changes need the controller).
+
+**Recovery:** device power cycle; controller reboot. Not critical for ongoing operations.
+
+---
+
+## Cross-cutting: what to check FIRST for ambiguous outages
+
+When symptoms are vague ("lots of things are down"), run this triage in order:
+
+```bash
+# 1. Is the Anaheim NAS alive?
+ping -c 2 10.250.50.50
+
+# 2. Are the hypervisors alive?
+for h in 10.250.250.31 10.100.250.60 10.0.250.35 10.0.50.55 10.250.250.115; do
+  ping -c 1 -W 2 $h >/dev/null && echo "$h OK" || echo "$h DOWN"
+done
+
+# 3. WAN reachability between sites?
+#    (from nh3-dev to ana-side IP; from ana-side to nh3-side IP)
+ping -c 2 10.250.50.70  # ana-docker from NH3
+```
+
+The first `DOWN` in the hypervisor list narrows blast radius to that site / that hypervisor's guests.
+
+---
+
+## What this runbook does NOT cover
+
+- Application-level restores from PBS / restic snapshots — that's per-service and lives in respective stack READMEs
+- Full site-failover (e.g., "move PFI fleet to NH3 entirely") — requires Phase-9-style DR plan, not built
+- Data-level integrity recovery (DB corruption, ZFS pool damage) — beyond this runbook's scope
+
+**Maintenance note:** update this file when:
+- New host gets registered (add to applicable tier)
+- New SPOF discovered (add blast-radius note)
+- A recovery procedure changes in practice
diff --git a/docs/runbooks/pbs-deployment.md b/docs/runbooks/pbs-deployment.md
index 9d0646b..a9a33bd 100644
--- a/docs/runbooks/pbs-deployment.md
+++ b/docs/runbooks/pbs-deployment.md
@@ -197,20 +197,77 @@ both have VM 100 for instance). Without namespaces, their backups
 collide under the same `/vm/100/` path in the datastore. Create a
 namespace per hypervisor up front:
 
+`proxmox-backup-manager` doesn't manage namespaces — they're created
+through the API/web UI or the `proxmox-backup-client` tool.
+
+**Easiest: web UI.** Datastore → `backups` → Content → top of pane
+there's a namespace selector with an **Add NS** button. Add one per
+hypervisor.
+
+**Scripted via `proxmox-backup-client`** (run on PBS-ANA):
+
 ```bash
-for h in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
-  proxmox-backup-manager namespace create backups $h
+export PBS_REPOSITORY='root@pam@localhost:backups'
+export PBS_PASSWORD='<root-pam-password>'
+for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
+  proxmox-backup-client namespace create "$ns"
 done
-proxmox-backup-manager namespace list backups
+proxmox-backup-client namespace list
 ```
 
-(If the CLI errors — subcommand names shift between PBS versions —
-use the web UI: Datastore → backups → Content → **Add NS**. Works
-reliably regardless of version.)
+**Scripted via API** (when ssh access is more convenient than shell
+on PBS-ANA):
+
+```bash
+TOKEN='<fleet-vzdump-secret>'
+for ns in pfi-pve nh3-pve esh-pve esh-pve-nas sfsrv-ana; do
+  curl -sk \
+    -H "Authorization: PBSAPIToken=root@pam!fleet-vzdump:$TOKEN" \
+    -X POST \
+    https://10.250.50.90:8007/api2/json/admin/datastore/backups/namespace \
+    -d "{\"ns\":\"$ns\"}"
+done
+```
 
 Each PVE client later (Phase 2.1, 3.1, 4) sets its Namespace field to
 its own hostname when configuring the PBS storage.
 
+### 1.4c. Create a verify job
+
+New snapshots land unverified — PBS treats "backup completed" and
+"backup verified" as separate states. A verify job hash-checks chunk
+data (not just the manifest), flips snapshots to verified, and catches
+bitrot on aging data.
+
+Web UI: **Datastore → backups → Verify Jobs → Add**:
+
+| Field | Value |
+|---|---|
+| Schedule | `sat 23:00` |
+| Ignore verified snapshots | ✓ |
+| Re-verify after (days) | `30` |
+| Max depth | blank (unlimited) |
+| Namespace | blank (root — recurses into all namespaces) |
+| Comment | `fleet verify — new + 30d re-check` |
+
+Rationale for these defaults:
+
+- **`sat 23:00`** — sits between the daily 03:00 backup window and the
+  Sunday 06:00 GC run, so verify never fights GC for I/O, and every
+  week's new backups get verified before pruning decisions happen.
+- **`ignore-verified: true`** + **`outdated-after: 30d`** — efficient
+  steady state. First run after a backup night does the new snapshots
+  only; a 30-day rolling re-verify catches silent chunk corruption.
+- **unlimited depth, root namespace** — one job covers all 5
+  hypervisor namespaces. Split into per-namespace jobs only if you
+  want per-hypervisor visibility into verify failures (not necessary
+  for a fleet this size).
+
+Verify runs are I/O-heavy on the datastore — on the NFS-backed
+PBS-ANA, expect a full-datastore verify to take hours once the
+datastore grows. The `ignore-verified` flag keeps incremental verify
+cheap; only the 30-day-aged portion is re-read each run.
+
 ### 1.5. Create an API token for hypervisors to use
 
 PBS web UI: **Configuration → Access Control → API Token → Add**:
@@ -405,32 +462,152 @@ Same pattern as Phase 1.1:
 
 Same as Phase 1.2.
 
-### 5.3. Local datastore
+### 5.3. Datastore backing — Synology NFS (chosen 2026-04-22)
 
-PBS-NH3 uses local storage rather than NFS (keeps the two PBSes on
-different failure domains). Two options:
+PBS-NH3 mounts a Synology NFS share rather than using a local virtual
+disk. Simpler storage admin; tradeoff is that PBS-NH3 now shares its
+failure domain with the other NH3 backup paths (nh3-docker restic,
+nh3-dev restic, Backrest repos). Acceptable for DR purposes because
+PBS-ANA remains primary.
 
-**Option A:** attach a second virtual disk to the VM (e.g. 2 TB on
-nh3-pve's `local-lvm` or whatever storage is available), format ext4,
-mount at `/mnt/pbs-datastore`.
+**Synology-side export setup (DSM):**
 
-**Option B:** mount Synology Btrfs share via NFS or CIFS. Simpler
-storage admin but couples PBS-NH3 to the Synology's availability.
+1. Create a dedicated share, e.g. `pbs-nh3`, on the target volume.
+2. **Control Panel → Shared Folder → [share] → Edit → NFS Permissions → Add/Edit:**
+   | Field | Value |
+   |---|---|
+   | Hostname/IP | PBS-NH3 VM IP |
+   | Privilege | Read/Write |
+   | Squash | **No mapping** (Synology's label for `no_root_squash`) |
+   | Security | sys |
+   | Enable asynchronous | on |
+   | Allow connections from non-privileged ports | on |
 
-Recommended: **Option A**. Keeps it self-contained.
+3. Verify no **Advanced Permissions** ACL denies `root` write — those
+   override NFS perms and cause silent write failures.
+
+**Client-side mount (on PBS-NH3 VM):**
+
+Use **NFSv3**, not NFSv4 — see the gotcha below. Synology's "Advanced
+Permissions" layer an NFSv4 ACL on top of POSIX mode that is invisible
+to `ls` but denies writes to unprivileged users (including the
+`backup` uid-34 that PBS runs as), even when the directory mode is
+777. Root bypasses this via `no_root_squash`, which is why a root
+`touch` succeeds but the datastore init fails.
 
 ```bash
-# Inside the PBS-NH3 VM, after attaching a disk /dev/sdb:
-mkfs.ext4 /dev/sdb
 mkdir -p /mnt/pbs-datastore
-echo '/dev/sdb  /mnt/pbs-datastore  ext4  defaults  0  2' >> /etc/fstab
+
+cat >> /etc/fstab <<'EOF'
+10.100.50.50:/volume1/pbs-nh3  /mnt/pbs-datastore  nfs  defaults,_netdev,bg,hard,timeo=600,retrans=2,vers=3  0  0
+EOF
+
 mount -a
+df -h /mnt/pbs-datastore
+
+# Two-layer smoke test: root AND the backup user that PBS runs as.
+# The backup-user check is the one that actually matters — if root
+# works but backup doesn't, you're hitting the ACL override.
+touch /mnt/pbs-datastore/root-write-test && rm /mnt/pbs-datastore/root-write-test
+sudo -u backup touch /mnt/pbs-datastore/backup-user-test && rm /mnt/pbs-datastore/backup-user-test
 ```
 
+Ensure NFSv3 is reachable on the Synology side: **Control Panel →
+File Services → NFS → Advanced** — confirm "Minimum NFS Protocol" is
+3 (not 4.0+). Default is usually 3; only an issue if someone
+hardened it previously.
+
+**Fallback if NFSv3 still fails the backup-user check:** the deny is
+a Synology-local syno_acl (`ls -la` on the Synology shows POSIX mode
+`d---------+` with a `+` for extended ACL). DSM "Enable Advanced
+Permissions" unchecked does NOT reset this once the archive flag
+`has_ACL,is_support_ACL` is set on the share.
+
+Diagnose from the Synology shell:
+
+```
+sudo synoacltool -get /volume1/<share>
+sudo ls -la /volume1/<share>/
+```
+
+A `+` after the permission string + a `group:administrators:allow:...`
+ACL entry + POSIX `d---------` is the smoking gun: only members of
+the `administrators` group have access, which is why root (with
+`no_root_squash`) writes but uid-34 (backup) doesn't.
+
+**Fix (confirmed working 2026-04-22):** keep **Squash: `No mapping`**
+(i.e. `no_root_squash`) AND flatten the share to Linux/POSIX mode
+with `chmod 777`. This drops the syno_acl entirely — verifiable by
+`synoacltool -get` returning "It's Linux mode" and `ls -la`
+showing `drwxrwxrwx` with NO trailing `+`.
+
+On the Synology shell:
+
+```bash
+sudo chmod 777 /volume1/<share>
+
+# Verify pure POSIX, no ACL
+sudo synoacltool -get /volume1/<share>   # should say "It's Linux mode"
+ls -la /volume1/<share>/                 # should show drwxrwxrwx (no '+')
+```
+
+Pure POSIX 777 is a cleaner long-term config than the ACL-grant
+approach — fewer permission-translation layers between NFSv3 and
+Btrfs, and no chance of ACL inheritance surprises on PBS-created
+subdirectories.
+
+**Why `all_squash` alone doesn't work:** it lets backup-user writes
+through (because the ACL-granted admin gets the mapped uid), but
+breaks PBS's `chown()` during init. Squashed-admin doesn't have
+CAP_CHOWN on the Synology side → EPERM. Only real root (via
+`no_root_squash`) can chown to uid 34.
+
+**Why squash alone fails:** even with `all_squash + anonuid=1024`
+letting backup-user writes succeed (because admin is in
+`administrators` ACL), PBS's datastore-init calls `chown` on
+newly-created paths. Squashed-admin doesn't have CAP_CHOWN on the
+Synology side → EPERM. Only a real root (via `no_root_squash`) can
+chown to uid 34.
+
+**Smoke-test all three paths after the fix:**
+
+```bash
+sudo touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo ROOT_OK
+sudo -u backup touch /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo BACKUP_OK
+touch /mnt/pbs-datastore/t && chown 34:34 /mnt/pbs-datastore/t && rm /mnt/pbs-datastore/t && echo CHOWN_OK
+```
+
+All three must pass before PBS will init cleanly.
+
+**Historical note:** both PBS instances ended up on NFSv3 for
+unrelated reasons — the ANA side because of ZFS case-insensitivity,
+the NH3 side because of Synology ACL override. NFSv3 is the safer
+default for PBS-on-NFS regardless of backend.
+
 ### 5.4. Create datastore
 
 Web UI: Datastore → Add, name `backups-mirror`, path `/mnt/pbs-datastore`.
 
+### 5.5. Create a verify job on the mirror
+
+Same pattern as Phase 1.4c on PBS-ANA — verification doesn't replicate
+across PBS instances, so the mirror needs its own job to catch bitrot
+on the local datastore disk.
+
+Web UI: **Datastore → backups-mirror → Verify Jobs → Add**:
+
+| Field | Value |
+|---|---|
+| Schedule | `sun 12:00` |
+| Ignore verified snapshots | ✓ |
+| Re-verify after (days) | `30` |
+| Max depth | blank (unlimited) |
+| Namespace | blank |
+| Comment | `mirror verify — catches DR-side bitrot` |
+
+Schedule sits after the 06:00 sync job finishes, so newly-synced
+snapshots get verified same day.
+
 ### Phase 5 done-state
 
 - PBS-NH3 reachable, datastore ready