vh
574c72daa5
Bundles the post-2026-04-21 work that built out the two-layer backup architecture (PBS for VM images + restic for file/DB), plus the cross- site mirror and the disaster-recovery runbook. - configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete *_offen_backup_data exclude (offen sidecars retired fleet-wide 2026-04-23; restic now covers the equivalent scope directly). - configs/restic/esh-vm-db/: new profile for the dedicated DB VM (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks. - configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven) and nh3-nas → ana-nas (05:00 daily, runs as root because DSM rest-server-nh3 writes mode-400 files only root can read). - docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook, refined during the 2026-04-22 deployment with per-hypervisor namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the Synology syno_acl flatten step. - docs/runbooks/disaster-recovery.md: blast-radius runbook ordered Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty); references incident memory + recovery-step playbooks per consumer.
ana-nas → nh3-nas restic mirror
Nightly rsync of the Ana-side restic repo to the NH3 Synology, giving file-level backups cross-site redundancy independent of the PBS layer.
What + where
| Source | ana-nas:/mnt/backup/restic/repo/ana/ (ZFS NASPool/backupStore, written by rest-server-ana on ana-docker via NFS). |
| Target | nh3-nas:/volume1/Backup/restic-ana-mirror/ (Btrfs /volume1, 16 TB free at setup time). |
| Runs on | ana-nas (Debian 12, VMID 100 on pfi-pve). |
| Runs as | lkraven (uid 1000) — owns the source data natively; no sudo needed. |
| Auth | Dedicated ed25519 keypair ~lkraven/.ssh/id_mirror_nh3 → syncuser@nh3-nas. No passphrase (for unattended systemd runs). |
| Schedule | Daily at 04:00, Persistent=true, 300s randomized delay. |
Scheduling rationale
Window sits between:
- 01:00–03:00 — ana-side restic clients run their nightly backups against rest-server-ana. We wait until those are clearly done before reading the repo.
- 06:00 — PBS-ANA → PBS-NH3 sync job pulls fresh snapshots across the WAN. Running the restic mirror earlier avoids competing for WAN bandwidth with the (much larger) PBS sync.
Files
| File | Install path on ana-nas |
|---|---|
restic-mirror-to-nh3.service |
/etc/systemd/system/restic-mirror-to-nh3.service |
restic-mirror-to-nh3.timer |
/etc/systemd/system/restic-mirror-to-nh3.timer |
These are the canonical copies; the on-host copies mirror them.
Deploy / redeploy
cd configs/rsync/ana-nas-to-nh3
scp restic-mirror-to-nh3.{service,timer} ana-nas:/tmp/
ssh -t ana-nas 'sudo install -m 644 /tmp/restic-mirror-to-nh3.service /etc/systemd/system/ && \
sudo install -m 644 /tmp/restic-mirror-to-nh3.timer /etc/systemd/system/ && \
sudo systemctl daemon-reload && \
sudo systemctl enable --now restic-mirror-to-nh3.timer && \
sudo systemctl list-timers restic-mirror-to-nh3.timer'
Operate
# Trigger a run manually (e.g. to smoke-test changes)
ssh -t ana-nas 'sudo systemctl start restic-mirror-to-nh3.service'
# Follow a running transfer
ssh ana-nas 'journalctl -u restic-mirror-to-nh3.service -f'
# Last run outcome
ssh ana-nas 'systemctl status restic-mirror-to-nh3.service --no-pager'
# When the timer fires next
ssh ana-nas 'systemctl list-timers restic-mirror-to-nh3.timer'
Restore flow (if ana-nas loses the repo)
The mirror is just files — point a restic client directly at the nh3-nas copy. Options:
Option A — pull repo back to ana-nas and use rest-server-ana as before. rsync in reverse:
ssh -t ana-nas 'sudo rsync -a \
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3" \
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/ \
/mnt/backup/restic/repo/ana/'
Option B — restic against the mirror directly. Expose
/volume1/Backup/restic-ana-mirror/ via a temporary rest-server or
NFS share, point the client at it for an emergency restore. Slow (WAN
hop) but no dataset copy required.
Target is --append-only: false on the mirror side — not a problem
for restore, but means a compromised mirror side could be tampered
with. Balanced against the operational cost of managing an
append-only mirror, we accept this for a mirror-of-a-mirror.
What this does NOT cover
- PBS snapshots — those replicate via PBS-ANA → PBS-NH3 (a
separate pipeline; see
docs/runbooks/pbs-deployment.md). - rest-server-nh3's own data — nh3-docker + nh3-dev backups that
land on
/volume1/Backup/restic/<user>/. Those are already at the NH3 Synology; ANA-side doesn't mirror them currently. Symmetric mirror back to ana-nas is a future-work item.