backup pipeline: configs, runbooks, NH3 Synology rest-server, cross-site rsync
Bundles the post-2026-04-21 work that built out the two-layer backup architecture (PBS for VM images + restic for file/DB), plus the cross- site mirror and the disaster-recovery runbook. - configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete *_offen_backup_data exclude (offen sidecars retired fleet-wide 2026-04-23; restic now covers the equivalent scope directly). - configs/restic/esh-vm-db/: new profile for the dedicated DB VM (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks. - configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven) and nh3-nas → ana-nas (05:00 daily, runs as root because DSM rest-server-nh3 writes mode-400 files only root can read). - docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook, refined during the 2026-04-22 deployment with per-hypervisor namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the Synology syno_acl flatten step. - docs/runbooks/disaster-recovery.md: blast-radius runbook ordered Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty); references incident memory + recovery-step playbooks per consumer.
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
# ana-nas → nh3-nas restic mirror
|
||||
|
||||
Nightly rsync of the Ana-side restic repo to the NH3 Synology, giving
|
||||
file-level backups cross-site redundancy independent of the PBS layer.
|
||||
|
||||
## What + where
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Source** | `ana-nas:/mnt/backup/restic/repo/ana/` (ZFS `NASPool/backupStore`, written by rest-server-ana on ana-docker via NFS). |
|
||||
| **Target** | `nh3-nas:/volume1/Backup/restic-ana-mirror/` (Btrfs `/volume1`, 16 TB free at setup time). |
|
||||
| **Runs on** | `ana-nas` (Debian 12, VMID 100 on pfi-pve). |
|
||||
| **Runs as** | `lkraven` (uid 1000) — owns the source data natively; no sudo needed. |
|
||||
| **Auth** | Dedicated ed25519 keypair `~lkraven/.ssh/id_mirror_nh3` → `syncuser@nh3-nas`. No passphrase (for unattended systemd runs). |
|
||||
| **Schedule** | Daily at 04:00, `Persistent=true`, 300s randomized delay. |
|
||||
|
||||
## Scheduling rationale
|
||||
|
||||
Window sits between:
|
||||
- **01:00–03:00** — ana-side restic clients run their nightly backups
|
||||
against rest-server-ana. We wait until those are clearly done before
|
||||
reading the repo.
|
||||
- **06:00** — PBS-ANA → PBS-NH3 sync job pulls fresh snapshots across
|
||||
the WAN. Running the restic mirror earlier avoids competing for WAN
|
||||
bandwidth with the (much larger) PBS sync.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Install path on ana-nas |
|
||||
|---|---|
|
||||
| `restic-mirror-to-nh3.service` | `/etc/systemd/system/restic-mirror-to-nh3.service` |
|
||||
| `restic-mirror-to-nh3.timer` | `/etc/systemd/system/restic-mirror-to-nh3.timer` |
|
||||
|
||||
These are the canonical copies; the on-host copies mirror them.
|
||||
|
||||
## Deploy / redeploy
|
||||
|
||||
```bash
|
||||
cd configs/rsync/ana-nas-to-nh3
|
||||
scp restic-mirror-to-nh3.{service,timer} ana-nas:/tmp/
|
||||
ssh -t ana-nas 'sudo install -m 644 /tmp/restic-mirror-to-nh3.service /etc/systemd/system/ && \
|
||||
sudo install -m 644 /tmp/restic-mirror-to-nh3.timer /etc/systemd/system/ && \
|
||||
sudo systemctl daemon-reload && \
|
||||
sudo systemctl enable --now restic-mirror-to-nh3.timer && \
|
||||
sudo systemctl list-timers restic-mirror-to-nh3.timer'
|
||||
```
|
||||
|
||||
## Operate
|
||||
|
||||
```bash
|
||||
# Trigger a run manually (e.g. to smoke-test changes)
|
||||
ssh -t ana-nas 'sudo systemctl start restic-mirror-to-nh3.service'
|
||||
|
||||
# Follow a running transfer
|
||||
ssh ana-nas 'journalctl -u restic-mirror-to-nh3.service -f'
|
||||
|
||||
# Last run outcome
|
||||
ssh ana-nas 'systemctl status restic-mirror-to-nh3.service --no-pager'
|
||||
|
||||
# When the timer fires next
|
||||
ssh ana-nas 'systemctl list-timers restic-mirror-to-nh3.timer'
|
||||
```
|
||||
|
||||
## Restore flow (if ana-nas loses the repo)
|
||||
|
||||
The mirror is just files — point a restic client directly at the
|
||||
nh3-nas copy. Options:
|
||||
|
||||
**Option A — pull repo back to ana-nas and use rest-server-ana as
|
||||
before.** rsync in reverse:
|
||||
|
||||
```bash
|
||||
ssh -t ana-nas 'sudo rsync -a \
|
||||
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3" \
|
||||
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/ \
|
||||
/mnt/backup/restic/repo/ana/'
|
||||
```
|
||||
|
||||
**Option B — restic against the mirror directly.** Expose
|
||||
`/volume1/Backup/restic-ana-mirror/` via a temporary rest-server or
|
||||
NFS share, point the client at it for an emergency restore. Slow (WAN
|
||||
hop) but no dataset copy required.
|
||||
|
||||
Target is `--append-only: false` on the mirror side — not a problem
|
||||
for restore, but means a compromised mirror side *could* be tampered
|
||||
with. Balanced against the operational cost of managing an
|
||||
append-only mirror, we accept this for a mirror-of-a-mirror.
|
||||
|
||||
## What this does NOT cover
|
||||
|
||||
- **PBS snapshots** — those replicate via PBS-ANA → PBS-NH3 (a
|
||||
separate pipeline; see `docs/runbooks/pbs-deployment.md`).
|
||||
- **rest-server-nh3's own data** — nh3-docker + nh3-dev backups that
|
||||
land on `/volume1/Backup/restic/<user>/`. Those are already at the
|
||||
NH3 Synology; ANA-side doesn't mirror them currently. Symmetric
|
||||
mirror back to ana-nas is a future-work item.
|
||||
@@ -0,0 +1,46 @@
|
||||
[Unit]
|
||||
Description=Mirror rest-server-ana restic repo to NH3 Synology
|
||||
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
ConditionPathIsDirectory=/mnt/backup/restic/repo/ana
|
||||
StartLimitBurst=3
|
||||
StartLimitIntervalSec=1h
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
|
||||
# Dedicated key for this job, ed25519, ana-nas → nh3-nas:syncuser.
|
||||
# --append-only on rest-server-ana means source files are never
|
||||
# rewritten or deleted by clients; --delete here mirrors any explicit
|
||||
# prune operations (done out-of-band during the quarterly ceremony).
|
||||
# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
|
||||
ExecStart=/usr/bin/rsync \
|
||||
--archive \
|
||||
--delete \
|
||||
--partial \
|
||||
--info=stats2 \
|
||||
--timeout=300 \
|
||||
--exclude=tmp/ \
|
||||
--exclude=.lock \
|
||||
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
|
||||
/mnt/backup/restic/repo/ana/ \
|
||||
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/
|
||||
|
||||
# ServerAlive{Interval,CountMax}=30/3 + rsync --timeout bound below
|
||||
# force detection of dead WAN within ~90s; without this, a silent TCP
|
||||
# drop holds the service in "activating" for hours on kernel retransmit
|
||||
# backoff. TimeoutStartSec=6h caps worst-case initial sync; after
|
||||
# that systemd kills the service even if rsync somehow wedges.
|
||||
TimeoutStartSec=6h
|
||||
|
||||
# Tight retry window — a transient WAN blip shouldn't wedge the timer
|
||||
# schedule, but spamming retries on a real outage is noise.
|
||||
# StartLimitBurst/Interval live in [Unit] (systemd v230+); the burst
|
||||
# cap prevents runaway retries even with the short RestartSec.
|
||||
Restart=on-failure
|
||||
RestartSec=30s
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=Daily mirror of rest-server-ana restic repo to NH3
|
||||
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
|
||||
|
||||
[Timer]
|
||||
# 04:00 daily — sits between ana-side restic clients (01:00 finish
|
||||
# window) and the PBS-ANA → PBS-NH3 sync at 06:00, so WAN contention
|
||||
# is minimized. Persistent=true catches missed runs if ana-nas was
|
||||
# offline; RandomizedDelaySec smears load if this ever scales to
|
||||
# multiple mirror jobs on the same host.
|
||||
OnCalendar=*-*-* 04:00:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=300
|
||||
Unit=restic-mirror-to-nh3.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user