backup pipeline: configs, runbooks, NH3 Synology rest-server, cross-site rsync
Bundles the post-2026-04-21 work that built out the two-layer backup architecture (PBS for VM images + restic for file/DB), plus the cross- site mirror and the disaster-recovery runbook. - configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete *_offen_backup_data exclude (offen sidecars retired fleet-wide 2026-04-23; restic now covers the equivalent scope directly). - configs/restic/esh-vm-db/: new profile for the dedicated DB VM (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks. - configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven) and nh3-nas → ana-nas (05:00 daily, runs as root because DSM rest-server-nh3 writes mode-400 files only root can read). - docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook, refined during the 2026-04-22 deployment with per-hypervisor namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the Synology syno_acl flatten step. - docs/runbooks/disaster-recovery.md: blast-radius runbook ordered Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty); references incident memory + recovery-step playbooks per consumer.
This commit is contained in:
@@ -52,8 +52,8 @@ default:
|
||||
# Docker internals
|
||||
- /var/lib/docker/volumes/backingFsBlockDev
|
||||
- /var/lib/docker/volumes/metadata.db
|
||||
# Offen sidecar's buffer / tmp (if any left behind)
|
||||
- /var/lib/docker/volumes/*_offen_backup_data
|
||||
# Offen `*_offen_backup_data` exclude removed 2026-04-23 — offen
|
||||
# sidecars retired fleet-wide; no more offen-scratch volumes exist.
|
||||
# Ephemeral / regenerable
|
||||
- /opt/docker/compose/*/logs
|
||||
- "**/*.log"
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
# restic / esh-vm-db
|
||||
|
||||
**Two-database host** at the ESH site (PostgreSQL 15 + MongoDB). Covered
|
||||
at the VM-image layer by PBS-ANA via esh-pve (or whichever ESH
|
||||
hypervisor owns this VM — confirm on next inventory pass). This restic
|
||||
profile adds DB-level granularity via pre-backup dumps.
|
||||
|
||||
## What's backed up
|
||||
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `/etc` | Host config — systemd, ssh, chrony, apt, pg_hba.conf, mongod.conf |
|
||||
| `/root` | Root's ad-hoc scripts, shell history, ssh keys |
|
||||
| `/home` | User homes (lkraven + any DB-admin locals) |
|
||||
| `/var/lib/restic/stage` | **pg_dumpall.sql.gz** + **mongodump/** produced by pre-backup.sh |
|
||||
|
||||
## What's **not** backed up (by design)
|
||||
|
||||
- **`/var/lib/postgresql`** — raw PGDATA. Live-capture risk;
|
||||
`pg_dumpall` in pre-backup covers it consistently.
|
||||
- **`/var/lib/mongodb`** — raw mongo dbPath. Same reasoning;
|
||||
`mongodump` covers it.
|
||||
- NFS mount `/mnt/backup` (from esh-nas — not ours to mirror).
|
||||
|
||||
## Pre-backup hook
|
||||
|
||||
`pre-backup.sh` runs as root before restic. It:
|
||||
|
||||
1. Checks `pg_isready` on :5432 — if OK, runs `pg_dumpall` piped
|
||||
through gzip to `$STAGE/pg_dumpall.sql.gz`
|
||||
2. Checks mongo ping via `mongosh` — if OK, runs `mongodump` into
|
||||
`$STAGE/mongodump/`
|
||||
|
||||
Both dumps are atomic (write to `.tmp`, then rename). If either DB is
|
||||
unreachable, the script logs a WARN and continues — a failed DB dump
|
||||
doesn't abort the whole restic run, and restic falls back to whatever
|
||||
stage content is left over from the prior successful dump.
|
||||
|
||||
## Deploy (one-time)
|
||||
|
||||
### 1. Create rest-server-ana htpasswd entry
|
||||
|
||||
**Do NOT use `sudo` for .htpasswd writes on ana-docker.** The file is
|
||||
NFS-mounted from ana-nas and owned by uid 1000 (the rest-server user,
|
||||
which equals lkraven). Sudo-root on the client gets squashed to
|
||||
nobody on the NFS server and can't read/write the file. lkraven
|
||||
writes it natively, using the docker group for the bcrypt helper.
|
||||
|
||||
```bash
|
||||
# Pick password in password manager first
|
||||
HTPW='<new-pw-saved-to-pw-manager>'
|
||||
|
||||
ssh -t ana-docker "docker run --rm httpd:2.4-alpine htpasswd -nbB esh-vm-db '$HTPW' | \
|
||||
tee /tmp/htline.txt > /dev/null && \
|
||||
sed -i '/^esh-vm-db:/d' /mnt/backup/restic/repo/ana/.htpasswd && \
|
||||
cat /tmp/htline.txt >> /mnt/backup/restic/repo/ana/.htpasswd && \
|
||||
rm /tmp/htline.txt && \
|
||||
grep ^esh-vm-db: /mnt/backup/restic/repo/ana/.htpasswd && \
|
||||
docker restart rest-server"
|
||||
unset HTPW
|
||||
```
|
||||
|
||||
### 2. Install secrets on esh-vm-db
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'sudo install -d -o root -g root -m 0700 /etc/restic /var/lib/restic /var/lib/restic/stage'
|
||||
|
||||
# restic.env — URL-encode the password if it has special chars
|
||||
ssh -t esh-vm-db "sudo bash -c '
|
||||
read -sp \"htpasswd pw for rest-server-ana: \" HTPW; echo
|
||||
cat > /etc/restic/restic.env <<EOF
|
||||
RESTIC_REPOSITORY=rest:http://esh-vm-db:\$HTPW@10.250.50.70:8000/esh-vm-db/
|
||||
EOF
|
||||
chmod 600 /etc/restic/restic.env
|
||||
'"
|
||||
|
||||
# Repo passphrase (prints once — save to password manager)
|
||||
ssh -t esh-vm-db 'sudo bash -c "
|
||||
openssl rand -base64 48 | tr -d \"\\n\" > /etc/restic/password
|
||||
chmod 600 /etc/restic/password
|
||||
echo === SAVE THIS TO PASSWORD MANAGER NOW ===
|
||||
cat /etc/restic/password
|
||||
echo
|
||||
"'
|
||||
```
|
||||
|
||||
### 3. Initialize the repo
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'sudo bash -c "
|
||||
set -a; . /etc/restic/restic.env; set +a
|
||||
RESTIC_PASSWORD_FILE=/etc/restic/password restic init
|
||||
"'
|
||||
```
|
||||
|
||||
### 4. Install prerequisites (restic, resticprofile, mongosh client)
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'which restic || sudo apt-get install -y restic; \
|
||||
which mongosh || echo "NOTE: mongosh not found; pre-backup mongo ping will fail safely — install via MongoDB APT repo if needed"; \
|
||||
curl -sfL https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh | sudo sh -s -- -b /usr/local/bin; \
|
||||
/usr/local/bin/resticprofile --version'
|
||||
```
|
||||
|
||||
### 5. Deploy profile + hook
|
||||
|
||||
```bash
|
||||
scp configs/restic/esh-vm-db/profiles.yaml esh-vm-db:/tmp/
|
||||
scp configs/restic/esh-vm-db/pre-backup.sh esh-vm-db:/tmp/
|
||||
|
||||
ssh -t esh-vm-db 'sudo install -o root -g root -m 0644 /tmp/profiles.yaml /etc/restic/profiles.yaml && \
|
||||
sudo install -o root -g root -m 0755 /tmp/pre-backup.sh /etc/restic/pre-backup.sh && \
|
||||
rm /tmp/profiles.yaml /tmp/pre-backup.sh'
|
||||
```
|
||||
|
||||
### 6. Schedule + verify
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml schedule --all && \
|
||||
systemctl list-timers "resticprofile*" --no-pager'
|
||||
|
||||
# First manual run
|
||||
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml backup --verbose'
|
||||
```
|
||||
|
||||
Expect first run to land ~50-200 MB (mostly the mongodump directory + pg_dumpall).
|
||||
Cross-check from Backrest UI on ana-docker.
|
||||
|
||||
## Restore
|
||||
|
||||
### Full host config
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /etc'
|
||||
```
|
||||
|
||||
### Just the PG dump
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /var/lib/restic/stage/pg_dumpall.sql.gz'
|
||||
# Then: gunzip + psql < pg_dumpall.sql
|
||||
```
|
||||
|
||||
### Just a mongo DB
|
||||
|
||||
```bash
|
||||
ssh -t esh-vm-db 'sudo resticprofile --config /etc/restic/profiles.yaml restore latest --target /tmp/restore --path /var/lib/restic/stage/mongodump'
|
||||
# Then: mongorestore /tmp/restore/var/lib/restic/stage/mongodump/
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **mongosh must be installed** or the mongo pre-backup step silently
|
||||
skips (logged as WARN). Install from the MongoDB APT repo if not
|
||||
already present — the stock Debian `mongodb-clients` package is
|
||||
out of date and doesn't include `mongosh`.
|
||||
- **Mongo authentication** — if mongod ever gets auth enabled (it's
|
||||
currently open to 0.0.0.0 with no auth, which is its own concern),
|
||||
`mongodump` will need `--username/--password` flags. Reference in
|
||||
pre-backup.sh when that change happens.
|
||||
- **pg_hba.conf** — `pg_dumpall` requires local postgres superuser
|
||||
access. Currently works via `sudo -u postgres` + peer auth on the
|
||||
local socket. If `pg_hba.conf` ever changes peer → md5 for local,
|
||||
the hook needs a `~postgres/.pgpass` entry.
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/bin/bash
|
||||
# pre-backup.sh — esh-vm-db.
|
||||
# Runs as root from resticprofile's `run-before`.
|
||||
#
|
||||
# Produces consistent DB dumps in /var/lib/restic/stage/. Two DBs here:
|
||||
# - Postgres 15 (port 5432, local) — pg_dumpall all databases
|
||||
# - MongoDB (port 27017, local) — mongodump all databases
|
||||
#
|
||||
# Peer-data DBs (paperless-ng on Postgres) are the primary consumers;
|
||||
# raw volume capture isn't in the restic source list so these dumps
|
||||
# are the ONLY way restic sees DB data.
|
||||
#
|
||||
# Errors in individual blocks log a WARN; whole script doesn't abort.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
STAGE=/var/lib/restic/stage
|
||||
install -d -o root -g root -m 0700 "$STAGE"
|
||||
|
||||
log() { printf '%s pre-backup(esh-vm-db): %s\n' "$(date -Is)" "$*"; }
|
||||
warn() { printf '%s pre-backup(esh-vm-db): WARN: %s\n' "$(date -Is)" "$*" >&2; }
|
||||
|
||||
# ---- Postgres ----------------------------------------------------------
|
||||
PG_DUMP="$STAGE/pg_dumpall.sql.gz"
|
||||
if sudo -u postgres pg_isready -h localhost -p 5432 > /dev/null 2>&1; then
|
||||
log "pg_dumpall starting → $PG_DUMP"
|
||||
if sudo -u postgres pg_dumpall -h localhost -p 5432 | gzip > "$PG_DUMP.tmp"; then
|
||||
mv "$PG_DUMP.tmp" "$PG_DUMP"
|
||||
log "pg_dumpall done ($(du -h "$PG_DUMP" | cut -f1))"
|
||||
else
|
||||
warn "pg_dumpall failed (exit $?); keeping previous dump if any"
|
||||
rm -f "$PG_DUMP.tmp"
|
||||
fi
|
||||
else
|
||||
warn "postgres not ready on :5432 — skipping pg_dumpall"
|
||||
fi
|
||||
|
||||
# ---- MongoDB -----------------------------------------------------------
|
||||
MONGO_DIR="$STAGE/mongodump"
|
||||
if mongosh --quiet --eval 'db.adminCommand({ping: 1}).ok' | grep -q '^1$'; then
|
||||
log "mongodump starting → $MONGO_DIR"
|
||||
rm -rf "$MONGO_DIR.tmp"
|
||||
if mongodump --out "$MONGO_DIR.tmp" > /dev/null 2>&1; then
|
||||
rm -rf "$MONGO_DIR"
|
||||
mv "$MONGO_DIR.tmp" "$MONGO_DIR"
|
||||
log "mongodump done ($(du -sh "$MONGO_DIR" | cut -f1))"
|
||||
else
|
||||
warn "mongodump failed (exit $?); keeping previous dump if any"
|
||||
rm -rf "$MONGO_DIR.tmp"
|
||||
fi
|
||||
else
|
||||
warn "mongo not reachable via mongosh — skipping mongodump"
|
||||
fi
|
||||
|
||||
# ---- Retention on stage dir --------------------------------------------
|
||||
# restic dedupes identical dumps at the chunk level, so we can safely keep
|
||||
# overwriting the same files. No explicit rotation needed here.
|
||||
|
||||
log "pre-backup complete"
|
||||
@@ -0,0 +1,89 @@
|
||||
# resticprofile config for esh-vm-db.
|
||||
#
|
||||
# Two-database host (Postgres 15 on :5432, MongoDB on :27017). Small
|
||||
# VM on an ESH hypervisor; backed up at VM-image level by PBS-ANA via
|
||||
# its hypervisor. This restic profile captures:
|
||||
#
|
||||
# 1. Host config (/etc, /root) — fast config recovery without
|
||||
# waiting for VM-image restore
|
||||
# 2. Per-DB dumps via pre-backup hook — pg_dumpall + mongodump,
|
||||
# written to /var/lib/restic/stage/ and included in the restic
|
||||
# snapshot. Gives us DB-level restore granularity alongside the
|
||||
# VM-image restore from PBS.
|
||||
#
|
||||
# What's DELIBERATELY NOT in source:
|
||||
# - /var/lib/postgresql — raw live PGDATA. Inconsistent if captured
|
||||
# while postgres is running; pg_dumpall above covers this.
|
||||
# - /var/lib/mongodb — same reasoning; mongodump covers it.
|
||||
#
|
||||
# Writes cross-site to rest-server-ana (10.250.50.70:8000). No local
|
||||
# NH3-style rest-server on the ESH side; esh-docker-vm and vm-esh-nas
|
||||
# also use rest-server-ana, so this follows fleet pattern.
|
||||
|
||||
version: "1"
|
||||
|
||||
global:
|
||||
priority: low
|
||||
ionice: true
|
||||
ionice-class: 2
|
||||
ionice-level: 7
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false
|
||||
lock: /var/lock/restic-esh-vm-db.lock
|
||||
|
||||
backup:
|
||||
verbose: 1
|
||||
run-before:
|
||||
- /etc/restic/pre-backup.sh
|
||||
run-after:
|
||||
- date +%s > /var/lib/restic/last-success
|
||||
source:
|
||||
- /etc # host config
|
||||
- /root # root's scripts, ssh keys, shell history
|
||||
- /home # user home dirs
|
||||
- /var/lib/restic/stage # pg_dumpall + mongodump outputs from pre-backup
|
||||
exclude:
|
||||
# Raw DB data is captured via pre-backup dumps, not volume-level
|
||||
- /var/lib/postgresql
|
||||
- /var/lib/mongodb
|
||||
# NFS mount from esh-nas (backup target for other stacks — not ours to mirror)
|
||||
- /mnt/backup
|
||||
# Ephemeral / regenerable
|
||||
- "**/*.log"
|
||||
- "**/*.log.*"
|
||||
- "**/*.pid"
|
||||
- /root/.cache
|
||||
- /root/.local/share/Trash
|
||||
- /root/.npm
|
||||
- /root/.python_history
|
||||
- /home/*/.cache
|
||||
- /home/*/.local/share/Trash
|
||||
- /home/*/.npm
|
||||
tag:
|
||||
- host:esh-vm-db
|
||||
- site:esh
|
||||
- fleet:home-lab
|
||||
schedule: "*-*-* 01:00:00"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-backup.log
|
||||
|
||||
forget:
|
||||
keep-daily: 7
|
||||
keep-weekly: 4
|
||||
keep-monthly: 12
|
||||
keep-yearly: 3
|
||||
tag:
|
||||
- host:esh-vm-db
|
||||
# Schedule removed: forget against --append-only rest-server always
|
||||
# fails (delete ops blocked). Run manually during prune ceremony.
|
||||
|
||||
check:
|
||||
read-data-subset: 10%
|
||||
schedule: "Sun *-*-* 05:00:00"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-check.log
|
||||
@@ -0,0 +1,96 @@
|
||||
# ana-nas → nh3-nas restic mirror
|
||||
|
||||
Nightly rsync of the Ana-side restic repo to the NH3 Synology, giving
|
||||
file-level backups cross-site redundancy independent of the PBS layer.
|
||||
|
||||
## What + where
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Source** | `ana-nas:/mnt/backup/restic/repo/ana/` (ZFS `NASPool/backupStore`, written by rest-server-ana on ana-docker via NFS). |
|
||||
| **Target** | `nh3-nas:/volume1/Backup/restic-ana-mirror/` (Btrfs `/volume1`, 16 TB free at setup time). |
|
||||
| **Runs on** | `ana-nas` (Debian 12, VMID 100 on pfi-pve). |
|
||||
| **Runs as** | `lkraven` (uid 1000) — owns the source data natively; no sudo needed. |
|
||||
| **Auth** | Dedicated ed25519 keypair `~lkraven/.ssh/id_mirror_nh3` → `syncuser@nh3-nas`. No passphrase (for unattended systemd runs). |
|
||||
| **Schedule** | Daily at 04:00, `Persistent=true`, 300s randomized delay. |
|
||||
|
||||
## Scheduling rationale
|
||||
|
||||
Window sits between:
|
||||
- **01:00–03:00** — ana-side restic clients run their nightly backups
|
||||
against rest-server-ana. We wait until those are clearly done before
|
||||
reading the repo.
|
||||
- **06:00** — PBS-ANA → PBS-NH3 sync job pulls fresh snapshots across
|
||||
the WAN. Running the restic mirror earlier avoids competing for WAN
|
||||
bandwidth with the (much larger) PBS sync.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Install path on ana-nas |
|
||||
|---|---|
|
||||
| `restic-mirror-to-nh3.service` | `/etc/systemd/system/restic-mirror-to-nh3.service` |
|
||||
| `restic-mirror-to-nh3.timer` | `/etc/systemd/system/restic-mirror-to-nh3.timer` |
|
||||
|
||||
These are the canonical copies; the on-host copies mirror them.
|
||||
|
||||
## Deploy / redeploy
|
||||
|
||||
```bash
|
||||
cd configs/rsync/ana-nas-to-nh3
|
||||
scp restic-mirror-to-nh3.{service,timer} ana-nas:/tmp/
|
||||
ssh -t ana-nas 'sudo install -m 644 /tmp/restic-mirror-to-nh3.service /etc/systemd/system/ && \
|
||||
sudo install -m 644 /tmp/restic-mirror-to-nh3.timer /etc/systemd/system/ && \
|
||||
sudo systemctl daemon-reload && \
|
||||
sudo systemctl enable --now restic-mirror-to-nh3.timer && \
|
||||
sudo systemctl list-timers restic-mirror-to-nh3.timer'
|
||||
```
|
||||
|
||||
## Operate
|
||||
|
||||
```bash
|
||||
# Trigger a run manually (e.g. to smoke-test changes)
|
||||
ssh -t ana-nas 'sudo systemctl start restic-mirror-to-nh3.service'
|
||||
|
||||
# Follow a running transfer
|
||||
ssh ana-nas 'journalctl -u restic-mirror-to-nh3.service -f'
|
||||
|
||||
# Last run outcome
|
||||
ssh ana-nas 'systemctl status restic-mirror-to-nh3.service --no-pager'
|
||||
|
||||
# When the timer fires next
|
||||
ssh ana-nas 'systemctl list-timers restic-mirror-to-nh3.timer'
|
||||
```
|
||||
|
||||
## Restore flow (if ana-nas loses the repo)
|
||||
|
||||
The mirror is just files — point a restic client directly at the
|
||||
nh3-nas copy. Options:
|
||||
|
||||
**Option A — pull repo back to ana-nas and use rest-server-ana as
|
||||
before.** rsync in reverse:
|
||||
|
||||
```bash
|
||||
ssh -t ana-nas 'sudo rsync -a \
|
||||
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3" \
|
||||
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/ \
|
||||
/mnt/backup/restic/repo/ana/'
|
||||
```
|
||||
|
||||
**Option B — restic against the mirror directly.** Expose
|
||||
`/volume1/Backup/restic-ana-mirror/` via a temporary rest-server or
|
||||
NFS share, point the client at it for an emergency restore. Slow (WAN
|
||||
hop) but no dataset copy required.
|
||||
|
||||
Target is `--append-only: false` on the mirror side — not a problem
|
||||
for restore, but means a compromised mirror side *could* be tampered
|
||||
with. Balanced against the operational cost of managing an
|
||||
append-only mirror, we accept this for a mirror-of-a-mirror.
|
||||
|
||||
## What this does NOT cover
|
||||
|
||||
- **PBS snapshots** — those replicate via PBS-ANA → PBS-NH3 (a
|
||||
separate pipeline; see `docs/runbooks/pbs-deployment.md`).
|
||||
- **rest-server-nh3's own data** — nh3-docker + nh3-dev backups that
|
||||
land on `/volume1/Backup/restic/<user>/`. Those are already at the
|
||||
NH3 Synology; ANA-side doesn't mirror them currently. Symmetric
|
||||
mirror back to ana-nas is a future-work item.
|
||||
@@ -0,0 +1,46 @@
|
||||
[Unit]
|
||||
Description=Mirror rest-server-ana restic repo to NH3 Synology
|
||||
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
ConditionPathIsDirectory=/mnt/backup/restic/repo/ana
|
||||
StartLimitBurst=3
|
||||
StartLimitIntervalSec=1h
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
|
||||
# Dedicated key for this job, ed25519, ana-nas → nh3-nas:syncuser.
|
||||
# --append-only on rest-server-ana means source files are never
|
||||
# rewritten or deleted by clients; --delete here mirrors any explicit
|
||||
# prune operations (done out-of-band during the quarterly ceremony).
|
||||
# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
|
||||
ExecStart=/usr/bin/rsync \
|
||||
--archive \
|
||||
--delete \
|
||||
--partial \
|
||||
--info=stats2 \
|
||||
--timeout=300 \
|
||||
--exclude=tmp/ \
|
||||
--exclude=.lock \
|
||||
-e "ssh -i /home/lkraven/.ssh/id_mirror_nh3 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
|
||||
/mnt/backup/restic/repo/ana/ \
|
||||
syncuser@10.100.50.50:/volume1/Backup/restic-ana-mirror/
|
||||
|
||||
# ServerAlive{Interval,CountMax}=30/3 + rsync --timeout bound below
|
||||
# force detection of dead WAN within ~90s; without this, a silent TCP
|
||||
# drop holds the service in "activating" for hours on kernel retransmit
|
||||
# backoff. TimeoutStartSec=6h caps worst-case initial sync; after
|
||||
# that systemd kills the service even if rsync somehow wedges.
|
||||
TimeoutStartSec=6h
|
||||
|
||||
# Tight retry window — a transient WAN blip shouldn't wedge the timer
|
||||
# schedule, but spamming retries on a real outage is noise.
|
||||
# StartLimitBurst/Interval live in [Unit] (systemd v230+); the burst
|
||||
# cap prevents runaway retries even with the short RestartSec.
|
||||
Restart=on-failure
|
||||
RestartSec=30s
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=Daily mirror of rest-server-ana restic repo to NH3
|
||||
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/ana-nas-to-nh3/README.md
|
||||
|
||||
[Timer]
|
||||
# 04:00 daily — sits between ana-side restic clients (01:00 finish
|
||||
# window) and the PBS-ANA → PBS-NH3 sync at 06:00, so WAN contention
|
||||
# is minimized. Persistent=true catches missed runs if ana-nas was
|
||||
# offline; RandomizedDelaySec smears load if this ever scales to
|
||||
# multiple mirror jobs on the same host.
|
||||
OnCalendar=*-*-* 04:00:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=300
|
||||
Unit=restic-mirror-to-nh3.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,104 @@
|
||||
# nh3-nas → ana-nas restic mirror (return leg)
|
||||
|
||||
Symmetric counterpart to `configs/rsync/ana-nas-to-nh3/`. Nightly
|
||||
rsync of the NH3-side restic repo to the Ana NAS, so file-level
|
||||
backups for nh3-docker and nh3-dev have cross-site redundancy.
|
||||
|
||||
## What + where
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Source** | `nh3-nas:/volume1/Backup/restic/` (Btrfs on cachedev_0, written by rest-server-nh3 on nh3-nas). Holds per-client repos: nh3-docker/, nh3-dev/, repo/. |
|
||||
| **Target** | `ana-nas:/mnt/backup/restic-nh3-mirror/` (ZFS `NASPool/backupStore`, 20 TB pool with ~20 TB free at setup). |
|
||||
| **Runs on** | `nh3-nas` (Synology RS2418+, DSM 7+ with native systemd). |
|
||||
| **Runs as** | **`root`** — required because rest-server-nh3's Docker container writes restic files as `admin:users mode 400`, and only root (via CAP_DAC_READ_SEARCH) can read them. Running as `syncuser` was tried first and denied. See gotchas below. |
|
||||
| **Auth** | Dedicated ed25519 keypair `/root/.ssh/id_mirror_ana` → `lkraven@ana-nas`. No passphrase. |
|
||||
| **Schedule** | Daily at 05:00, `Persistent=true`, 300s randomized delay. |
|
||||
|
||||
## Why 05:00
|
||||
|
||||
Slots between:
|
||||
- 04:00 ana-nas → nh3-nas (the other mirror) — lets that finish before
|
||||
NH3 starts pushing anything back.
|
||||
- 06:00 PBS-ANA → PBS-NH3 sync — avoids WAN contention with the much
|
||||
larger PBS replication.
|
||||
|
||||
## DSM-specific notes
|
||||
|
||||
- DSM 7.x ships a stripped/older systemd. Unit files in
|
||||
`/etc/systemd/system/` persist across DSM updates but are NOT
|
||||
guaranteed to — verify after each DSM major update.
|
||||
- **DSM systemd is pre-v247** — it refuses `Restart=on-failure` on
|
||||
`Type=oneshot` services, and does not recognize `StartLimitBurst`
|
||||
or `StartLimitIntervalSec`. This service deliberately omits those;
|
||||
a failed run just means the daily timer picks up again tomorrow.
|
||||
For a mirror used as DR (not primary), that's acceptable.
|
||||
- `systemctl --now` flag is also absent. Use `enable` + `start`
|
||||
separately on DSM.
|
||||
- `syncuser` has `administrators` group membership, so sudo works
|
||||
for unit installation. `syncuser` itself runs the service (no
|
||||
privilege escalation at runtime).
|
||||
- rsync lives at `/usr/bin/rsync` on DSM (same as Debian). Older
|
||||
`/bin/rsync` doesn't exist.
|
||||
- `scp` to DSM needs `-O` (legacy protocol) — sftp subsystem is off
|
||||
by default. Either enable SFTP in DSM (**Control Panel → File
|
||||
Services → FTP → SFTP**) or keep using `scp -O`.
|
||||
- **rest-server-nh3 writes restic files as `admin:users mode 400`**
|
||||
(via DSM ContainerManager). syncuser cannot read these even with
|
||||
admins-group ACL because the newer DSM doesn't ship `synoacltool`
|
||||
to add per-user ACEs. The service therefore runs as root
|
||||
(CAP_DAC_READ_SEARCH bypasses the 400 mode).
|
||||
- **rsync `--archive` broke cross-filesystem perm/owner preservation**
|
||||
from Btrfs-with-syno_acl → ZFS-on-Debian — the dest dir ended up
|
||||
mode `0000` and rsync couldn't recover on subsequent runs. Switched
|
||||
to `-rlt --no-perms --no-owner --no-group --chmod=D755,F644` which
|
||||
copies contents verbatim and forces sensible dest perms. Restic's
|
||||
repo integrity doesn't rely on Unix perms so this is a clean swap.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Install path on nh3-nas |
|
||||
|---|---|
|
||||
| `restic-mirror-to-ana.service` | `/etc/systemd/system/restic-mirror-to-ana.service` |
|
||||
| `restic-mirror-to-ana.timer` | `/etc/systemd/system/restic-mirror-to-ana.timer` |
|
||||
|
||||
## Prereqs (one-time setup)
|
||||
|
||||
1. Generate the keypair on nh3-nas as syncuser:
|
||||
```bash
|
||||
ssh nh3-nas "test -f ~/.ssh/id_mirror_ana || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_mirror_ana -C 'nh3-nas→ana-nas restic mirror'"
|
||||
```
|
||||
2. Install the pubkey on ana-nas:
|
||||
```bash
|
||||
ssh nh3-nas 'cat ~/.ssh/id_mirror_ana.pub' | \
|
||||
ssh ana-nas 'cat >> ~/.ssh/authorized_keys'
|
||||
```
|
||||
3. Create the destination dir on ana-nas:
|
||||
```bash
|
||||
ssh ana-nas 'sudo mkdir -p /mnt/backup/restic-nh3-mirror && \
|
||||
sudo chown lkraven:lkraven /mnt/backup/restic-nh3-mirror'
|
||||
```
|
||||
4. Smoke-test from nh3-nas as syncuser:
|
||||
```bash
|
||||
ssh nh3-nas 'ssh -i ~/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new \
|
||||
lkraven@10.250.50.50 "hostname && ls -ld /mnt/backup/restic-nh3-mirror"'
|
||||
```
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
cd configs/rsync/nh3-nas-to-ana
|
||||
scp restic-mirror-to-ana.{service,timer} nh3-nas:/tmp/
|
||||
ssh -t nh3-nas 'sudo install -m 644 /tmp/restic-mirror-to-ana.service /etc/systemd/system/ && \
|
||||
sudo install -m 644 /tmp/restic-mirror-to-ana.timer /etc/systemd/system/ && \
|
||||
sudo systemctl daemon-reload && \
|
||||
sudo systemctl enable --now restic-mirror-to-ana.timer && \
|
||||
sudo systemctl start restic-mirror-to-ana.service && \
|
||||
sudo systemctl list-timers restic-mirror-to-ana.timer'
|
||||
```
|
||||
|
||||
## Operate
|
||||
|
||||
Mirrors the ana-nas side — see that README. Replace
|
||||
`restic-mirror-to-nh3` with `restic-mirror-to-ana` and `ana-nas` with
|
||||
`nh3-nas` in the commands there.
|
||||
@@ -0,0 +1,52 @@
|
||||
[Unit]
|
||||
Description=Mirror rest-server-nh3 restic repo to ANA NAS
|
||||
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/nh3-nas-to-ana/README.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
ConditionPathIsDirectory=/volume1/Backup/restic
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
Group=root
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
|
||||
# Runs as root because rest-server-nh3's Docker container writes the
|
||||
# restic repo files as admin:users mode 400 — only root bypasses that
|
||||
# via CAP_DAC_READ_SEARCH. Running as syncuser (even with admins
|
||||
# group membership) was denied by POSIX mode. Earlier attempt with
|
||||
# synoacltool ACL grants failed because this DSM version ships
|
||||
# without the tool.
|
||||
#
|
||||
# Dedicated key for this job, ed25519, nh3-nas:root → ana-nas:lkraven.
|
||||
# Does NOT include /volume1/Backup/restic-ana-mirror (that's data
|
||||
# ana-nas just sent us; mirroring it back would be a dedupe-less loop).
|
||||
# Source is the per-host tree under /volume1/Backup/restic/ only:
|
||||
# nh3-docker/, nh3-dev/, repo/.
|
||||
# tmp/ and .lock excluded to avoid mirroring in-flight transfers.
|
||||
# SSH keepalive + rsync --timeout ensure a dead WAN is detected in
|
||||
# ~90s instead of hanging on kernel TCP retransmit backoff.
|
||||
#
|
||||
# NO Restart=on-failure here: DSM's systemd is pre-v247 and refuses
|
||||
# Restart= on Type=oneshot. If a run fails, the daily timer picks
|
||||
# up again tomorrow — a single missed mirror is acceptable for DR.
|
||||
ExecStart=/usr/bin/rsync \
|
||||
--recursive \
|
||||
--links \
|
||||
--times \
|
||||
--no-perms \
|
||||
--no-owner \
|
||||
--no-group \
|
||||
--chmod=D755,F644 \
|
||||
--delete \
|
||||
--partial \
|
||||
--info=stats2 \
|
||||
--timeout=300 \
|
||||
--exclude=tmp/ \
|
||||
--exclude=.lock \
|
||||
-e "ssh -i /root/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ConnectTimeout=30" \
|
||||
/volume1/Backup/restic/ \
|
||||
lkraven@10.250.50.50:/mnt/backup/restic-nh3-mirror/
|
||||
|
||||
TimeoutStartSec=6h
|
||||
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=Daily mirror of rest-server-nh3 restic repo to ANA
|
||||
Documentation=https://github.com/lkraven/eshpfi-management/blob/main/configs/rsync/nh3-nas-to-ana/README.md
|
||||
|
||||
[Timer]
|
||||
# 05:00 daily — runs after ana→nh3 at 04:00, before PBS sync at
|
||||
# 06:00. Same persistence + jitter pattern.
|
||||
OnCalendar=*-*-* 05:00:00
|
||||
Persistent=true
|
||||
RandomizedDelaySec=300
|
||||
Unit=restic-mirror-to-ana.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in New Issue
Block a user