vh
574c72daa5
Bundles the post-2026-04-21 work that built out the two-layer backup architecture (PBS for VM images + restic for file/DB), plus the cross- site mirror and the disaster-recovery runbook. - configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete *_offen_backup_data exclude (offen sidecars retired fleet-wide 2026-04-23; restic now covers the equivalent scope directly). - configs/restic/esh-vm-db/: new profile for the dedicated DB VM (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks. - configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven) and nh3-nas → ana-nas (05:00 daily, runs as root because DSM rest-server-nh3 writes mode-400 files only root can read). - docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook, refined during the 2026-04-22 deployment with per-hypervisor namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the Synology syno_acl flatten step. - docs/runbooks/disaster-recovery.md: blast-radius runbook ordered Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty); references incident memory + recovery-step playbooks per consumer.
105 lines
4.8 KiB
Markdown
105 lines
4.8 KiB
Markdown
# nh3-nas → ana-nas restic mirror (return leg)
|
|
|
|
Symmetric counterpart to `configs/rsync/ana-nas-to-nh3/`. Nightly
|
|
rsync of the NH3-side restic repo to the Ana NAS, so file-level
|
|
backups for nh3-docker and nh3-dev have cross-site redundancy.
|
|
|
|
## What + where
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Source** | `nh3-nas:/volume1/Backup/restic/` (Btrfs on cachedev_0, written by rest-server-nh3 on nh3-nas). Holds per-client repos: nh3-docker/, nh3-dev/, repo/. |
|
|
| **Target** | `ana-nas:/mnt/backup/restic-nh3-mirror/` (ZFS `NASPool/backupStore`, 20 TB pool with ~20 TB free at setup). |
|
|
| **Runs on** | `nh3-nas` (Synology RS2418+, DSM 7+ with native systemd). |
|
|
| **Runs as** | **`root`** — required because rest-server-nh3's Docker container writes restic files as `admin:users mode 400`, and only root (via CAP_DAC_READ_SEARCH) can read them. Running as `syncuser` was tried first and denied. See gotchas below. |
|
|
| **Auth** | Dedicated ed25519 keypair `/root/.ssh/id_mirror_ana` → `lkraven@ana-nas`. No passphrase. |
|
|
| **Schedule** | Daily at 05:00, `Persistent=true`, 300s randomized delay. |
|
|
|
|
## Why 05:00
|
|
|
|
Slots between:
|
|
- 04:00 ana-nas → nh3-nas (the other mirror) — lets that finish before
|
|
NH3 starts pushing anything back.
|
|
- 06:00 PBS-ANA → PBS-NH3 sync — avoids WAN contention with the much
|
|
larger PBS replication.
|
|
|
|
## DSM-specific notes
|
|
|
|
- DSM 7.x ships a stripped/older systemd. Unit files in
|
|
`/etc/systemd/system/` persist across DSM updates but are NOT
|
|
guaranteed to — verify after each DSM major update.
|
|
- **DSM systemd is pre-v247** — it refuses `Restart=on-failure` on
|
|
`Type=oneshot` services, and does not recognize `StartLimitBurst`
|
|
or `StartLimitIntervalSec`. This service deliberately omits those;
|
|
a failed run just means the daily timer picks up again tomorrow.
|
|
For a mirror used as DR (not primary), that's acceptable.
|
|
- `systemctl --now` flag is also absent. Use `enable` + `start`
|
|
separately on DSM.
|
|
- `syncuser` has `administrators` group membership, so sudo works
|
|
for unit installation. `syncuser` itself runs the service (no
|
|
privilege escalation at runtime).
|
|
- rsync lives at `/usr/bin/rsync` on DSM (same as Debian). Older
|
|
`/bin/rsync` doesn't exist.
|
|
- `scp` to DSM needs `-O` (legacy protocol) — sftp subsystem is off
|
|
by default. Either enable SFTP in DSM (**Control Panel → File
|
|
Services → FTP → SFTP**) or keep using `scp -O`.
|
|
- **rest-server-nh3 writes restic files as `admin:users mode 400`**
|
|
(via DSM ContainerManager). syncuser cannot read these even with
|
|
admins-group ACL because the newer DSM doesn't ship `synoacltool`
|
|
to add per-user ACEs. The service therefore runs as root
|
|
(CAP_DAC_READ_SEARCH bypasses the 400 mode).
|
|
- **rsync `--archive` broke cross-filesystem perm/owner preservation**
|
|
from Btrfs-with-syno_acl → ZFS-on-Debian — the dest dir ended up
|
|
mode `0000` and rsync couldn't recover on subsequent runs. Switched
|
|
to `-rlt --no-perms --no-owner --no-group --chmod=D755,F644` which
|
|
copies contents verbatim and forces sensible dest perms. Restic's
|
|
repo integrity doesn't rely on Unix perms so this is a clean swap.
|
|
|
|
## Files
|
|
|
|
| File | Install path on nh3-nas |
|
|
|---|---|
|
|
| `restic-mirror-to-ana.service` | `/etc/systemd/system/restic-mirror-to-ana.service` |
|
|
| `restic-mirror-to-ana.timer` | `/etc/systemd/system/restic-mirror-to-ana.timer` |
|
|
|
|
## Prereqs (one-time setup)
|
|
|
|
1. Generate the keypair on nh3-nas as syncuser:
|
|
```bash
|
|
ssh nh3-nas "test -f ~/.ssh/id_mirror_ana || ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_mirror_ana -C 'nh3-nas→ana-nas restic mirror'"
|
|
```
|
|
2. Install the pubkey on ana-nas:
|
|
```bash
|
|
ssh nh3-nas 'cat ~/.ssh/id_mirror_ana.pub' | \
|
|
ssh ana-nas 'cat >> ~/.ssh/authorized_keys'
|
|
```
|
|
3. Create the destination dir on ana-nas:
|
|
```bash
|
|
ssh ana-nas 'sudo mkdir -p /mnt/backup/restic-nh3-mirror && \
|
|
sudo chown lkraven:lkraven /mnt/backup/restic-nh3-mirror'
|
|
```
|
|
4. Smoke-test from nh3-nas as syncuser:
|
|
```bash
|
|
ssh nh3-nas 'ssh -i ~/.ssh/id_mirror_ana -o StrictHostKeyChecking=accept-new \
|
|
lkraven@10.250.50.50 "hostname && ls -ld /mnt/backup/restic-nh3-mirror"'
|
|
```
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
cd configs/rsync/nh3-nas-to-ana
|
|
scp restic-mirror-to-ana.{service,timer} nh3-nas:/tmp/
|
|
ssh -t nh3-nas 'sudo install -m 644 /tmp/restic-mirror-to-ana.service /etc/systemd/system/ && \
|
|
sudo install -m 644 /tmp/restic-mirror-to-ana.timer /etc/systemd/system/ && \
|
|
sudo systemctl daemon-reload && \
|
|
sudo systemctl enable --now restic-mirror-to-ana.timer && \
|
|
sudo systemctl start restic-mirror-to-ana.service && \
|
|
sudo systemctl list-timers restic-mirror-to-ana.timer'
|
|
```
|
|
|
|
## Operate
|
|
|
|
Mirrors the ana-nas side — see that README. Replace
|
|
`restic-mirror-to-nh3` with `restic-mirror-to-ana` and `ana-nas` with
|
|
`nh3-nas` in the commands there.
|