backup pipeline: configs, runbooks, NH3 Synology rest-server, cross-site rsync
Bundles the post-2026-04-21 work that built out the two-layer backup architecture (PBS for VM images + restic for file/DB), plus the cross- site mirror and the disaster-recovery runbook. - configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete *_offen_backup_data exclude (offen sidecars retired fleet-wide 2026-04-23; restic now covers the equivalent scope directly). - configs/restic/esh-vm-db/: new profile for the dedicated DB VM (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks. - configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven) and nh3-nas → ana-nas (05:00 daily, runs as root because DSM rest-server-nh3 writes mode-400 files only root can read). - docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook, refined during the 2026-04-22 deployment with per-hypervisor namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the Synology syno_acl flatten step. - docs/runbooks/disaster-recovery.md: blast-radius runbook ordered Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty); references incident memory + recovery-step playbooks per consumer.
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# resticprofile config for esh-vm-db.
|
||||
#
|
||||
# Two-database host (Postgres 15 on :5432, MongoDB on :27017). Small
|
||||
# VM on an ESH hypervisor; backed up at VM-image level by PBS-ANA via
|
||||
# its hypervisor. This restic profile captures:
|
||||
#
|
||||
# 1. Host config (/etc, /root) — fast config recovery without
|
||||
# waiting for VM-image restore
|
||||
# 2. Per-DB dumps via pre-backup hook — pg_dumpall + mongodump,
|
||||
# written to /var/lib/restic/stage/ and included in the restic
|
||||
# snapshot. Gives us DB-level restore granularity alongside the
|
||||
# VM-image restore from PBS.
|
||||
#
|
||||
# What's DELIBERATELY NOT in source:
|
||||
# - /var/lib/postgresql — raw live PGDATA. Inconsistent if captured
|
||||
# while postgres is running; pg_dumpall above covers this.
|
||||
# - /var/lib/mongodb — same reasoning; mongodump covers it.
|
||||
#
|
||||
# Writes cross-site to rest-server-ana (10.250.50.70:8000). No local
|
||||
# NH3-style rest-server on the ESH side; esh-docker-vm and vm-esh-nas
|
||||
# also use rest-server-ana, so this follows fleet pattern.
|
||||
|
||||
version: "1"
|
||||
|
||||
global:
|
||||
priority: low
|
||||
ionice: true
|
||||
ionice-class: 2
|
||||
ionice-level: 7
|
||||
min-memory: 100
|
||||
|
||||
default:
|
||||
env-file: /etc/restic/restic.env
|
||||
env:
|
||||
RESTIC_PASSWORD_FILE: /etc/restic/password
|
||||
initialize: false
|
||||
lock: /var/lock/restic-esh-vm-db.lock
|
||||
|
||||
backup:
|
||||
verbose: 1
|
||||
run-before:
|
||||
- /etc/restic/pre-backup.sh
|
||||
run-after:
|
||||
- date +%s > /var/lib/restic/last-success
|
||||
source:
|
||||
- /etc # host config
|
||||
- /root # root's scripts, ssh keys, shell history
|
||||
- /home # user home dirs
|
||||
- /var/lib/restic/stage # pg_dumpall + mongodump outputs from pre-backup
|
||||
exclude:
|
||||
# Raw DB data is captured via pre-backup dumps, not volume-level
|
||||
- /var/lib/postgresql
|
||||
- /var/lib/mongodb
|
||||
# NFS mount from esh-nas (backup target for other stacks — not ours to mirror)
|
||||
- /mnt/backup
|
||||
# Ephemeral / regenerable
|
||||
- "**/*.log"
|
||||
- "**/*.log.*"
|
||||
- "**/*.pid"
|
||||
- /root/.cache
|
||||
- /root/.local/share/Trash
|
||||
- /root/.npm
|
||||
- /root/.python_history
|
||||
- /home/*/.cache
|
||||
- /home/*/.local/share/Trash
|
||||
- /home/*/.npm
|
||||
tag:
|
||||
- host:esh-vm-db
|
||||
- site:esh
|
||||
- fleet:home-lab
|
||||
schedule: "*-*-* 01:00:00"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-backup.log
|
||||
|
||||
forget:
|
||||
keep-daily: 7
|
||||
keep-weekly: 4
|
||||
keep-monthly: 12
|
||||
keep-yearly: 3
|
||||
tag:
|
||||
- host:esh-vm-db
|
||||
# Schedule removed: forget against --append-only rest-server always
|
||||
# fails (delete ops blocked). Run manually during prune ceremony.
|
||||
|
||||
check:
|
||||
read-data-subset: 10%
|
||||
schedule: "Sun *-*-* 05:00:00"
|
||||
schedule-permission: system
|
||||
schedule-log: /var/log/restic-check.log
|
||||
Reference in New Issue
Block a user