Work by a parallel session on 2026-09-12; committed here with the rest of the
day's changes. Rationale in persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md.
The only visible symptom was a systemd-failed unit from a Sep 6 repository
network timeout after boot. The real fault was quieter and much worse: the
pre-backup hook logged failures as WARN and returned zero, so pg_dumpall could
fail every single night -- it used TCP localhost and wanted a password nobody
supplied -- while restic dutifully backed up the stale April 23 dump still
sitting in the staging directory and reported success. Mongo was fine, which
is part of why it went unnoticed.
Postgres now dumps over the /var/run/postgresql socket with peer auth and -w,
and both database failures now fail the backup rather than masking it, while
still preserving any prior per-DB dump rather than truncating to nothing. An
ERRORS counter replaces the warn-and-continue path, and the staging directory
is overridable via RESTIC_STAGE_DIR so the new test can exercise it.
Adds backup.contract.md, retry.conf and test_pre_backup.py -- three red-green
regression tests covering the failure modes above. systemd drop-ins on both
jobs add network-online ordering plus Restart=on-failure with a 5m delay and a
3-per-hour limit, which addresses the original boot-timeout symptom.
Verified against a real run: snapshot bc5eeaff at 07:01 PDT with a fresh 3.46MB
PG dump, retrieved from the repository with decompression and completion marker
checked (not a full restore). Repository check passed, 99 snapshots. The old
hook and stale dump are preserved root-only at /var/lib/restic/repair-20260912.
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
vh