Commit Graph

3 Commits

Author SHA1 Message Date
vh 574c72daa5 backup pipeline: configs, runbooks, NH3 Synology rest-server, cross-site rsync
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.

- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
  *_offen_backup_data exclude (offen sidecars retired fleet-wide
  2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
  (10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
  and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
  rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
  refined during the 2026-04-22 deployment with per-hypervisor
  namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
  Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
  Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
  references incident memory + recovery-step playbooks per consumer.
2026-04-24 21:56:22 -07:00
vh 76a0768fdb restic: drop scheduled forget across all 6 hosts
Forget against an --append-only rest-server fails every night (delete
ops blocked). The resulting daily failure cluttered service status and
logs without ever actually retiring old snapshots. Schedule is now
removed from the forget block in all six profiles; the keep-daily /
keep-weekly / keep-monthly / keep-yearly policy remains so manual
invocations (during prune ceremonies, when --append-only is
temporarily off) honor the intended retention.

Files:
  configs/restic/ana-docker/profiles.yaml
  configs/restic/ana-ml2/profiles.yaml
  configs/restic/nh3-docker/profiles.yaml
  configs/restic/esh-docker-vm/profiles.yaml
  configs/restic/vm-esh-nas/profiles.yaml
  configs/restic/nh3-dev/profiles.yaml

Each file has an inline comment marking why the schedule was dropped
so a future reader doesn't re-add it thinking it was an oversight.

STATUS.md: removed the "install Backrest nightly-restart timer" line
item. User confirmed the UI timeout hits even at startup, so periodic
restart wouldn't actually help. Root cause remains deferred.
2026-04-21 17:00:23 -07:00
vh 1e24e0f799 restic/esh-docker-vm: profile + DB hooks + deploy guide
Closes the last file-level backup gap. Primary ESH home-lab Docker host
— five services with state worth consistent dumps:

  - paperless-ngx → external Postgres on 10.0.50.60 (host pg_dump)
  - home-assistant → local SQLite ~50MB (host sqlite3 .backup)
  - calibre-web-automated → local SQLite (in-container sqlite3)
  - pgadmin → local SQLite (host sqlite3)
  - uptime-kuma → local SQLite (host sqlite3; container name may vary
    after force-recreate, detect by label)

Unique to this host: HA/pgadmin/uptime-kuma images don't bundle sqlite3.
Rather than maintaining custom images, pre-backup.sh runs sqlite3 from
the HOST against the volume bind-mount paths. Requires sqlite3 +
postgresql-client installed on esh-docker-vm.

Cross-site writes to rest-server-ana since ESH has no local rest-server.

NFS mounts (/mnt/{backup,books,compose,documents}) explicitly excluded
— hundreds of GB of NAS-side content backed up at the NAS layer. Also
excludes offen-sidecar buffer volumes (paperless + pgadmin currently
run offen/docker-volume-backup alongside — retire once restic has a
week of clean runs).

Found in audit (non-blocking but noted in README follow-ups):
  - paperless-ngx Postgres password is literally "paperless-ng" —
    trivially weak, rotate at next opportunity.
2026-04-21 11:04:37 -07:00