Repair ana-docker database backups and bound CI build caches

2026-09-10 09:10:01 -07:00
parent eb75713c1b
commit 44c853cd20
11 changed files with 277 additions and 26 deletions
@@ -0,0 +1,42 @@
+# ana-docker database staging
+
+`pre-backup.sh` runs as root through resticprofile's `run-before`. Deploy and
+exercise it with `scripts/elway infra-ops@10.250.50.70 --playbook
+playbooks/ana-docker-backup-repair.yaml` (allow about 12 minutes for Gitea).
+
+Required dump failures now abort the backup. Dumps are prepared in a private
+pending directory; previous staged files are replaced only after successful
+generation and nonempty checks. Gitea additionally gets tar validation and a
+private in-container scratch directory with exit cleanup. Ordinary failures
+clean up scratch; SIGKILL or host crashes cannot run shell traps.
+
+Regression checks: `python3 configs/restic/ana-docker/test_pre_backup.py`.
+The stage/credential path overrides are for isolated tests; production defaults
+remain `/var/lib/restic/stage` and `/etc/restic/dbcreds.env`.
+
+## Repair verified 2026-09-10
+
+Gitea dumps had failed since June 4 because five root-only historical app.ini
+copies inside its config directory were unreadable to the git dump user.
+Those copies retain root-only permissions under
+`/opt/docker/backups/gitea-config-history/`, included in backups. Do not put
+unreadable config history back inside Gitea's dump tree.
+
+Vaultwarden's stale backup credentials were synchronized with its live database
+connection; the root:600 host file is also saved as Vaultwarden item
+`ana-docker/restic-dbcreds.env`. No secrets belong in this repository.
+
+Fresh stage files were saved to Restic snapshot `2ec5a37c`. Both database dumps
+were restored FROM that repository snapshot into disposable PostgreSQL 16 with
+network disabled and tmpfs storage: Vaultwarden 7 users, Gitea 9 users and
+93 repositories. The temporary database container was removed afterward.
+This verifies database restore, not a complete application disaster-recovery drill.
+
+Only after successful restores, 101 abandoned Gitea dump files (47.31 GiB logical
+size) were deleted. Their inventory is root-only at
+`/opt/docker/backups/gitea-config-history/removed-dump-manifest-20260910.json`.
+Old failed-dump history itself was not retained; the fresh replacement is in
+Restic. The verification marker is `/var/lib/restic/verified-repair-20260910`.
+
+OpenWebUI's existing warning/fallback to volume capture when sqlite3 is absent
+is unchanged; this repair does not claim a verified OpenWebUI database backup.