vh
80d982d1d8
The FV edge firewall was not backed up anywhere. Its config now lands in
/var/lib/restic/stage/fv-gateway-config.xml via ana-docker's pre-backup hook,
so the existing 01:00 restic snapshot captures it. ana-docker is one of the
three egress addresses the firewall's WAN allowlist permits, which is why the
pull lives there rather than with the FV hardware — a site that has lost power
cannot back itself up, and FV lost power two days ago.
Non-fatal by design: an unreachable firewall must not abort the nightly
database dumps. But a bad pull must not be promoted either. The summary loop
only rejects EMPTY staged files, and this endpoint answers an auth failure
with a perfectly non-empty HTML error page — which would have been backed up
as a firewall config that is the right size and restores nothing. The block
checks the body really contains <opnsense> and writes nothing otherwise.
Three tests cover it, including the HTML-error-page case. The first draft of
those tests was worthless: _fv returned a Path out of a TemporaryDirectory
context, so the tree was deleted before the assertions ran and every
exists()-is-False check passed regardless of what the script did. Only the
positive test failed, which is the sole reason the broken negatives were
caught. They now snapshot inside the tempdir's lifetime, and the docstring
says why.
Also records two OPNsense API lessons in docs/pfi/opnsense-api-reference.md:
endpoints are actions and must never be probed for existence by POSTing at
them — that is how /api/core/system/reboot took the FV site dark for 3.5
minutes while looking for an apply call this same file already documented —
and the apply step is service/reconfigure, which auth/user notably lacks, so
an API-only key edit persists in config.xml and does nothing until the OS user
sync runs at boot.
Credentials in /etc/restic/fv-gateway.env (root:600), template committed,
values vaulted as fv-gateway/opnsense-api-{key,secret}. Pre-change config
snapshot vaulted as fv-gateway/config-backup-20260914.
ana-docker database staging
pre-backup.sh runs as root through resticprofile's run-before. Deploy and
exercise it with scripts/elway infra-ops@10.250.50.70 --playbook playbooks/ana-docker-backup-repair.yaml (allow about 12 minutes for Gitea).
Required dump failures now abort the backup. Dumps are prepared in a private pending directory; previous staged files are replaced only after successful generation and nonempty checks. Gitea additionally gets tar validation and a private in-container scratch directory with exit cleanup. Ordinary failures clean up scratch; SIGKILL or host crashes cannot run shell traps.
Regression checks: python3 configs/restic/ana-docker/test_pre_backup.py.
The stage/credential path overrides are for isolated tests; production defaults
remain /var/lib/restic/stage and /etc/restic/dbcreds.env.
Repair verified 2026-09-10
Gitea dumps had failed since June 4 because five root-only historical app.ini
copies inside its config directory were unreadable to the git dump user.
Those copies retain root-only permissions under
/opt/docker/backups/gitea-config-history/, included in backups. Do not put
unreadable config history back inside Gitea's dump tree.
Vaultwarden's stale backup credentials were synchronized with its live database
connection; the root:600 host file is also saved as Vaultwarden item
ana-docker/restic-dbcreds.env. No secrets belong in this repository.
Fresh stage files were saved to Restic snapshot 2ec5a37c. Both database dumps
were restored FROM that repository snapshot into disposable PostgreSQL 16 with
network disabled and tmpfs storage: Vaultwarden 7 users, Gitea 9 users and
93 repositories. The temporary database container was removed afterward.
This verifies database restore, not a complete application disaster-recovery drill.
Only after successful restores, 101 abandoned Gitea dump files (47.31 GiB logical
size) were deleted. Their inventory is root-only at
/opt/docker/backups/gitea-config-history/removed-dump-manifest-20260910.json.
Old failed-dump history itself was not retained; the fresh replacement is in
Restic. The verification marker is /var/lib/restic/verified-repair-20260910.
OpenWebUI's existing warning/fallback to volume capture when sqlite3 is absent is unchanged; this repair does not claim a verified OpenWebUI database backup.