From 80d982d1d8f7e108a5c86e55d4cd11ad51da65b8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 00:12:50 -0700 Subject: [PATCH] feat(backup): stage the FV firewall config in ana-docker's nightly restic run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The FV edge firewall was not backed up anywhere. Its config now lands in /var/lib/restic/stage/fv-gateway-config.xml via ana-docker's pre-backup hook, so the existing 01:00 restic snapshot captures it. ana-docker is one of the three egress addresses the firewall's WAN allowlist permits, which is why the pull lives there rather than with the FV hardware — a site that has lost power cannot back itself up, and FV lost power two days ago. Non-fatal by design: an unreachable firewall must not abort the nightly database dumps. But a bad pull must not be promoted either. The summary loop only rejects EMPTY staged files, and this endpoint answers an auth failure with a perfectly non-empty HTML error page — which would have been backed up as a firewall config that is the right size and restores nothing. The block checks the body really contains and writes nothing otherwise. Three tests cover it, including the HTML-error-page case. The first draft of those tests was worthless: _fv returned a Path out of a TemporaryDirectory context, so the tree was deleted before the assertions ran and every exists()-is-False check passed regardless of what the script did. Only the positive test failed, which is the sole reason the broken negatives were caught. They now snapshot inside the tempdir's lifetime, and the docstring says why. Also records two OPNsense API lessons in docs/pfi/opnsense-api-reference.md: endpoints are actions and must never be probed for existence by POSTing at them — that is how /api/core/system/reboot took the FV site dark for 3.5 minutes while looking for an apply call this same file already documented — and the apply step is service/reconfigure, which auth/user notably lacks, so an API-only key edit persists in config.xml and does nothing until the OS user sync runs at boot. Credentials in /etc/restic/fv-gateway.env (root:600), template committed, values vaulted as fv-gateway/opnsense-api-{key,secret}. Pre-change config snapshot vaulted as fv-gateway/config-backup-20260914. --- .../restic/ana-docker/fv-gateway.env.example | 8 +++ configs/restic/ana-docker/pre-backup.sh | 42 +++++++++++++++ configs/restic/ana-docker/test_pre_backup.py | 51 +++++++++++++++++++ docs/pfi/opnsense-api-reference.md | 44 ++++++++++++++++ 4 files changed, 145 insertions(+) create mode 100644 configs/restic/ana-docker/fv-gateway.env.example diff --git a/configs/restic/ana-docker/fv-gateway.env.example b/configs/restic/ana-docker/fv-gateway.env.example new file mode 100644 index 0000000..a8326f0 --- /dev/null +++ b/configs/restic/ana-docker/fv-gateway.env.example @@ -0,0 +1,8 @@ +# /etc/restic/fv-gateway.env on ana-docker — root:root 0600. +# Real values are vaulted: `secret get fv-gateway/opnsense-api-key` +# `secret get fv-gateway/opnsense-api-secret` +# OPNsense API auth is HTTP Basic with key=username, secret=password. +FV_API_KEY= +FV_API_SECRET= +# Optional; defaults to the FV WAN address. +#FV_GATEWAY_HOST=172.83.89.66 diff --git a/configs/restic/ana-docker/pre-backup.sh b/configs/restic/ana-docker/pre-backup.sh index 27a9797..abdc24d 100755 --- a/configs/restic/ana-docker/pre-backup.sh +++ b/configs/restic/ana-docker/pre-backup.sh @@ -13,6 +13,14 @@ # - gitea (`gitea dump` captures DB + repos + config + LFS) # - openwebui (local SQLite × 2 — main db + ChromaDB vector store) # +# Also staged here (not a container): +# - fv-gateway (OPNsense config.xml for the Fountain Valley edge +# firewall, pulled over its WAN admin API). ana-docker is +# one of the three egress addresses that firewall's +# allowlist permits, which is why the pull lives on this +# host rather than with the FV hardware — a site that has +# lost power cannot back itself up. +# # External DB credentials live in /etc/restic/dbcreds.env (root:600). # Template: configs/restic/ana-docker/dbcreds.env.example in the repo. # @@ -48,6 +56,13 @@ if [ -r "$CREDS" ]; then set -a; . "$CREDS"; set +a fi +# FV gateway API creds, same shape and posture as dbcreds.env (root:600). +# Template: configs/restic/ana-docker/fv-gateway.env.example in the repo. +FVCREDS=${RESTIC_FV_CREDS_FILE:-/etc/restic/fv-gateway.env} +if [ -r "$FVCREDS" ]; then + set -a; . "$FVCREDS"; set +a +fi + # ---------- synapse (internal Postgres) --------------------------------------- if docker inspect synapse-db >/dev/null 2>&1; then log "dumping synapse postgres" @@ -155,6 +170,33 @@ else log "skip openwebui: container not present" fi +# ---------- fv-gateway (OPNsense edge firewall config) ------------------------- +# Non-fatal by design: a firewall we cannot reach must not abort the nightly +# database dumps. But a bad pull must not be PROMOTED either — the summary loop +# below only rejects EMPTY files, and this endpoint answers an auth failure or a +# captive portal with a perfectly non-empty HTML error page. So validate that the +# body is really an OPNsense config and write nothing at all otherwise. +FV_HOST=${FV_GATEWAY_HOST:-172.83.89.66} +if [ -n "${FV_API_KEY:-}" ] && [ -n "${FV_API_SECRET:-}" ]; then + log "pulling fv-gateway config from $FV_HOST" + fv_tmp="$WORK/.fv-config.raw" + if curl -fsS --max-time 60 -u "$FV_API_KEY:$FV_API_SECRET" \ + -o "$fv_tmp" "http://$FV_HOST/api/core/backup/download/this" 2>/dev/null; then + if head -c 200 "$fv_tmp" | grep -q ''; then + mv -f -- "$fv_tmp" "$WORK/fv-gateway-config.xml" + log "fv-gateway config staged ($(wc -c < "$WORK/fv-gateway-config.xml") bytes)" + else + rm -f -- "$fv_tmp" + warn "fv-gateway: response was not an OPNsense config (auth failure or error page?)" + fi + else + rm -f -- "$fv_tmp" + warn "fv-gateway: config pull failed (site unreachable?)" + fi +else + log "skip fv-gateway: no API creds in $FVCREDS" +fi + # ---------- summary ----------------------------------------------------------- if [ "$ERRORS" -ne 0 ]; then log "FAILED: $ERRORS required database dump(s) failed; previous stage preserved" diff --git a/configs/restic/ana-docker/test_pre_backup.py b/configs/restic/ana-docker/test_pre_backup.py index 84a81fd..aa67947 100644 --- a/configs/restic/ana-docker/test_pre_backup.py +++ b/configs/restic/ana-docker/test_pre_backup.py @@ -49,4 +49,55 @@ class BackupHookTests(unittest.TestCase): self.assertEqual(previous.read_bytes(),b'previous-good-archive') self.assertEqual(list(stage.glob('.pending.*')),[]) + def _fv(self, curl_body, curl_rc=0): + """Run the hook with a stubbed curl for the fv-gateway pull. + + Returns a snapshot taken INSIDE the temp dir's lifetime. Returning the + Path instead is a trap: TemporaryDirectory deletes the tree on exit, so + every `exists()` assertion afterwards reads False and the negative tests + pass no matter what the script did. Caught 2026-09-14 — the positive + test failed, which is the only reason the broken negatives were noticed. + + The guard under test is subtle: the summary loop only rejects EMPTY + staged files, and the OPNsense endpoint answers an auth failure with a + perfectly non-empty HTML error page. Without the content check that page + would be promoted and backed up as if it were a firewall config — a + backup that exists, is a plausible size, and restores nothing. + """ + with tempfile.TemporaryDirectory(prefix='backup-fv-test-') as d: + root=Path(d);stage=root/'stage';stage.mkdir();binpath=root/'bin';binpath.mkdir() + docker=binpath/'docker';docker.write_text('#!/bin/sh\nexit 1\n');docker.chmod(0o755) + curl=binpath/'curl' + curl.write_text('#!/bin/sh\nout=""\nwhile [ "$#" -gt 0 ]; do [ "$1" = -o ] && { shift; out=$1; }; shift; done\n' + 'printf %s "$FV_STUB_BODY" > "$out"\nexit '+str(curl_rc)+'\n') + curl.chmod(0o755) + creds=root/'fv.env';creds.write_text('FV_API_KEY=k\nFV_API_SECRET=s\n') + env=dict(os.environ,PATH=str(binpath)+':'+os.environ['PATH'], + RESTIC_STAGE_DIR=str(stage),RESTIC_DB_CREDS_FILE=str(root/'absent'), + RESTIC_FV_CREDS_FILE=str(creds),FV_STUB_BODY=curl_body) + r=subprocess.run(['bash',str(SCRIPT)],env=env,capture_output=True,text=True) + cfg=stage/'fv-gateway-config.xml' + return {'rc':r.returncode,'out':r.stdout,'err':r.stderr, + 'staged':sorted(p.name for p in stage.iterdir()), + 'body':cfg.read_text() if cfg.exists() else None} + + def test_fv_gateway_config_is_staged_when_body_is_a_real_config(self): + g=self._fv('') + self.assertEqual(g['rc'],0,g['out']+g['err']) + self.assertIn('fv-gateway-config.xml',g['staged']) + self.assertIn('',g['body']) + + def test_fv_gateway_html_error_page_is_never_staged(self): + g=self._fv('401 Unauthorized') + self.assertEqual(g['rc'],0,g['out']+g['err']) # non-fatal by design + self.assertEqual(g['staged'],[]) + self.assertIsNone(g['body']) + self.assertIn('not an OPNsense config',g['err']) + + def test_fv_gateway_unreachable_is_non_fatal_and_stages_nothing(self): + g=self._fv('',curl_rc=7) + self.assertEqual(g['rc'],0,g['out']+g['err']) + self.assertEqual(g['staged'],[]) + self.assertIn('config pull failed',g['err']) + if __name__=='__main__':unittest.main() diff --git a/docs/pfi/opnsense-api-reference.md b/docs/pfi/opnsense-api-reference.md index c8d718a..f5062c7 100644 --- a/docs/pfi/opnsense-api-reference.md +++ b/docs/pfi/opnsense-api-reference.md @@ -42,6 +42,50 @@ writes config **and** applies it, which is normally the one you want after a — note **`disabled` ≠ `stopped`**: a disabled service will not start until its model's `enabled` field is set to `"1"`. +## ⚠ Endpoints are ACTIONS — never probe for existence by calling them + +A 404 tells you an endpoint is absent; a 200 tells you it **ran**. There is no +safe "does this exist?" POST against a live firewall. + +**2026-09-15:** looking for the call that applies a user change, a loop POSTed an +empty body at four guessed endpoints to see which returned 404. One of them was +`/api/core/system/reboot`. It returned 200 because it rebooted the FV edge +firewall, taking the whole site — including the BMC, which sits behind it — +dark for 3.5 minutes. The call it was actually looking for is documented +directly above, in § Service control, in this file. + +- Read this reference and the upstream endpoint list first. +- If you must discover, use **GET** on a `get`/`search`/`status` command, never + POST on an unknown name. +- Take `/api/core/backup/download/this` **before** any write. That part went + right and is the only reason the change was reversible. + +### The one useful thing that fell out of it + +`POST /api/core/system/reboot` with `{}` is a **reliable remote reboot** for the +FV gateway — it came back cleanly on its own in ~3.5 min from an API-initiated +restart, which is a capability worth knowing deliberately rather than by +accident. `/api/core/service/restart/` (e.g. `openssh`) restarts one service +without the site outage, and is almost always what you want instead. + +## Applying a change — `service/reconfigure`, not a reboot + +`settings/set` (and `auth/user/set`) write config.xml. They do **not** sync the +change to the running system. The apply step is the module's service endpoint: + +```sh +POST /api//service/reconfigure {} +``` + +⚠ Some modules have no `reconfigure` and return `{"errorMessage":"Endpoint not +found"}` — `auth/user` is one. For those the OS-level sync happens on the UI's +own save path or at boot, so an API-only key edit sits in config.xml and does +nothing until then. Verified 2026-09-15: `authorizedkeys` + `shell` for +`infra-ops` persisted immediately but SSH kept refusing, and started working +after a reboot completed the user sync. + +⚠ `POST` with **no body at all** returns `411 Length Required`. Send `{}`. + ## Field shapes — the part upstream does not document ⚠ **A `settings/get` response is NOT a valid `settings/set` body.** They are