Files

T

vh 838132cd6b memory: snapshot — FV cross-site routing fixed, fleet conventions pinned

Session captured: the FV outbound-NAT root cause and its diagnostic signature,
the fv-ml1 dead man's switch, fleet identity/group/path conventions and the
root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome
modernisation and the kb KB-search tool, and the Hermes bearer rotation
release. Six new detail files.

Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them
(which rebooted the FV firewall), advertising a /32 from nh3-dev, and the
nh3-scale remote-site masquerade rules that fired but were not the fix.

Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21
oversized inline entries split into detail files per the two-tier rule -- they
had been sitting fully inline in the index, which is what the split exists to
prevent. Two pointers to a detail file archived this run were repointed at
archival-memory.md.

The index is 389 lines, still over the ~300 soft cap. The archival guards stop
it there: only 4 further entries are old enough to move and every one carries an
open deferred-work pointer. An over-cap file that keeps live decisions beats a
scannable one that lost a deferred call.

2026-09-15 00:53:48 -07:00

2.4 KiB

Raw Blame History

`[2026-09-15]` I rebooted the FV edge firewall by probing API endpoints

Looking for the call that applies an OPNsense user change, I POSTed an empty body at four guessed endpoints to see which returned 404. One of them was /api/core/system/reboot. It returned 200 because it ran. The whole FV site — including the BMC, which sits behind that gateway — went dark for 3.5 minutes.

⭐ The call I was looking for is documented in this repo, in docs/pfi/opnsense-api-reference.md § Service control: "reconfigure writes config and applies it, which is normally the one you want after a settings/set." I had opened that file twice and read around it.

The rule

Endpoints are ACTIONS. A 404 tells you an endpoint is absent; a 200 tells you it ran. There is no safe "does this exist?" POST against a live firewall. Read the reference first; if you must discover, use GET on a get/search/status command, never POST on an unknown name.

Compounding failures worth naming separately

I kept polling FV afterwards — its own runbook (fv-site-dark-20260913.md) says in the header "Do not leave watchers running against FV addresses."
⚠⚠ I reported the site still dark while holding, unread, the file that said it was up. My own background watcher had logged WAN admin: 200 / gateway OK / fv-ml1 OK / ssh ALIVE at ~204 s. The operator was weighing a midnight drive against an outage that had already ended. Actual outage 3.5 min; I reported ~15.

The one useful thing that fell out

POST /api/core/system/reboot with {} is a reliable remote reboot for the FV gateway — it came back cleanly on its own, which is a capability worth having deliberately rather than by accident. /api/core/service/restart/<id> restarts one service without the site outage and is almost always what you want instead.

Also learned on the OPNsense API

auth/user has no reconfigure; an API-only key edit persists in config.xml and does nothing until the OS user sync runs at boot. Verified: authorizedkeys + shell for infra-ops persisted immediately, SSH kept refusing, and started working after the reboot.
POST with no body at all returns 411 Length Required. Send {}.
Outbound-NAT rules written with write_config are invisible to source_nat/search_rule. See 2026-09-15-fv-cross-site-snat.

2.4 KiB Raw Blame History

[2026-09-15] I rebooted the FV edge firewall by probing API endpoints