vh
838132cd6b
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call.
48 lines
2.4 KiB
Markdown
48 lines
2.4 KiB
Markdown
# `[2026-09-15]` I rebooted the FV edge firewall by probing API endpoints
|
|
|
|
Looking for the call that applies an OPNsense user change, I POSTed an empty
|
|
body at four **guessed** endpoints to see which returned 404. One of them was
|
|
`/api/core/system/reboot`. It returned 200 because it **ran**. The whole FV site
|
|
— including the BMC, which sits behind that gateway — went dark for **3.5
|
|
minutes**.
|
|
|
|
⭐ **The call I was looking for is documented in this repo**, in
|
|
`docs/pfi/opnsense-api-reference.md` § Service control: *"`reconfigure` writes
|
|
config and applies it, which is normally the one you want after a
|
|
`settings/set`."* I had opened that file twice and read around it.
|
|
|
|
## The rule
|
|
|
|
**Endpoints are ACTIONS.** A 404 tells you an endpoint is absent; a 200 tells
|
|
you it ran. There is no safe "does this exist?" POST against a live firewall.
|
|
Read the reference first; if you must discover, use **GET** on a
|
|
`get`/`search`/`status` command, never POST on an unknown name.
|
|
|
|
## Compounding failures worth naming separately
|
|
|
|
- **I kept polling FV afterwards** — its own runbook
|
|
(`fv-site-dark-20260913.md`) says in the header *"Do not leave watchers
|
|
running against FV addresses."*
|
|
- ⚠⚠ **I reported the site still dark while holding, unread, the file that said
|
|
it was up.** My own background watcher had logged
|
|
`WAN admin: 200 / gateway OK / fv-ml1 OK / ssh ALIVE` at ~204 s. The operator
|
|
was weighing a midnight drive against an outage that had already ended.
|
|
Actual outage 3.5 min; I reported ~15.
|
|
|
|
## The one useful thing that fell out
|
|
|
|
`POST /api/core/system/reboot` with `{}` is a **reliable remote reboot** for the
|
|
FV gateway — it came back cleanly on its own, which is a capability worth having
|
|
deliberately rather than by accident. `/api/core/service/restart/<id>` restarts
|
|
one service without the site outage and is almost always what you want instead.
|
|
|
|
## Also learned on the OPNsense API
|
|
|
|
- `auth/user` has **no** `reconfigure`; an API-only key edit persists in
|
|
`config.xml` and does nothing until the OS user sync runs at boot. Verified:
|
|
`authorizedkeys` + `shell` for `infra-ops` persisted immediately, SSH kept
|
|
refusing, and started working after the reboot.
|
|
- `POST` with **no body at all** returns `411 Length Required`. Send `{}`.
|
|
- Outbound-NAT rules written with `write_config` are **invisible** to
|
|
`source_nat/search_rule`. See [[2026-09-15-fv-cross-site-snat]].
|