memory: snapshot — FV recovered, fv-ml1 seat reorg, gen-large blocked on orca runtime

FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000, GPU caps 275W/card). All-night fv-ml1 seat reorganization: - flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1), inverting vLLM's 4xH100 recipe; KV 14->10 GiB. - gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases repointed); 27B dense gen seat retired, 38 GB freed on GPU0. - char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier. - Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7 cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm). gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded (170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE; the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell image). Runtime decision pending -- this is the resume point. Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad. Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.
2026-09-14 01:57:10 -07:00
parent c2a70c13db
commit ed3f1ede3b
5 changed files with 376 additions and 96 deletions
@@ -0,0 +1,104 @@
+# OPNsense API — vendored reference + field-shape notes
+
+**Source:** <https://docs.opnsense.org/development/api.html> (fetched 2026-09-13)
+**Why vendored:** the upstream page documents the REST conventions but explicitly
+*not* parameter shapes — "the auto-generated API documentation captures endpoints
+and HTTP methods but not all parameter details." Those shapes are where the time
+goes, so the hard-won ones are recorded in § Field shapes below.
+
+## Authentication
+
+HTTP Basic auth: **key = username, secret = password**.
+
+```sh
+curl -s -u "$KEY:$SECRET" "http://<gw>/api/<module>/<controller>/<command>"
+```
+
+Fleet creds are vaulted: `fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`
+(see `secret get`). Authorization is per-user **Effective Privileges** — a key only
+reaches endpoints its owner is authorized for.
+
+## Conventions
+
+- **Endpoint pattern:** `/api/<module>/<controller>/<command>/[<param>/...]`
+- **GET** retrieves; **POST** creates, updates, or executes an action.
+- Request bodies and responses are `application/json`.
+
+Search endpoints take a paging body and return a rows envelope:
+
+```json
+{"current":1,"rowCount":7,"sort":{},"searchPhrase":""}
+```
+```json
+{"total":10,"rowCount":7,"current":1,
+ "rows":[{"id":"configd","locked":1,"running":1,"description":"System Configuration Daemon","name":"configd"}]}
+```
+
+## Service control
+
+`/api/<module>/service/{status,start,stop,restart,reconfigure}` — `reconfigure`
+writes config **and** applies it, which is normally the one you want after a
+`settings/set`. Observed status values include `disabled`, `stopped`, `running`
+— note **`disabled` ≠ `stopped`**: a disabled service will not start until its
+model's `enabled` field is set to `"1"`.
+
+## Field shapes — the part upstream does not document
+
+⚠ **A `settings/get` response is NOT a valid `settings/set` body.** They are
+different shapes, and mixing them returns a bare
+`{"errorMessage":"Unexpected error, check log for details"}` with no indication
+of which field was wrong.
+
+**Selection / option fields.** `get` returns every option with a `selected` flag;
+`set` wants only the selected key.
+
+```jsonc
+// get
+"useExitNode": {"": {"value": "None", "selected": 1}, "abc": {"value":"node-a","selected":0}}
+// set
+"useExitNode": ""
+```
+
+**Array / list fields are UUID-keyed objects, not lists.** This is the one that
+costs an afternoon. `get` renders an empty array as `[]`, which misleads you into
+POSTing a list.
+
+```jsonc
+// get, when empty  -> looks like a plain list
+"subnets": {"subnet4": []}
+// set  -> object keyed by a UUID you generate, values are the item's own fields
+"subnets": {"subnet4": {"<uuid4>": {"subnet": "10.251.0.0/16", "description": "..."}}}
+```
+
+**Discovering an array item's inner fields — two reliable tricks:**
+
+1. POST the wrong shape on purpose. Validation names the full path:
+   `{"result":"failed","validations":{"settings.subnets.subnet4.<uuid>.subnet":"A value is required."}}`
+2. Many models expose a template getter: `/api/<module>/settings/get_<item>`
+   returns the blank item, e.g. `get_subnet` → `{"subnet4":{"subnet":"","description":""}}`
+
+**Partial `set` is supported** — POST only the fields you are changing rather than
+round-tripping the whole model. This sidesteps every other field's shape problem
+and is the recommended approach for a surgical change.
+
+**Validation error format:** `{"result":"failed","validations":{"<dotted.path>":"<message>"}}`.
+A successful save is `{"result":"saved"}`.
+
+## Working example — enabling a subnet router (2026-09-13, FV gateway)
+
+```jsonc
+POST /api/tailscale/settings/set
+{"settings":{"enabled":"1",
+  "subnets":{"subnet4":{"<uuid4>":{"subnet":"10.251.0.0/16",
+                                   "description":"FV site subnet router"}}}}}
+POST /api/tailscale/service/reconfigure   {}
+```
+
+**Always read back.** `settings/get` after the write, and re-check anything the
+change could disturb (here: `/api/firewall/source_nat/search_rule` total stayed
+at 2). A write that reports success is not evidence the value landed.
+
+## Related
+
+- `docs/runbooks/fv-to-ana-nat.md` — the FV gateway's manual SNAT rule
+- `docs/runbooks/fv-site-dark-20260913.md` — the outage this reference came out of