memory: snapshot — FV recovered, fv-ml1 seat reorg, gen-large blocked on orca runtime

FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000,
GPU caps 275W/card). All-night fv-ml1 seat reorganization:

- flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1),
  inverting vLLM's 4xH100 recipe; KV 14->10 GiB.
- gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases
  repointed); 27B dense gen seat retired, 38 GB freed on GPU0.
- char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a
  leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier.
- Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7
  cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm).

gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded
(170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE;
the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell
image). Runtime decision pending -- this is the resume point.

Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift
alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad.

Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.

docs/pfi/opnsense-api-reference.md

@@ -0,0 +1,104 @@
+# OPNsense API — vendored reference + field-shape notes
+
+**Source:** <https://docs.opnsense.org/development/api.html> (fetched 2026-09-13)
+**Why vendored:** the upstream page documents the REST conventions but explicitly
+*not* parameter shapes — "the auto-generated API documentation captures endpoints
+and HTTP methods but not all parameter details." Those shapes are where the time
+goes, so the hard-won ones are recorded in § Field shapes below.
+
+## Authentication
+
+HTTP Basic auth: **key = username, secret = password**.
+
+```sh
+curl -s -u "$KEY:$SECRET" "http://<gw>/api/<module>/<controller>/<command>"
+```
+
+Fleet creds are vaulted: `fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`
+(see `secret get`). Authorization is per-user **Effective Privileges** — a key only
+reaches endpoints its owner is authorized for.
+
+## Conventions
+
+- **Endpoint pattern:** `/api/<module>/<controller>/<command>/[<param>/...]`
+- **GET** retrieves; **POST** creates, updates, or executes an action.
+- Request bodies and responses are `application/json`.
+
+Search endpoints take a paging body and return a rows envelope:
+
+```json
+{"current":1,"rowCount":7,"sort":{},"searchPhrase":""}
+```
+```json
+{"total":10,"rowCount":7,"current":1,
+ "rows":[{"id":"configd","locked":1,"running":1,"description":"System Configuration Daemon","name":"configd"}]}
+```
+
+## Service control
+
+`/api/<module>/service/{status,start,stop,restart,reconfigure}` — `reconfigure`
+writes config **and** applies it, which is normally the one you want after a
+`settings/set`. Observed status values include `disabled`, `stopped`, `running`
+— note **`disabled` ≠ `stopped`**: a disabled service will not start until its
+model's `enabled` field is set to `"1"`.
+
+## Field shapes — the part upstream does not document
+
+⚠ **A `settings/get` response is NOT a valid `settings/set` body.** They are
+different shapes, and mixing them returns a bare
+`{"errorMessage":"Unexpected error, check log for details"}` with no indication
+of which field was wrong.
+
+**Selection / option fields.** `get` returns every option with a `selected` flag;
+`set` wants only the selected key.
+
+```jsonc
+// get
+"useExitNode": {"": {"value": "None", "selected": 1}, "abc": {"value":"node-a","selected":0}}
+// set
+"useExitNode": ""
+```
+
+**Array / list fields are UUID-keyed objects, not lists.** This is the one that
+costs an afternoon. `get` renders an empty array as `[]`, which misleads you into
+POSTing a list.
+
+```jsonc
+// get, when empty  -> looks like a plain list
+"subnets": {"subnet4": []}
+// set  -> object keyed by a UUID you generate, values are the item's own fields
+"subnets": {"subnet4": {"<uuid4>": {"subnet": "10.251.0.0/16", "description": "..."}}}
+```
+
+**Discovering an array item's inner fields — two reliable tricks:**
+
+1. POST the wrong shape on purpose. Validation names the full path:
+   `{"result":"failed","validations":{"settings.subnets.subnet4.<uuid>.subnet":"A value is required."}}`
+2. Many models expose a template getter: `/api/<module>/settings/get_<item>`
+   returns the blank item, e.g. `get_subnet` → `{"subnet4":{"subnet":"","description":""}}`
+
+**Partial `set` is supported** — POST only the fields you are changing rather than
+round-tripping the whole model. This sidesteps every other field's shape problem
+and is the recommended approach for a surgical change.
+
+**Validation error format:** `{"result":"failed","validations":{"<dotted.path>":"<message>"}}`.
+A successful save is `{"result":"saved"}`.
+
+## Working example — enabling a subnet router (2026-09-13, FV gateway)
+
+```jsonc
+POST /api/tailscale/settings/set
+{"settings":{"enabled":"1",
+  "subnets":{"subnet4":{"<uuid4>":{"subnet":"10.251.0.0/16",
+                                   "description":"FV site subnet router"}}}}}
+POST /api/tailscale/service/reconfigure   {}
+```
+
+**Always read back.** `settings/get` after the write, and re-check anything the
+change could disturb (here: `/api/firewall/source_nat/search_rule` total stayed
+at 2). A write that reports success is not evidence the value landed.
+
+## Related
+
+- `docs/runbooks/fv-to-ana-nat.md` — the FV gateway's manual SNAT rule
+- `docs/runbooks/fv-site-dark-20260913.md` — the outage this reference came out of

persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md

@@ -0,0 +1,97 @@
+# 2026-09-14 — fv-ml1 seat reorganization + the gen-large "orca" blocker
+
+An all-night GPU-seat overhaul on fv-ml1 after the FV colo recovered. Everything
+here is on the running box; regenerate the authoritative view with
+`scripts/seat-inventory.py` (reads the live containers). Cross-refs:
+[[2026-09-13-flash-next-seat-and-fv-outage]].
+
+## Seat topology now (2026-09-14 ~01:40 PT)
+
+| GPU | seats |
+|---|---|
+| 0 | `mog-sec` (`sec` :8019, dflash k=7) · **`sentinel-r3`** (`sentinel-r3` :8025, dflash k=7 — NEW) |
+| 1 | **`meromero-charrp`** (`char-rp` :8016, MeroMero-v2-31B dense — restored) · `erp-seat` (`char-rp-fast` :8021) · reward · coder · embed · rerank |
+| 2 | `flash-next` (`gen-large` :8022) — **DOWN** (orca swap failed; dealignai stopped) |
+| 3 | **RESERVED scratch** — empty, operator directive; benches/quants/probes only |
+
+## What changed tonight
+
+1. **flash-next gained MTP k=3.** Campaign in `services/flash-next-mtp-bench/`
+   measured MTP a WIN on this hardware (+29/41/27% at k1, +42/52/38% k2, +52/51/34%
+   k3 across conc 1/4/8), inverting vLLM's 4×H100 recipe. k=3 deployed (single-user
+   → conc=1 dominates). KV pinned 14→10 GiB (MTP adds 5.08 GiB draft-head weights;
+   14 GiB OOMs). Warm decode ~121 tok/s. `stacks/flash-next-seat/compose.yaml`.
+2. **gen consolidation.** All 8 `gen`/`gen-reasoning`/`summarizer`/`summarizer-large`/
+   `classifier`/`chat-judge`/`image-judge`/`qwen-image-bench` LiteLLM entries repointed
+   to flash-next (:8022); the **27B dense gen seat RETIRED**, freeing 38.4 GB on GPU0.
+   ⚠ judge aliases now score against different weights — prior scores incomparable.
+3. **char-rp restored to MeroMero-v2-31B.** Was serving a leftover-test RedHatAI 26B
+   MoE W4A4; operator wanted the in-house dense-31B heretic W4A16 back. `char-rp-fast`
+   (:8021, the 26B MoE) is the deliberate speed tier — the throughput answer, so
+   char-rp stays the quality seat. `stacks/meromero-charrp/`.
+4. **Sentinel-R3 A/B + dflash.** `glyphsoftware/sentinel-r3` (proprietary license —
+   operator's call) is a REAL SFT pentest finetune vs mog-sec's persona-on-stock.
+   Served alongside mog-sec for A/B. Then measured dflash vs MTP on Sentinel's
+   finetuned body: **dflash 2.40 vs MTP 2.18 mean acceptance length (+11%)**, warm
+   decode ~121 tok/s (faster than sec ~102). dflash k=7 cut over. `stacks/sentinel-r3/`.
+   ⚠ MEASUREMENT LESSON (again): first decode bench read 39 tok/s — a COLD-boot +
+   concurrent-contention artifact; warm+isolated it was 121. Operator caught it by
+   testing the running `sec` (102 tok/s) as reference.
+
+## THE BLOCKER — gen-large "orca" swap (resume here)
+
+Operator wants gen-large on **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`**
+(weight-only compressed-tensors: NVFP4 experts + FP8 attn + bf16 PLE) instead of the
+incumbent **dealignai** build (modelopt_fp4 **W4A4** — 4-bit activations, the
+long-context degradation mode). orcarouter is the trusted author (= `gen`), better on
+the activation axis. Downloaded to `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4`
+(170 GB, complete, 18 shards + MTP head, integrity-verified).
+
+**IT DOES NOT LOAD.** Root cause (read the loader source, definitive):
+`Qwen4ExpPLEEmbeddingMethod.from_quant_config` in vLLM — even the LATEST mainline
+nightly — has branches only for **ModelOpt** and **FP8** configs; for a
+CompressedTensorsConfig it raises `NotImplementedError` BEFORE ever consulting the
+ignore list. So:
+- dealignai (modelopt_fp4) loads fine; orcarouter (compressed-tensors) cannot, on any
+  mainline vLLM.
+- orcarouter's config also does NOT list its `ngram_embedding` in `ignore` (would need
+  adding) — but that's moot until the loader has a compressed-tensors branch.
+- orcarouter's OWN card prescribes **TP=4 + `--enable-expert-parallel`** and
+  `vllm>=<qwen4exp-supporting release>` — a LITERAL UNFILLED version placeholder. They
+  shipped a build whose runtime they couldn't pin.
+
+**Backport vetted → UNFIT.** `wtdcode/vllm-backport` (docker `lazymio/vllm-backport`,
+212★, Apache-2.0, DCO, active) resolved this exact error (issue #72) BUT: its purpose is
+OLD GPUs (images sm86/sm80/sm89 only — **no Blackwell/sm_120**), it's built around its
+OWN `wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16` quant not orcarouter's NVFP4, and Blackwell
+has native FP4/FP8 so the whole backport premise is moot for us. Using it = source-build
+for an untested arch to load a quant it never tested.
+
+**Options (operator decision pending):**
+- (a) **Restore dealignai** — works today, identical architecture, ~90% of the quality
+  (loses only the weight-only-vs-W4A4 activation edge). The pragmatic interim.
+- (b) Cherry-pick the ~10-line compressed-tensors PLE-loader branch onto a mainline
+  source build for Blackwell (mainline already has Blackwell FP4 compute). More
+  auditable than the backport, still a from-source build + maintenance.
+- (c) Request orcarouter's own **W4A4** build (they offer it) — would load on the current
+  mainline image like dealignai.
+- (d) Wait for mainline vLLM to merge compressed-tensors qwen4_exp.
+
+⚠ Operator said "don't reload dealignai unprompted, priority orca" — but orca needs
+runtime work, so gen-large is intentionally DOWN pending the decision. Both configs
+backed up on host: `compose.yaml.bak-20260914-orcarouter`, `.env.bak-20260914-orcarouter`.
+
+## Other open items
+
+- **cyberprev quant** (3rd sec candidate `hotdogs/Qwen3.8-27B-abliterated-cyber-preview`,
+  bf16 at `/tank/aimodels/cyberprev-bf16`): restart crashed on a transformers
+  head-count config error in `quant-work/.venv` — the SAME venv that reached 49/65 on
+  Sept 11, so the original run likely used the canonical vllm-image+llmcompressor path
+  (per `services/gen-seat-mixed-quant/README.md`), not the venv. Retry that way.
+- **sec vs sentinel-r3 quality A/B** — both live and gateway-callable; operator to judge.
+- ⚠ **Suspect vault entry**: `fv-gateway/infra-ops-password` is 16 chars matching a
+  boot-UUID prefix exactly — possibly malformed. Eyeball.
+- **os-nut not installed** on the FV OPNsense — with the firewall now on the 5P1000 UPS,
+  a NUT transfer-to-battery event is a direct "breaker tripped" alarm nobody gets today.
+- **Branch breaker rating + 4-card ammeter reading** still open — every power table is
+  arithmetic on an ESTIMATED ~300 W platform draw. The ammeter converts it to fact.

stacks/sentinel-r3/.env.example

@@ -0,0 +1,10 @@
+SENTINEL_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013
+API_KEY=
+SENTINEL_GPU_ID=0
+SENTINEL_PORT=8025
+SENTINEL_MODEL=/tank/aimodels/sentinel-r3-nvfp4-mixed
+SENTINEL_GPU_MEM_UTIL=0.40
+SENTINEL_MAX_MODEL_LEN=163840
+SENTINEL_KV_CACHE_MEMORY=8589934592
+SENTINEL_MAX_NUM_BATCHED_TOKENS=4096
+SENTINEL_DRAFT_MODEL=/tank/aimodels/qwen38-27b-dflash2-drafter

stacks/sentinel-r3/compose.yaml

@@ -0,0 +1,124 @@
+# sentinel-r3 — Sentinel-R3 pen-test seat, A/B candidate ALONGSIDE mog-sec on fv-ml1 GPU 0.
+#
+# glyphsoftware/sentinel-r3: a REAL SFT finetune of stock Qwen/Qwen3.8-27B on 1,230
+# authorized-pentest agent trajectories (recon -> foothold -> privesc -> writeup) over a
+# 19-tool surface that matches our own harness. Contrast mog-sec, which is a persona
+# system prompt on stock weights. Quantized in-house to the same mixed NVFP4 W4A4(MLP
+# 0-55) + FP8 W8A8(attn/lm_head/MLP 56-63) recipe as mog-sec/gen. → sentinel-r3-nvfp4-mixed.PROVENANCE.txt
+#
+# ⚠ PROPRIETARY LICENSE (Glyph Proprietary v1.0) — operator's fair-use/licensee call, unlike
+#   mog-sec's Apache. Served here on operator instruction 2026-09-14.
+#
+# ⚠ dflash speculative decoding ENABLED 2026-09-14 after measurement (see the spec-config note
+#   below). Originally served without spec to establish a clean baseline; the probe measured
+#   dflash k=7 at 2.40 acceptance length vs MTP k=3 at 2.18 on this finetuned body.
+# (historical) SERVED WITHOUT --speculative-config ON PURPOSE. The MTP head is a VERBATIM base graft
+#   (source shipped zero mtp.*), and its acceptance on this SFT-finetuned body is UNVERIFIED —
+#   the gate is a measured >=~40% on a probe serve, not an assumption. A clean no-spec boot
+#   establishes the quality baseline for the mog-sec A/B first; MTP acceptance is a separate
+#   measurement (add the qwen3_5_mtp spec-config and read the acceptance metric).
+#
+# ⚠ max-model-len 163840, NOT native 262K. Sentinel is the SAME base + hybrid Qwen3_5 arch as
+#   mog-sec, so the identical deep-context lesson applies: what the KV pool HOLDS and what the
+#   card PROCESSES at depth are different numbers, and mog-sec crashed five times before 163840
+#   bought a clean 400-refusal above the measured ceiling instead of an engine death. Do not
+#   raise without re-running the deep-ctx probe on THIS checkpoint.
+#
+# ⚠ Serve with a PROSE system prompt — Sentinel was trained on prose tool descriptions, not
+#   structured `tools=`.
+
+name: sentinel-r3
+
+services:
+  vllm-sentinel-r3:
+    image: ${SENTINEL_IMAGE:-vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013}
+    container_name: ${SENTINEL_CONTAINER_NAME:-vllm-sentinel-r3}
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${SENTINEL_PORT:-8025}:8000"
+    volumes:
+      - /tank/aimodels/huggingface:/hfcache
+      - ${SENTINEL_MODEL:-/tank/aimodels/sentinel-r3-nvfp4-mixed}:/model:ro
+      - ${SENTINEL_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro
+    environment:
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - /model
+      - --served-model-name
+      - ${SENTINEL_SERVED_NAME:-sentinel-r3}
+      - ${SENTINEL_SERVED_NAME_THINK:-sentinel-r3-thinking}
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --quantization
+      - ${SENTINEL_QUANT:-compressed-tensors}
+      - --gpu-memory-utilization
+      - "${SENTINEL_GPU_MEM_UTIL:-0.40}"
+      # KV pinned in bytes — same discipline as mog-sec/erp-seat: a ratio yields a different
+      # cache depending on what else is resident at start, an explicit figure is reproducible.
+      # 8 GiB is generous for an A/B probe (conc 1/4/8 short prompts never approach it).
+      - --kv-cache-memory
+      - "${SENTINEL_KV_CACHE_MEMORY:-8589934592}"
+      - --max-model-len
+      - "${SENTINEL_MAX_MODEL_LEN:-163840}"
+      - --max-num-seqs
+      - "${SENTINEL_MAX_NUM_SEQS:-16}"
+      - --max-num-batched-tokens
+      - "${SENTINEL_MAX_NUM_BATCHED_TOKENS:-4096}"
+      - --trust-remote-code
+      - --dtype
+      - auto
+      - --mamba-cache-dtype
+      - float32
+      - --kv-cache-dtype
+      - ${SENTINEL_KV_CACHE_DTYPE:-fp8}
+      - --enable-prefix-caching
+      - --enable-chunked-prefill
+      - --limit-mm-per-prompt
+      - '${SENTINEL_LIMIT_MM:-{"image": 4}}'
+      - --mm-processor-kwargs
+      - '${SENTINEL_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
+      - --reasoning-parser
+      - ${SENTINEL_REASONING_PARSER:-qwen3}
+      - --default-chat-template-kwargs
+      - '{"reasoning_effort": "${SENTINEL_REASONING_EFFORT:-medium}"}'
+      - --enable-auto-tool-choice
+      - --tool-call-parser
+      - qwen3_coder
+      # dflash speculative decoding — MEASURED 2026-09-14 on THIS finetuned body: dflash k=7
+      # gives mean acceptance length 2.40 vs MTP k=3 at 2.18 (+11%, clean n=2 separation).
+      # dflash wins on reach (accepts a longer tail to k=7) despite equal ~60% position-1
+      # acceptance. The drafter is the same qwen38-27b-dflash2-drafter mog-sec uses (Sentinel
+      # shares its 64-layer Qwen3.8-27B base). Costs +3.6 GB for the drafter.
+      - --speculative-config
+      - '${SENTINEL_SPEC_CONFIG:-{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}}'
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${SENTINEL_GPU_ID:-0}"
+              capabilities:
+                - gpu
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 900s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Inference
+      - homepage.name=Sentinel-R3 (pen-test A/B)
+      - homepage.icon=mdi-shield-search
+      - homepage.description=SFT pen-test finetune of Qwen3.8-27B, A/B candidate vs mog-sec (fv-ml1 GPU 0)
+      - homepage.href=http://10.251.50.54:${SENTINEL_PORT:-8025}/docs
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true