memory: snapshot — FV recovered, fv-ml1 seat reorg, gen-large blocked on orca runtime

FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000,
GPU caps 275W/card). All-night fv-ml1 seat reorganization:

- flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1),
  inverting vLLM's 4xH100 recipe; KV 14->10 GiB.
- gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases
  repointed); 27B dense gen seat retired, 38 GB freed on GPU0.
- char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a
  leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier.
- Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7
  cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm).

gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded
(170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE;
the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell
image). Runtime decision pending -- this is the resume point.

Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift
alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad.

Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.

persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md

@@ -0,0 +1,97 @@
+# 2026-09-14 — fv-ml1 seat reorganization + the gen-large "orca" blocker
+
+An all-night GPU-seat overhaul on fv-ml1 after the FV colo recovered. Everything
+here is on the running box; regenerate the authoritative view with
+`scripts/seat-inventory.py` (reads the live containers). Cross-refs:
+[[2026-09-13-flash-next-seat-and-fv-outage]].
+
+## Seat topology now (2026-09-14 ~01:40 PT)
+
+| GPU | seats |
+|---|---|
+| 0 | `mog-sec` (`sec` :8019, dflash k=7) · **`sentinel-r3`** (`sentinel-r3` :8025, dflash k=7 — NEW) |
+| 1 | **`meromero-charrp`** (`char-rp` :8016, MeroMero-v2-31B dense — restored) · `erp-seat` (`char-rp-fast` :8021) · reward · coder · embed · rerank |
+| 2 | `flash-next` (`gen-large` :8022) — **DOWN** (orca swap failed; dealignai stopped) |
+| 3 | **RESERVED scratch** — empty, operator directive; benches/quants/probes only |
+
+## What changed tonight
+
+1. **flash-next gained MTP k=3.** Campaign in `services/flash-next-mtp-bench/`
+   measured MTP a WIN on this hardware (+29/41/27% at k1, +42/52/38% k2, +52/51/34%
+   k3 across conc 1/4/8), inverting vLLM's 4×H100 recipe. k=3 deployed (single-user
+   → conc=1 dominates). KV pinned 14→10 GiB (MTP adds 5.08 GiB draft-head weights;
+   14 GiB OOMs). Warm decode ~121 tok/s. `stacks/flash-next-seat/compose.yaml`.
+2. **gen consolidation.** All 8 `gen`/`gen-reasoning`/`summarizer`/`summarizer-large`/
+   `classifier`/`chat-judge`/`image-judge`/`qwen-image-bench` LiteLLM entries repointed
+   to flash-next (:8022); the **27B dense gen seat RETIRED**, freeing 38.4 GB on GPU0.
+   ⚠ judge aliases now score against different weights — prior scores incomparable.
+3. **char-rp restored to MeroMero-v2-31B.** Was serving a leftover-test RedHatAI 26B
+   MoE W4A4; operator wanted the in-house dense-31B heretic W4A16 back. `char-rp-fast`
+   (:8021, the 26B MoE) is the deliberate speed tier — the throughput answer, so
+   char-rp stays the quality seat. `stacks/meromero-charrp/`.
+4. **Sentinel-R3 A/B + dflash.** `glyphsoftware/sentinel-r3` (proprietary license —
+   operator's call) is a REAL SFT pentest finetune vs mog-sec's persona-on-stock.
+   Served alongside mog-sec for A/B. Then measured dflash vs MTP on Sentinel's
+   finetuned body: **dflash 2.40 vs MTP 2.18 mean acceptance length (+11%)**, warm
+   decode ~121 tok/s (faster than sec ~102). dflash k=7 cut over. `stacks/sentinel-r3/`.
+   ⚠ MEASUREMENT LESSON (again): first decode bench read 39 tok/s — a COLD-boot +
+   concurrent-contention artifact; warm+isolated it was 121. Operator caught it by
+   testing the running `sec` (102 tok/s) as reference.
+
+## THE BLOCKER — gen-large "orca" swap (resume here)
+
+Operator wants gen-large on **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`**
+(weight-only compressed-tensors: NVFP4 experts + FP8 attn + bf16 PLE) instead of the
+incumbent **dealignai** build (modelopt_fp4 **W4A4** — 4-bit activations, the
+long-context degradation mode). orcarouter is the trusted author (= `gen`), better on
+the activation axis. Downloaded to `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4`
+(170 GB, complete, 18 shards + MTP head, integrity-verified).
+
+**IT DOES NOT LOAD.** Root cause (read the loader source, definitive):
+`Qwen4ExpPLEEmbeddingMethod.from_quant_config` in vLLM — even the LATEST mainline
+nightly — has branches only for **ModelOpt** and **FP8** configs; for a
+CompressedTensorsConfig it raises `NotImplementedError` BEFORE ever consulting the
+ignore list. So:
+- dealignai (modelopt_fp4) loads fine; orcarouter (compressed-tensors) cannot, on any
+  mainline vLLM.
+- orcarouter's config also does NOT list its `ngram_embedding` in `ignore` (would need
+  adding) — but that's moot until the loader has a compressed-tensors branch.
+- orcarouter's OWN card prescribes **TP=4 + `--enable-expert-parallel`** and
+  `vllm>=<qwen4exp-supporting release>` — a LITERAL UNFILLED version placeholder. They
+  shipped a build whose runtime they couldn't pin.
+
+**Backport vetted → UNFIT.** `wtdcode/vllm-backport` (docker `lazymio/vllm-backport`,
+212★, Apache-2.0, DCO, active) resolved this exact error (issue #72) BUT: its purpose is
+OLD GPUs (images sm86/sm80/sm89 only — **no Blackwell/sm_120**), it's built around its
+OWN `wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16` quant not orcarouter's NVFP4, and Blackwell
+has native FP4/FP8 so the whole backport premise is moot for us. Using it = source-build
+for an untested arch to load a quant it never tested.
+
+**Options (operator decision pending):**
+- (a) **Restore dealignai** — works today, identical architecture, ~90% of the quality
+  (loses only the weight-only-vs-W4A4 activation edge). The pragmatic interim.
+- (b) Cherry-pick the ~10-line compressed-tensors PLE-loader branch onto a mainline
+  source build for Blackwell (mainline already has Blackwell FP4 compute). More
+  auditable than the backport, still a from-source build + maintenance.
+- (c) Request orcarouter's own **W4A4** build (they offer it) — would load on the current
+  mainline image like dealignai.
+- (d) Wait for mainline vLLM to merge compressed-tensors qwen4_exp.
+
+⚠ Operator said "don't reload dealignai unprompted, priority orca" — but orca needs
+runtime work, so gen-large is intentionally DOWN pending the decision. Both configs
+backed up on host: `compose.yaml.bak-20260914-orcarouter`, `.env.bak-20260914-orcarouter`.
+
+## Other open items
+
+- **cyberprev quant** (3rd sec candidate `hotdogs/Qwen3.8-27B-abliterated-cyber-preview`,
+  bf16 at `/tank/aimodels/cyberprev-bf16`): restart crashed on a transformers
+  head-count config error in `quant-work/.venv` — the SAME venv that reached 49/65 on
+  Sept 11, so the original run likely used the canonical vllm-image+llmcompressor path
+  (per `services/gen-seat-mixed-quant/README.md`), not the venv. Retry that way.
+- **sec vs sentinel-r3 quality A/B** — both live and gateway-callable; operator to judge.
+- ⚠ **Suspect vault entry**: `fv-gateway/infra-ops-password` is 16 chars matching a
+  boot-UUID prefix exactly — possibly malformed. Eyeball.
+- **os-nut not installed** on the FV OPNsense — with the firewall now on the 5P1000 UPS,
+  a NUT transfer-to-battery event is a direct "breaker tripped" alarm nobody gets today.
+- **Branch breaker rating + 4-card ammeter reading** still open — every power table is
+  arithmetic on an ESTIMATED ~300 W platform draw. The ammeter converts it to fact.