esh-pfi-infrastructure/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md

2026-09-14 — fv-ml1 seat reorganization + the gen-large "orca" blocker

An all-night GPU-seat overhaul on fv-ml1 after the FV colo recovered. Everything here is on the running box; regenerate the authoritative view with scripts/seat-inventory.py (reads the live containers). Cross-refs: 2026-09-13-flash-next-seat-and-fv-outage.

Seat topology now (2026-09-14 ~01:40 PT)

GPU	seats
0	mog-sec (sec :8019, dflash k=7) · sentinel-r3 (sentinel-r3 :8025, dflash k=7 — NEW)
1	meromero-charrp (char-rp :8016, MeroMero-v2-31B dense — restored) · erp-seat (char-rp-fast :8021) · reward · coder · embed · rerank
2	flash-next (gen-large :8022) — UP on orcarouter (PLE converted bf16→FP8; MTP k=3, 60.4% accept)
3	RESERVED scratch — empty, operator directive; benches/quants/probes only

What changed tonight

1. flash-next gained MTP k=3. Campaign in services/flash-next-mtp-bench/ measured MTP a WIN on this hardware (+29/41/27% at k1, +42/52/38% k2, +52/51/34% k3 across conc 1/4/8), inverting vLLM's 4×H100 recipe. k=3 deployed (single-user → conc=1 dominates). KV pinned 14→10 GiB (MTP adds 5.08 GiB draft-head weights; 14 GiB OOMs). Warm decode ~121 tok/s. stacks/flash-next-seat/compose.yaml.
2. gen consolidation. All 8 gen/gen-reasoning/summarizer/summarizer-large/ classifier/chat-judge/image-judge/qwen-image-bench LiteLLM entries repointed to flash-next (:8022); the 27B dense gen seat RETIRED, freeing 38.4 GB on GPU0. ⚠ judge aliases now score against different weights — prior scores incomparable.
3. char-rp restored to MeroMero-v2-31B. Was serving a leftover-test RedHatAI 26B MoE W4A4; operator wanted the in-house dense-31B heretic W4A16 back. char-rp-fast (:8021, the 26B MoE) is the deliberate speed tier — the throughput answer, so char-rp stays the quality seat. stacks/meromero-charrp/.
4. Sentinel-R3 A/B + dflash. glyphsoftware/sentinel-r3 (proprietary license — operator's call) is a REAL SFT pentest finetune vs mog-sec (M.O.G.-SEC), which is ALSO a finetune — a refusal-free offense+defense cyber SFT, NOT a persona-on-stock as earlier notes claimed. All three sec seats are Qwen3.8-27B finetunes; they differ in focus. Served alongside mog-sec for A/B. Then measured dflash vs MTP on Sentinel's finetuned body: dflash 2.40 vs MTP 2.18 mean acceptance length (+11%), warm decode ~121 tok/s (faster than sec ~102). dflash k=7 cut over. stacks/sentinel-r3/. ⚠ MEASUREMENT LESSON (again): first decode bench read 39 tok/s — a COLD-boot + concurrent-contention artifact; warm+isolated it was 121. Operator caught it by testing the running sec (102 tok/s) as reference.

✅ THE BLOCKER — RESOLVED 2026-09-14 (no source build needed)

gen-large now serves orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 from /tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8. Healthy, coherent, MTP k=3.

The earlier diagnosis was right about the symptom and wrong about the cost. It said the only auditable path was cherry-picking a PLE-loader branch onto a from-source Blackwell build. Reading the loader in the running nightly showed otherwise:

from_quant_config  (vllm/models/qwen4_exp/nvidia/ngram_embedding.py:168)
  1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method   <-- BEFORE any type check
  2. quant_config is None                   -> unquantized
  3. ModelOptMixedPrecisionConfig           -> FP8 / unquantized
  4. ModelOptQuantConfigBase + excluded     -> unquantized
  5. not isinstance(quant_config, Fp8Config)-> NotImplementedError   <-- the blocker

Branch 1 is unconditional, and the NotImplementedError is scoped to the PLE embedding path only — experts and dense layers of a compressed-tensors qwen4_exp build load through vLLM's ordinary compressed-tensors paths. Falsified directly before doing any work: built the real CompressedTensorsConfig from orca's own config and called from_quant_config both ways — as-shipped raises, with the declaration returns Qwen4ExpPLEFp8EmbeddingMethod.

What was actually done

1. Converted the PLE table bf16 → FP8. orca's 128 PLE tensors live in exactly ONE shard (model-00002-of-00017.safetensors, 95.4 GiB) with no other tensors in it — a clean split. Converted on GPU3 (reserved scratch) in 139 s into 8 model-plefp8-* files.
   • global amax 0.0894; per-shard outlier ratio only 1.66x, so one global scale fits
   • scale chosen exactly representable in bf16 (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax maps to 446.17 / 448 → no clipping
   • round-trip 2.655 % RMS relative, 0.002 % underflow, 0 saturation
   • weight_scale written BF16 [1], matching gorbatjovy's published format (read from its actual safetensors header, not guessed)
   • MTP head (31 tensors, BF16) and the 333 vision tensors carried through untouched
2. Declared it: text_config.ple_embedding_dtype = "float8_e4m3fn".
3. Renamed layer_types: orca labels its 12 QSA layers qwen_sparse_attention; vLLM accepts only linear_attention / full_attention and picks QSA via indexer_n_heads. ⚠ Verified indexer_n_heads == 4 in BOTH orca and dealignai before renaming — without it the rename silently selects PLAIN attention and serves a subtly wrong model that still looks healthy. Every QSA/indexer key matches dealignai exactly.
4. .env: FN_MODEL → the converted dir, FN_QUANT → compressed-tensors.

Measured on the live seat

warm decode, conc=1, greedy 300 tok, n=5	median 167.5 tok/s (min 150.0, max 170.4, spread 12.2 %)
MTP k=3	acceptance 60.4 %, mean acceptance length 2.81 (per-pos 80.6/60.8/40.8 %)
KV	344,155 tokens @ 262,144 ctx, 1.31x concurrency, pinned 10 GiB
on card	~75 GiB; PLE 47.7 GiB pinned host RAM

⚠ Do NOT read 167.5 as a win over dealignai. dealignai's ~121 tok/s in this file came from a different harness/prompt; cross-harness comparison is invalid. ⚠ The 167.5 may itself have been contended — the operator was using the seat around that window — so treat it as a LOWER BOUND, not a clean solo figure. Operator's own session reported 140 tok/s average in real use while the 258K depth probe was running against the same card: an independent, contended floor that agrees with the picture. What IS established is that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did not cost decode speed, which was the standing risk of giving up FP4 tensor-core compute.

Still open on this seat

• Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context degradation axis). Needs a controlled harness + noise floor.
• ✅ Deep-prefill probe DONE 2026-09-14: clean to 258,517 tokens. Non-repeating prompts, 6 depths 32K->258K, all 200 OK, and zero allocator OOM/CUBLAS/illegal-memory in the engine log — the detector that caught dealignai's 155K near-miss. Run under real operator load, so a stricter test than solo. Positive control passed (a ~265K prompt got a clean 400 naming the limit). #54919 did not reproduce: 258K prefilled in 28.9 s, ~8,900 tok/s, near-linear. ⚠ The probe's MEMORY column was blind and must not be reused: --kv-cache-memory pins the pool ("skipped memory profiling"), so GPU use is flat vs depth, and before/after nvidia-smi bracketing cannot see a transient mid-prefill spike. Identical readings across an 8x depth range were the tell. Peak-activation headroom remains UNMEASURED. ⭐ Calibration for re-runs: random hex words tokenize at 7.9 tok/word.
• ⚠⚠ NO LOCAL ROLLBACK. dealignai weights DELETED 2026-09-14 on operator instruction (125 GiB reclaimed). .env.bak-preorca-20260914-023408 still names the old paths but they no longer exist — it is a record, not a revert. Reverting = 126 GiB re-download. The pristine 170 GiB qwen38-flash-next-orcarouter-nvfp4 IS retained (redo the conversion from it; do not delete it without a reason).
• ✅ Disk reclaimed 2026-09-14: 76 GiB. The 28 non-PLE shards duplicated between the pristine and converted orca dirs are now hardlinked (294G apparent → 218G actual). All 28 verified byte-identical by SHA-256 first, then ln to a temp name + atomic rename over the target — never rm-then-ln, which leaves a window with no file. Done live with the seat serving; it never blinked. ⚠ The two dirs now SHARE INODES. Editing a shared file in place in either dir changes both. Shards are never edited in place, and config.json / model.safetensors.index.json are deliberately NOT shared (the conversion changed them) — but a future session must copy-then-edit, not edit in place.

Other open items

• cyberprev quant (3rd sec candidate hotdogs/Qwen3.8-27B-abliterated-cyber-preview, bf16 at /tank/aimodels/cyberprev-bf16): restart crashed on a transformers head-count config error in quant-work/.venv — the SAME venv that reached 49/65 on Sept 11, so the original run likely used the canonical vllm-image+llmcompressor path (per services/gen-seat-mixed-quant/README.md), not the venv. Retry that way.
• sec vs sentinel-r3 quality A/B — both live and gateway-callable; operator to judge.
• ⚠ Suspect vault entry: fv-gateway/infra-ops-password is 16 chars matching a boot-UUID prefix exactly — possibly malformed. Eyeball.
• os-nut not installed on the FV OPNsense — with the firewall now on the 5P1000 UPS, a NUT transfer-to-battery event is a direct "breaker tripped" alarm nobody gets today.
• Branch breaker rating + 4-card ammeter reading still open — every power table is arithmetic on an ESTIMATED ~300 W platform draw. The ammeter converts it to fact.