memory: snapshot — FV recovered, fv-ml1 seat reorg, gen-large blocked on orca runtime
FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000, GPU caps 275W/card). All-night fv-ml1 seat reorganization: - flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1), inverting vLLM's 4xH100 recipe; KV 14->10 GiB. - gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases repointed); 27B dense gen seat retired, 38 GB freed on GPU0. - char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier. - Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7 cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm). gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded (170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE; the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell image). Runtime decision pending -- this is the resume point. Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad. Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# OPNsense API — vendored reference + field-shape notes
|
||||
|
||||
**Source:** <https://docs.opnsense.org/development/api.html> (fetched 2026-09-13)
|
||||
**Why vendored:** the upstream page documents the REST conventions but explicitly
|
||||
*not* parameter shapes — "the auto-generated API documentation captures endpoints
|
||||
and HTTP methods but not all parameter details." Those shapes are where the time
|
||||
goes, so the hard-won ones are recorded in § Field shapes below.
|
||||
|
||||
## Authentication
|
||||
|
||||
HTTP Basic auth: **key = username, secret = password**.
|
||||
|
||||
```sh
|
||||
curl -s -u "$KEY:$SECRET" "http://<gw>/api/<module>/<controller>/<command>"
|
||||
```
|
||||
|
||||
Fleet creds are vaulted: `fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`
|
||||
(see `secret get`). Authorization is per-user **Effective Privileges** — a key only
|
||||
reaches endpoints its owner is authorized for.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Endpoint pattern:** `/api/<module>/<controller>/<command>/[<param>/...]`
|
||||
- **GET** retrieves; **POST** creates, updates, or executes an action.
|
||||
- Request bodies and responses are `application/json`.
|
||||
|
||||
Search endpoints take a paging body and return a rows envelope:
|
||||
|
||||
```json
|
||||
{"current":1,"rowCount":7,"sort":{},"searchPhrase":""}
|
||||
```
|
||||
```json
|
||||
{"total":10,"rowCount":7,"current":1,
|
||||
"rows":[{"id":"configd","locked":1,"running":1,"description":"System Configuration Daemon","name":"configd"}]}
|
||||
```
|
||||
|
||||
## Service control
|
||||
|
||||
`/api/<module>/service/{status,start,stop,restart,reconfigure}` — `reconfigure`
|
||||
writes config **and** applies it, which is normally the one you want after a
|
||||
`settings/set`. Observed status values include `disabled`, `stopped`, `running`
|
||||
— note **`disabled` ≠ `stopped`**: a disabled service will not start until its
|
||||
model's `enabled` field is set to `"1"`.
|
||||
|
||||
## Field shapes — the part upstream does not document
|
||||
|
||||
⚠ **A `settings/get` response is NOT a valid `settings/set` body.** They are
|
||||
different shapes, and mixing them returns a bare
|
||||
`{"errorMessage":"Unexpected error, check log for details"}` with no indication
|
||||
of which field was wrong.
|
||||
|
||||
**Selection / option fields.** `get` returns every option with a `selected` flag;
|
||||
`set` wants only the selected key.
|
||||
|
||||
```jsonc
|
||||
// get
|
||||
"useExitNode": {"": {"value": "None", "selected": 1}, "abc": {"value":"node-a","selected":0}}
|
||||
// set
|
||||
"useExitNode": ""
|
||||
```
|
||||
|
||||
**Array / list fields are UUID-keyed objects, not lists.** This is the one that
|
||||
costs an afternoon. `get` renders an empty array as `[]`, which misleads you into
|
||||
POSTing a list.
|
||||
|
||||
```jsonc
|
||||
// get, when empty -> looks like a plain list
|
||||
"subnets": {"subnet4": []}
|
||||
// set -> object keyed by a UUID you generate, values are the item's own fields
|
||||
"subnets": {"subnet4": {"<uuid4>": {"subnet": "10.251.0.0/16", "description": "..."}}}
|
||||
```
|
||||
|
||||
**Discovering an array item's inner fields — two reliable tricks:**
|
||||
|
||||
1. POST the wrong shape on purpose. Validation names the full path:
|
||||
`{"result":"failed","validations":{"settings.subnets.subnet4.<uuid>.subnet":"A value is required."}}`
|
||||
2. Many models expose a template getter: `/api/<module>/settings/get_<item>`
|
||||
returns the blank item, e.g. `get_subnet` → `{"subnet4":{"subnet":"","description":""}}`
|
||||
|
||||
**Partial `set` is supported** — POST only the fields you are changing rather than
|
||||
round-tripping the whole model. This sidesteps every other field's shape problem
|
||||
and is the recommended approach for a surgical change.
|
||||
|
||||
**Validation error format:** `{"result":"failed","validations":{"<dotted.path>":"<message>"}}`.
|
||||
A successful save is `{"result":"saved"}`.
|
||||
|
||||
## Working example — enabling a subnet router (2026-09-13, FV gateway)
|
||||
|
||||
```jsonc
|
||||
POST /api/tailscale/settings/set
|
||||
{"settings":{"enabled":"1",
|
||||
"subnets":{"subnet4":{"<uuid4>":{"subnet":"10.251.0.0/16",
|
||||
"description":"FV site subnet router"}}}}}
|
||||
POST /api/tailscale/service/reconfigure {}
|
||||
```
|
||||
|
||||
**Always read back.** `settings/get` after the write, and re-check anything the
|
||||
change could disturb (here: `/api/firewall/source_nat/search_rule` total stayed
|
||||
at 2). A write that reports success is not evidence the value landed.
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/runbooks/fv-to-ana-nat.md` — the FV gateway's manual SNAT rule
|
||||
- `docs/runbooks/fv-site-dark-20260913.md` — the outage this reference came out of
|
||||
@@ -0,0 +1,97 @@
|
||||
# 2026-09-14 — fv-ml1 seat reorganization + the gen-large "orca" blocker
|
||||
|
||||
An all-night GPU-seat overhaul on fv-ml1 after the FV colo recovered. Everything
|
||||
here is on the running box; regenerate the authoritative view with
|
||||
`scripts/seat-inventory.py` (reads the live containers). Cross-refs:
|
||||
[[2026-09-13-flash-next-seat-and-fv-outage]].
|
||||
|
||||
## Seat topology now (2026-09-14 ~01:40 PT)
|
||||
|
||||
| GPU | seats |
|
||||
|---|---|
|
||||
| 0 | `mog-sec` (`sec` :8019, dflash k=7) · **`sentinel-r3`** (`sentinel-r3` :8025, dflash k=7 — NEW) |
|
||||
| 1 | **`meromero-charrp`** (`char-rp` :8016, MeroMero-v2-31B dense — restored) · `erp-seat` (`char-rp-fast` :8021) · reward · coder · embed · rerank |
|
||||
| 2 | `flash-next` (`gen-large` :8022) — **DOWN** (orca swap failed; dealignai stopped) |
|
||||
| 3 | **RESERVED scratch** — empty, operator directive; benches/quants/probes only |
|
||||
|
||||
## What changed tonight
|
||||
|
||||
1. **flash-next gained MTP k=3.** Campaign in `services/flash-next-mtp-bench/`
|
||||
measured MTP a WIN on this hardware (+29/41/27% at k1, +42/52/38% k2, +52/51/34%
|
||||
k3 across conc 1/4/8), inverting vLLM's 4×H100 recipe. k=3 deployed (single-user
|
||||
→ conc=1 dominates). KV pinned 14→10 GiB (MTP adds 5.08 GiB draft-head weights;
|
||||
14 GiB OOMs). Warm decode ~121 tok/s. `stacks/flash-next-seat/compose.yaml`.
|
||||
2. **gen consolidation.** All 8 `gen`/`gen-reasoning`/`summarizer`/`summarizer-large`/
|
||||
`classifier`/`chat-judge`/`image-judge`/`qwen-image-bench` LiteLLM entries repointed
|
||||
to flash-next (:8022); the **27B dense gen seat RETIRED**, freeing 38.4 GB on GPU0.
|
||||
⚠ judge aliases now score against different weights — prior scores incomparable.
|
||||
3. **char-rp restored to MeroMero-v2-31B.** Was serving a leftover-test RedHatAI 26B
|
||||
MoE W4A4; operator wanted the in-house dense-31B heretic W4A16 back. `char-rp-fast`
|
||||
(:8021, the 26B MoE) is the deliberate speed tier — the throughput answer, so
|
||||
char-rp stays the quality seat. `stacks/meromero-charrp/`.
|
||||
4. **Sentinel-R3 A/B + dflash.** `glyphsoftware/sentinel-r3` (proprietary license —
|
||||
operator's call) is a REAL SFT pentest finetune vs mog-sec's persona-on-stock.
|
||||
Served alongside mog-sec for A/B. Then measured dflash vs MTP on Sentinel's
|
||||
finetuned body: **dflash 2.40 vs MTP 2.18 mean acceptance length (+11%)**, warm
|
||||
decode ~121 tok/s (faster than sec ~102). dflash k=7 cut over. `stacks/sentinel-r3/`.
|
||||
⚠ MEASUREMENT LESSON (again): first decode bench read 39 tok/s — a COLD-boot +
|
||||
concurrent-contention artifact; warm+isolated it was 121. Operator caught it by
|
||||
testing the running `sec` (102 tok/s) as reference.
|
||||
|
||||
## THE BLOCKER — gen-large "orca" swap (resume here)
|
||||
|
||||
Operator wants gen-large on **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`**
|
||||
(weight-only compressed-tensors: NVFP4 experts + FP8 attn + bf16 PLE) instead of the
|
||||
incumbent **dealignai** build (modelopt_fp4 **W4A4** — 4-bit activations, the
|
||||
long-context degradation mode). orcarouter is the trusted author (= `gen`), better on
|
||||
the activation axis. Downloaded to `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4`
|
||||
(170 GB, complete, 18 shards + MTP head, integrity-verified).
|
||||
|
||||
**IT DOES NOT LOAD.** Root cause (read the loader source, definitive):
|
||||
`Qwen4ExpPLEEmbeddingMethod.from_quant_config` in vLLM — even the LATEST mainline
|
||||
nightly — has branches only for **ModelOpt** and **FP8** configs; for a
|
||||
CompressedTensorsConfig it raises `NotImplementedError` BEFORE ever consulting the
|
||||
ignore list. So:
|
||||
- dealignai (modelopt_fp4) loads fine; orcarouter (compressed-tensors) cannot, on any
|
||||
mainline vLLM.
|
||||
- orcarouter's config also does NOT list its `ngram_embedding` in `ignore` (would need
|
||||
adding) — but that's moot until the loader has a compressed-tensors branch.
|
||||
- orcarouter's OWN card prescribes **TP=4 + `--enable-expert-parallel`** and
|
||||
`vllm>=<qwen4exp-supporting release>` — a LITERAL UNFILLED version placeholder. They
|
||||
shipped a build whose runtime they couldn't pin.
|
||||
|
||||
**Backport vetted → UNFIT.** `wtdcode/vllm-backport` (docker `lazymio/vllm-backport`,
|
||||
212★, Apache-2.0, DCO, active) resolved this exact error (issue #72) BUT: its purpose is
|
||||
OLD GPUs (images sm86/sm80/sm89 only — **no Blackwell/sm_120**), it's built around its
|
||||
OWN `wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16` quant not orcarouter's NVFP4, and Blackwell
|
||||
has native FP4/FP8 so the whole backport premise is moot for us. Using it = source-build
|
||||
for an untested arch to load a quant it never tested.
|
||||
|
||||
**Options (operator decision pending):**
|
||||
- (a) **Restore dealignai** — works today, identical architecture, ~90% of the quality
|
||||
(loses only the weight-only-vs-W4A4 activation edge). The pragmatic interim.
|
||||
- (b) Cherry-pick the ~10-line compressed-tensors PLE-loader branch onto a mainline
|
||||
source build for Blackwell (mainline already has Blackwell FP4 compute). More
|
||||
auditable than the backport, still a from-source build + maintenance.
|
||||
- (c) Request orcarouter's own **W4A4** build (they offer it) — would load on the current
|
||||
mainline image like dealignai.
|
||||
- (d) Wait for mainline vLLM to merge compressed-tensors qwen4_exp.
|
||||
|
||||
⚠ Operator said "don't reload dealignai unprompted, priority orca" — but orca needs
|
||||
runtime work, so gen-large is intentionally DOWN pending the decision. Both configs
|
||||
backed up on host: `compose.yaml.bak-20260914-orcarouter`, `.env.bak-20260914-orcarouter`.
|
||||
|
||||
## Other open items
|
||||
|
||||
- **cyberprev quant** (3rd sec candidate `hotdogs/Qwen3.8-27B-abliterated-cyber-preview`,
|
||||
bf16 at `/tank/aimodels/cyberprev-bf16`): restart crashed on a transformers
|
||||
head-count config error in `quant-work/.venv` — the SAME venv that reached 49/65 on
|
||||
Sept 11, so the original run likely used the canonical vllm-image+llmcompressor path
|
||||
(per `services/gen-seat-mixed-quant/README.md`), not the venv. Retry that way.
|
||||
- **sec vs sentinel-r3 quality A/B** — both live and gateway-callable; operator to judge.
|
||||
- ⚠ **Suspect vault entry**: `fv-gateway/infra-ops-password` is 16 chars matching a
|
||||
boot-UUID prefix exactly — possibly malformed. Eyeball.
|
||||
- **os-nut not installed** on the FV OPNsense — with the firewall now on the 5P1000 UPS,
|
||||
a NUT transfer-to-battery event is a direct "breaker tripped" alarm nobody gets today.
|
||||
- **Branch breaker rating + 4-card ammeter reading** still open — every power table is
|
||||
arithmetic on an ESTIMATED ~300 W platform draw. The ammeter converts it to fact.
|
||||
+41
-96
File diff suppressed because one or more lines are too long
@@ -0,0 +1,10 @@
|
||||
SENTINEL_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013
|
||||
API_KEY=
|
||||
SENTINEL_GPU_ID=0
|
||||
SENTINEL_PORT=8025
|
||||
SENTINEL_MODEL=/tank/aimodels/sentinel-r3-nvfp4-mixed
|
||||
SENTINEL_GPU_MEM_UTIL=0.40
|
||||
SENTINEL_MAX_MODEL_LEN=163840
|
||||
SENTINEL_KV_CACHE_MEMORY=8589934592
|
||||
SENTINEL_MAX_NUM_BATCHED_TOKENS=4096
|
||||
SENTINEL_DRAFT_MODEL=/tank/aimodels/qwen38-27b-dflash2-drafter
|
||||
@@ -0,0 +1,124 @@
|
||||
# sentinel-r3 — Sentinel-R3 pen-test seat, A/B candidate ALONGSIDE mog-sec on fv-ml1 GPU 0.
|
||||
#
|
||||
# glyphsoftware/sentinel-r3: a REAL SFT finetune of stock Qwen/Qwen3.8-27B on 1,230
|
||||
# authorized-pentest agent trajectories (recon -> foothold -> privesc -> writeup) over a
|
||||
# 19-tool surface that matches our own harness. Contrast mog-sec, which is a persona
|
||||
# system prompt on stock weights. Quantized in-house to the same mixed NVFP4 W4A4(MLP
|
||||
# 0-55) + FP8 W8A8(attn/lm_head/MLP 56-63) recipe as mog-sec/gen. → sentinel-r3-nvfp4-mixed.PROVENANCE.txt
|
||||
#
|
||||
# ⚠ PROPRIETARY LICENSE (Glyph Proprietary v1.0) — operator's fair-use/licensee call, unlike
|
||||
# mog-sec's Apache. Served here on operator instruction 2026-09-14.
|
||||
#
|
||||
# ⚠ dflash speculative decoding ENABLED 2026-09-14 after measurement (see the spec-config note
|
||||
# below). Originally served without spec to establish a clean baseline; the probe measured
|
||||
# dflash k=7 at 2.40 acceptance length vs MTP k=3 at 2.18 on this finetuned body.
|
||||
# (historical) SERVED WITHOUT --speculative-config ON PURPOSE. The MTP head is a VERBATIM base graft
|
||||
# (source shipped zero mtp.*), and its acceptance on this SFT-finetuned body is UNVERIFIED —
|
||||
# the gate is a measured >=~40% on a probe serve, not an assumption. A clean no-spec boot
|
||||
# establishes the quality baseline for the mog-sec A/B first; MTP acceptance is a separate
|
||||
# measurement (add the qwen3_5_mtp spec-config and read the acceptance metric).
|
||||
#
|
||||
# ⚠ max-model-len 163840, NOT native 262K. Sentinel is the SAME base + hybrid Qwen3_5 arch as
|
||||
# mog-sec, so the identical deep-context lesson applies: what the KV pool HOLDS and what the
|
||||
# card PROCESSES at depth are different numbers, and mog-sec crashed five times before 163840
|
||||
# bought a clean 400-refusal above the measured ceiling instead of an engine death. Do not
|
||||
# raise without re-running the deep-ctx probe on THIS checkpoint.
|
||||
#
|
||||
# ⚠ Serve with a PROSE system prompt — Sentinel was trained on prose tool descriptions, not
|
||||
# structured `tools=`.
|
||||
|
||||
name: sentinel-r3
|
||||
|
||||
services:
|
||||
vllm-sentinel-r3:
|
||||
image: ${SENTINEL_IMAGE:-vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013}
|
||||
container_name: ${SENTINEL_CONTAINER_NAME:-vllm-sentinel-r3}
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${SENTINEL_PORT:-8025}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
- ${SENTINEL_MODEL:-/tank/aimodels/sentinel-r3-nvfp4-mixed}:/model:ro
|
||||
- ${SENTINEL_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro
|
||||
environment:
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- /model
|
||||
- --served-model-name
|
||||
- ${SENTINEL_SERVED_NAME:-sentinel-r3}
|
||||
- ${SENTINEL_SERVED_NAME_THINK:-sentinel-r3-thinking}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --quantization
|
||||
- ${SENTINEL_QUANT:-compressed-tensors}
|
||||
- --gpu-memory-utilization
|
||||
- "${SENTINEL_GPU_MEM_UTIL:-0.40}"
|
||||
# KV pinned in bytes — same discipline as mog-sec/erp-seat: a ratio yields a different
|
||||
# cache depending on what else is resident at start, an explicit figure is reproducible.
|
||||
# 8 GiB is generous for an A/B probe (conc 1/4/8 short prompts never approach it).
|
||||
- --kv-cache-memory
|
||||
- "${SENTINEL_KV_CACHE_MEMORY:-8589934592}"
|
||||
- --max-model-len
|
||||
- "${SENTINEL_MAX_MODEL_LEN:-163840}"
|
||||
- --max-num-seqs
|
||||
- "${SENTINEL_MAX_NUM_SEQS:-16}"
|
||||
- --max-num-batched-tokens
|
||||
- "${SENTINEL_MAX_NUM_BATCHED_TOKENS:-4096}"
|
||||
- --trust-remote-code
|
||||
- --dtype
|
||||
- auto
|
||||
- --mamba-cache-dtype
|
||||
- float32
|
||||
- --kv-cache-dtype
|
||||
- ${SENTINEL_KV_CACHE_DTYPE:-fp8}
|
||||
- --enable-prefix-caching
|
||||
- --enable-chunked-prefill
|
||||
- --limit-mm-per-prompt
|
||||
- '${SENTINEL_LIMIT_MM:-{"image": 4}}'
|
||||
- --mm-processor-kwargs
|
||||
- '${SENTINEL_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}'
|
||||
- --reasoning-parser
|
||||
- ${SENTINEL_REASONING_PARSER:-qwen3}
|
||||
- --default-chat-template-kwargs
|
||||
- '{"reasoning_effort": "${SENTINEL_REASONING_EFFORT:-medium}"}'
|
||||
- --enable-auto-tool-choice
|
||||
- --tool-call-parser
|
||||
- qwen3_coder
|
||||
# dflash speculative decoding — MEASURED 2026-09-14 on THIS finetuned body: dflash k=7
|
||||
# gives mean acceptance length 2.40 vs MTP k=3 at 2.18 (+11%, clean n=2 separation).
|
||||
# dflash wins on reach (accepts a longer tail to k=7) despite equal ~60% position-1
|
||||
# acceptance. The drafter is the same qwen38-27b-dflash2-drafter mog-sec uses (Sentinel
|
||||
# shares its 64-layer Qwen3.8-27B base). Costs +3.6 GB for the drafter.
|
||||
- --speculative-config
|
||||
- '${SENTINEL_SPEC_CONFIG:-{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}}'
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${SENTINEL_GPU_ID:-0}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 900s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=Sentinel-R3 (pen-test A/B)
|
||||
- homepage.icon=mdi-shield-search
|
||||
- homepage.description=SFT pen-test finetune of Qwen3.8-27B, A/B candidate vs mog-sec (fv-ml1 GPU 0)
|
||||
- homepage.href=http://10.251.50.54:${SENTINEL_PORT:-8025}/docs
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
Reference in New Issue
Block a user