feat(fv-ml1): generate the seat inventory from the live box instead of maintaining it by hand
The seat documentation must stay current, and a hand-written document cannot. The LiteLLM config described char-rp as a 31B model on a host and GPU it had not been on since 2026-08-24 -- three weeks of silent drift in a file that read as authoritative, and the reason a seat spent that period serving a model nobody intended. Anything typed here drifts the same way; anything read off the running containers cannot. scripts/seat-inventory.py derives the whole document from the host: - placement and VRAM from nvidia-smi compute-apps, mapped to containers through /proc/<pid>/cgroup -- nvidia-smi reports the vLLM engine child while docker reports the container pid, so matching them directly silently yields nothing - weights and KV tokens parsed from each engine's own startup log, not derived arithmetically, with concurrency computed as KV tokens over context - architecture, layer and expert counts, and the exact quantization group scheme (W4A4 vs W4A16 distinguished) from each model's config.json - speculative-decoding method and k from the container argv, which is how the three incompatible methods on this box became visible - lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model directory, not inside it, which is why an earlier pass wrongly reported two fully-documented seats as having no provenance - gateway aliases resolved from the LiteLLM config on ana-docker --check compares the committed document against the live box and exits non-zero when they diverge, ignoring only the generation timestamp. Suitable for CI or a scheduled drift alarm; read-only throughout, safe against production. Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which asserts the flag exists in the derived argv and aborts rather than running a campaign that silently ignored it.
This commit is contained in:
@@ -1,152 +1,144 @@
|
||||
# fv-ml1 — GPU seat inventory and model lineage
|
||||
|
||||
**Reconciled 2026-09-13 against the RUNNING containers**, not against compose files
|
||||
or the LiteLLM config. Every field below was read from `docker inspect`,
|
||||
`nvidia-smi`, or the model's own `config.json` on `/tank`. Where the gateway's
|
||||
documentation disagrees with what is deployed, the discrepancy is recorded in
|
||||
§ Drift rather than silently corrected.
|
||||
<!-- GENERATED FILE — DO NOT EDIT BY HAND.
|
||||
Regenerate: scripts/seat-inventory.py
|
||||
Check drift: scripts/seat-inventory.py --check
|
||||
Hand-edits are overwritten and, worse, drift silently while looking
|
||||
authoritative — which is exactly the failure this file replaced. -->
|
||||
|
||||
Host: `fv-ml1` 10.251.50.54 · 4× RTX PRO 6000 Blackwell Max-Q, 96 GB each
|
||||
Power cap: **275 W/card = 1100 W aggregate** (`gpu-power-cap.service`, before docker)
|
||||
**Generated 2026-09-14 06:00 UTC** by `scripts/seat-inventory.py`, read from the running
|
||||
containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own
|
||||
`config.json`, and the `.PROVENANCE.txt` siblings on `/tank`.
|
||||
|
||||
## Placement
|
||||
⚠ `.PROVENANCE.txt` lives *beside* the model directory, not inside it:
|
||||
`/tank/aimodels/<model>.PROVENANCE.txt`. `ls <model>/` will not show it.
|
||||
|
||||
| GPU | VRAM used | seats |
|
||||
|---|---|---|
|
||||
| 0 | 86.2 GB / 95.6 | `vllm-mog-sec` 47.8 · `vllm-gen` 38.4 |
|
||||
| 1 | 73.9 GB / 95.6 | `vllm-erp-seat` 27.0 · `vllm-gemma4-charrp` 26.0 · `vllm-reward` 9.2 · `vllm-coder` 6.1 · `vllm-embed` 3.5 · `vllm-rerank-a3` 2.1 · `scriberr` (no persistent VRAM) |
|
||||
| 2 | 96.4 GB / 95.6 | `vllm-flash-next` |
|
||||
| 3 | **empty — reserved** | held free for ad-hoc work (operator, 2026-09-13) |
|
||||
## Placement, KV cache and concurrency
|
||||
|
||||
⚠ **GPU 3 is deliberately empty.** Do not place a seat there without asking; it is
|
||||
the scratch card for benches, quants and probes. The MTP campaign that produced
|
||||
`services/flash-next-mtp-bench/` ran there.
|
||||
| GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | `vllm-mog-sec` | 46.6 GiB | 25.47 GiB | 342,920 | 163840 | **2.09×** | 0.50 |
|
||||
| 0 | `vllm-gen` | 38.3 GiB | 21.97 GiB | 268,205 | 262144 | **1.02×** | 0.38 |
|
||||
| 1 | `vllm-meromero-rp` | 37.9 GiB | 19.51 GiB | 266,334 | 262144 | **1.02×** | 0.40 |
|
||||
| 1 | `vllm-erp-seat` | 26.3 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.30 |
|
||||
| 1 | `vllm-reward` | 9.0 GiB | 4.41 GiB | 26,224 | 16384 | **1.60×** | 0.10 |
|
||||
| 1 | `vllm-coder` | 6.0 GiB | 2.98 GiB | 112,624 | 8192 | **13.75×** | 0.06 |
|
||||
| 1 | `vllm-embed` | 3.4 GiB | 1.12 GiB | 10,272 | 8192 | **1.25×** | 0.03 |
|
||||
| 1 | `vllm-rerank-a3` | 2.1 GiB | 1.06 GiB | — | 8192 | — | 0.03 |
|
||||
| 2 | `vllm-flash-next` | 94.7 GiB | 79.44 GiB | 344,155 | 262144 | **1.31×** | 0.96 |
|
||||
|
||||
## Seats
|
||||
**Concurrency** = KV tokens ÷ context: how many full-length requests fit at
|
||||
once. Below ~1.0× the seat cannot hold even one conversation at its declared
|
||||
context.
|
||||
|
||||
| container | GPU | port | served-model-name | ctx | util | seqs |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `vllm-mog-sec` | 0 | 8019 | `mog-sec-27b`, `mog-sec-27b-thinking` | 163840 | 0.50 | 16 |
|
||||
| `vllm-gen` | 0 | 8015 | `qwen3.8-27b-uncensored`, `-thinking` | 262144 | 0.38 | 16 |
|
||||
| `vllm-erp-seat` | 1 | 8021 | `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | 262144 | 0.30 | 32 |
|
||||
| `vllm-gemma4-charrp` | 1 | 8016 | `char-rp`, `char-rp-thinking` | 262144 | 0.27 | 32 |
|
||||
| `vllm-reward` | 1 | 8003 | `Skywork/Skywork-Reward-V2-Llama-3.x` | 16384 | 0.10 | — |
|
||||
| `vllm-coder` | 1 | 8020 | `qwen2.5-coder-1.5b` | 8192 | 0.06 | 32 |
|
||||
| `vllm-embed` | 1 | 8001 | `Qwen/Qwen3-Embedding-0.6B` | 8192 | 0.03 | — |
|
||||
| `vllm-rerank-a3` | 1 | 8013 | `BAAI/bge-reranker-v2-m3` | 8192 | 0.03 | — |
|
||||
| `vllm-flash-next` | 2 | 8022 | `qwen3.8-flash-next-uncensored`, `-thinking` | 262144 | 0.96 | 16 |
|
||||
## Lineage and quantization
|
||||
|
||||
## Lineage — read from each model's own `config.json`
|
||||
### `vllm-gen` — GPU 0
|
||||
|
||||
| seat | on-disk path | architecture | layers | experts | quantization |
|
||||
|---|---|---|---|---|---|
|
||||
| `mog-sec` | `/tank/aimodels/mog-sec-27b-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 MLPs (layers 0–55) + FP8 attn/lm_head/layers 56–63 |
|
||||
| `gen` | `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 + FP8, same per-layer-group scheme |
|
||||
| `flash-next` | `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` | `Qwen4ExpForConditionalGeneration` (`qwen4_exp`) | 48 | **512, 10 active** | `modelopt_fp4` (declared on the CLI; no `quantization_config` in config.json) |
|
||||
| `char-rp` | `/tank/aimodels/gemma4-26b-a4b-it-nvfp4` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
| `erp-seat` | `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
- **serves:** `qwen3.8-27b-uncensored`, `qwen3.8-27b-uncensored-thinking`
|
||||
- **model:** `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed`
|
||||
- **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers
|
||||
- **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized)
|
||||
- **speculative decoding:** `{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}`
|
||||
- **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0`
|
||||
|
||||
`mog-sec` carries `max_position_embeddings` **1,000,000** in config but is served at
|
||||
163,840 — the CLI, not the checkpoint, is the operative limit.
|
||||
### `vllm-mog-sec` — GPU 0
|
||||
|
||||
**`char-rp` and `erp-seat` are the same architecture, size and expert count** — both
|
||||
Gemma-4 26B A4B MoE at NVFP4. `erp-seat` is the uncensored/"heretic" abliteration of
|
||||
that family; `char-rp` is the stock instruct build. That is why they are co-located.
|
||||
- **serves:** `mog-sec-27b`, `mog-sec-27b-thinking`
|
||||
- **model:** `/tank/aimodels/mog-sec-27b-nvfp4-mixed`
|
||||
- **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers
|
||||
- **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized)
|
||||
- **speculative decoding:** `{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}`
|
||||
- **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013`
|
||||
- **provenance:**
|
||||
```
|
||||
model: Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (bf16, pen-test seat source)
|
||||
source_url: https://huggingface.co/Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16
|
||||
revision_pinned: deede67794b4eaaf31f016d02a0aaf71f1a303b9
|
||||
pulled_by: infra-ops (as llmuser)
|
||||
pulled_at_utc: 2026-08-21T09:10Z
|
||||
size_on_disk: 52 GB (18 shards, index total_size 55.6 GB)
|
||||
```
|
||||
|
||||
## Speculative decoding — THREE different methods in play
|
||||
### `vllm-coder` — GPU 1
|
||||
|
||||
This is the least obvious thing about the box and the easiest to get wrong.
|
||||
- **serves:** `qwen2.5-coder-1.5b`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
|
||||
| seat | method | k | drafter |
|
||||
|---|---|---|---|
|
||||
| `vllm-mog-sec` | **`dflash`** | **7** | separate model at `/drafter` |
|
||||
| `vllm-gen` | **`qwen3_5_mtp`** | 3 | self (MTP head) |
|
||||
| `vllm-flash-next` | **`mtp`** | 3 | self (MTP head) |
|
||||
| all others | none | — | — |
|
||||
### `vllm-embed` — GPU 1
|
||||
|
||||
⚠ The method string is **architecture-specific and not interchangeable**:
|
||||
`qwen3_5_mtp` for the qwen3_5 seats, plain `mtp` for `qwen4_exp`. vLLM warns that
|
||||
`qwen4_exp_mtp` is deprecated and rewrites it to `mtp` at boot. Copying a
|
||||
`--speculative-config` between seats without changing `method` will not work.
|
||||
- **serves:** `Qwen/Qwen3-Embedding-0.6B`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
|
||||
`flash-next` gained MTP on **2026-09-13** — see `services/flash-next-mtp-bench/`
|
||||
for the measurement (+52% at conc=1) and `stacks/flash-next-seat/compose.yaml` for
|
||||
the rationale and the KV-cache cost that came with it.
|
||||
### `vllm-erp-seat` — GPU 1
|
||||
|
||||
## Image sprawl
|
||||
- **serves:** `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`
|
||||
- **model:** `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`
|
||||
- **architecture:** Gemma4ForConditionalGeneration (gemma4), 30 layers, 128 experts
|
||||
- **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized)
|
||||
- **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0`
|
||||
|
||||
Five distinct vLLM images across nine seats:
|
||||
### `vllm-meromero-rp` — GPU 1
|
||||
|
||||
| image | seats |
|
||||
- **serves:** `char-rp`, `char-rp-thinking`
|
||||
- **model:** `/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16`
|
||||
- **architecture:** Gemma4ForConditionalGeneration (gemma4), 60 layers
|
||||
- **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized)
|
||||
- **image:** `vllm/vllm-openai:v0.26.0`
|
||||
|
||||
### `vllm-rerank-a3` — GPU 1
|
||||
|
||||
- **serves:** `BAAI/bge-reranker-v2-m3`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:v0.24.0`
|
||||
|
||||
### `vllm-reward` — GPU 1
|
||||
|
||||
- **serves:** `Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ`
|
||||
- **model:** `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ`
|
||||
- **architecture:** LlamaForSequenceClassification (llama), 32 layers
|
||||
- **quantization:** compressed-tensors / pack-quantized — W4A16 (pack-quantized)
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
|
||||
### `vllm-flash-next` — GPU 2
|
||||
|
||||
- **serves:** `qwen3.8-flash-next-uncensored`, `qwen3.8-flash-next-uncensored-thinking`
|
||||
- **model:** `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4`
|
||||
- **architecture:** Qwen4ExpForConditionalGeneration (qwen4_exp), 48 layers, 512 experts
|
||||
- **quantization:** modelopt / None — W4A4 (None)
|
||||
- **speculative decoding:** `{"method": "mtp", "num_speculative_tokens": 3}`
|
||||
- **image:** `vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`
|
||||
|
||||
## Gateway aliases resolving to this host
|
||||
|
||||
19 aliases. Ports with no listening seat are marked dead.
|
||||
|
||||
| alias | port |
|
||||
|---|---|
|
||||
| `nightly-eed1f3d0…` | `flash-next` |
|
||||
| `nightly-311b3513…` | `gen`, `erp-seat` |
|
||||
| `nightly-e9d1398d…` | `mog-sec` |
|
||||
| `v0.26.0` | `gemma4-charrp` |
|
||||
| `v0.24.0` | `rerank-a3` |
|
||||
| **`latest`** | `reward`, `coder`, `embed` |
|
||||
| `char-rp` | 8016 |
|
||||
| `char-rp-fast` | 8021 |
|
||||
| `char-rp-reasoning` | 8016 |
|
||||
| `chat-judge` | 8015 |
|
||||
| `classifier` | 8015 |
|
||||
| `coder-fast` | 8020 |
|
||||
| `erp-tune-v2` | 8098 |
|
||||
| `gemma4-26b-a4b-it-base` | 8099 |
|
||||
| `gen` | 8015 |
|
||||
| `gen-large` | 8022 |
|
||||
| `gen-reasoning` | 8015 |
|
||||
| `image-judge` | 8015 |
|
||||
| `qwen-image-bench` | 8015 |
|
||||
| `qwen3-embedding` | 8001 |
|
||||
| `reranker` | 8013 |
|
||||
| `sec` | 8019 |
|
||||
| `sec-reasoning` | 8019 |
|
||||
| `summarizer` | 8015 |
|
||||
| `summarizer-large` | 8015 |
|
||||
|
||||
⚠ **`:latest` is a floating tag on three seats.** Whatever those pull on next
|
||||
recreate is not what is running now, and there is no record of what they were built
|
||||
against. Pin them at the next touch.
|
||||
---
|
||||
|
||||
## Drift found 2026-09-13 — gateway docs vs reality
|
||||
|
||||
**1. `char-rp` is documented as a different model on a different host and GPU.**
|
||||
The LiteLLM config comments describe `char-rp` as *"MeroMero-v2 NON-THINKING prose
|
||||
seat (:8016, vLLM, meromero-charrp stack on ana-ml2 GPU 0), G4-MeroMero-v2-31B
|
||||
NVFP4A16"*. What is actually serving :8016 is **`gemma4-26b-a4b-it-nvfp4` — a 26B,
|
||||
30-layer, 128-expert stock Gemma-4** — in container `vllm-gemma4-charrp` on
|
||||
**fv-ml1 GPU 1**. Part of that is stale host naming (ana-ml2 → fv-ml1, relocated
|
||||
2026-09), but **31B MeroMero-v2 and 26B stock gemma4 are different models**, and the
|
||||
comment block carries sampler defaults keyed to the model it names. Those defaults
|
||||
have not been re-validated against what is deployed.
|
||||
|
||||
**2. Container name does not match the alias it serves.** `vllm-erp-seat` (:8021)
|
||||
backs the **`char-rp-fast`** alias, not an "erp" alias. The name predates the routing.
|
||||
|
||||
**3. Two gateway aliases point at ports with nothing listening.** Both confirmed
|
||||
dead (`curl` → no response, no container bound):
|
||||
|
||||
| alias | target | status |
|
||||
|---|---|---|
|
||||
| `erp-tune-v2` | `10.251.50.54:8098` | **dead** |
|
||||
| `gemma4-26b-a4b-it-base` | `10.251.50.54:8099` | **dead** |
|
||||
|
||||
These are almost certainly leftovers from the quant/eval containers still present
|
||||
as exited (`erp-v7-quant`, `erp-v6-quant`, `a4b-moe-quant`, `erp-eval-v2`,
|
||||
`vllm-gemma4-trainee-bench`). A caller hitting either alias gets a connection
|
||||
failure, not a clear "model retired" error. Per the standing
|
||||
never-alias-a-model-under-a-stale-name rule, they should be removed from the
|
||||
gateway or repointed deliberately — not left to fail.
|
||||
|
||||
## Gateway aliases resolving to this host (19)
|
||||
|
||||
`char-rp` · `char-rp-fast` · `char-rp-reasoning` · `chat-judge` · `classifier` ·
|
||||
`coder-fast` · `erp-tune-v2` ⚠dead · `gemma4-26b-a4b-it-base` ⚠dead · `gen` ·
|
||||
`gen-large` · `gen-reasoning` · `image-judge` · `qwen-image-bench` ·
|
||||
`qwen3-embedding` · `reranker` · `sec` · `sec-reasoning` · `summarizer` ·
|
||||
`summarizer-large`
|
||||
|
||||
**Seven aliases share one seat.** `summarizer`, `summarizer-large`, `classifier`,
|
||||
`chat-judge`, `image-judge`, `qwen-image-bench` and `gen` all resolve to
|
||||
`vllm-gen` on :8015 — so anything that loads the summarizer loads GPU 0. This is
|
||||
why GPU 0 shows background draw during ordinary OpenWebUI use: its auto-titling
|
||||
fires the `summarizer` alias on every exchange.
|
||||
|
||||
## How to regenerate this
|
||||
|
||||
Read the containers, not the compose files:
|
||||
|
||||
```sh
|
||||
# placement + footprint
|
||||
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv
|
||||
# map pid -> container via /proc/<pid>/cgroup, then docker inspect
|
||||
docker inspect <name> --format '{{json .Args}}' # served names, quant, spec-config
|
||||
docker inspect <name> --format '{{json .Mounts}}' # resolve /model to a /tank path
|
||||
# lineage
|
||||
python3 -c "import json;c=json.load(open('<path>/config.json'));print(c['architectures'])"
|
||||
```
|
||||
|
||||
⚠ `docker inspect` reports the **container's** PID; `nvidia-smi` reports the vLLM
|
||||
engine child. Map through `/proc/<pid>/cgroup`, not by matching PIDs directly.
|
||||
*Regenerate with `scripts/seat-inventory.py` after ANY seat change —
|
||||
model swap, quant change, context or utilization edit, or speculative-decoding
|
||||
change. Run `--check` in CI to catch a stale document.*
|
||||
|
||||
Reference in New Issue
Block a user