feat(fv-ml1): generate the seat inventory from the live box instead of maintaining it by hand
The seat documentation must stay current, and a hand-written document cannot. The LiteLLM config described char-rp as a 31B model on a host and GPU it had not been on since 2026-08-24 -- three weeks of silent drift in a file that read as authoritative, and the reason a seat spent that period serving a model nobody intended. Anything typed here drifts the same way; anything read off the running containers cannot. scripts/seat-inventory.py derives the whole document from the host: - placement and VRAM from nvidia-smi compute-apps, mapped to containers through /proc/<pid>/cgroup -- nvidia-smi reports the vLLM engine child while docker reports the container pid, so matching them directly silently yields nothing - weights and KV tokens parsed from each engine's own startup log, not derived arithmetically, with concurrency computed as KV tokens over context - architecture, layer and expert counts, and the exact quantization group scheme (W4A4 vs W4A16 distinguished) from each model's config.json - speculative-decoding method and k from the container argv, which is how the three incompatible methods on this box became visible - lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model directory, not inside it, which is why an earlier pass wrongly reported two fully-documented seats as having no provenance - gateway aliases resolved from the LiteLLM config on ana-docker --check compares the committed document against the live box and exits non-zero when they diverge, ignoring only the generation timestamp. Suitable for CI or a scheduled drift alarm; read-only throughout, safe against production. Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which asserts the flag exists in the derived argv and aborts rather than running a campaign that silently ignored it.
This commit is contained in:
@@ -1,152 +1,144 @@
|
||||
# fv-ml1 — GPU seat inventory and model lineage
|
||||
|
||||
**Reconciled 2026-09-13 against the RUNNING containers**, not against compose files
|
||||
or the LiteLLM config. Every field below was read from `docker inspect`,
|
||||
`nvidia-smi`, or the model's own `config.json` on `/tank`. Where the gateway's
|
||||
documentation disagrees with what is deployed, the discrepancy is recorded in
|
||||
§ Drift rather than silently corrected.
|
||||
<!-- GENERATED FILE — DO NOT EDIT BY HAND.
|
||||
Regenerate: scripts/seat-inventory.py
|
||||
Check drift: scripts/seat-inventory.py --check
|
||||
Hand-edits are overwritten and, worse, drift silently while looking
|
||||
authoritative — which is exactly the failure this file replaced. -->
|
||||
|
||||
Host: `fv-ml1` 10.251.50.54 · 4× RTX PRO 6000 Blackwell Max-Q, 96 GB each
|
||||
Power cap: **275 W/card = 1100 W aggregate** (`gpu-power-cap.service`, before docker)
|
||||
**Generated 2026-09-14 06:00 UTC** by `scripts/seat-inventory.py`, read from the running
|
||||
containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own
|
||||
`config.json`, and the `.PROVENANCE.txt` siblings on `/tank`.
|
||||
|
||||
## Placement
|
||||
⚠ `.PROVENANCE.txt` lives *beside* the model directory, not inside it:
|
||||
`/tank/aimodels/<model>.PROVENANCE.txt`. `ls <model>/` will not show it.
|
||||
|
||||
| GPU | VRAM used | seats |
|
||||
|---|---|---|
|
||||
| 0 | 86.2 GB / 95.6 | `vllm-mog-sec` 47.8 · `vllm-gen` 38.4 |
|
||||
| 1 | 73.9 GB / 95.6 | `vllm-erp-seat` 27.0 · `vllm-gemma4-charrp` 26.0 · `vllm-reward` 9.2 · `vllm-coder` 6.1 · `vllm-embed` 3.5 · `vllm-rerank-a3` 2.1 · `scriberr` (no persistent VRAM) |
|
||||
| 2 | 96.4 GB / 95.6 | `vllm-flash-next` |
|
||||
| 3 | **empty — reserved** | held free for ad-hoc work (operator, 2026-09-13) |
|
||||
## Placement, KV cache and concurrency
|
||||
|
||||
⚠ **GPU 3 is deliberately empty.** Do not place a seat there without asking; it is
|
||||
the scratch card for benches, quants and probes. The MTP campaign that produced
|
||||
`services/flash-next-mtp-bench/` ran there.
|
||||
| GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | `vllm-mog-sec` | 46.6 GiB | 25.47 GiB | 342,920 | 163840 | **2.09×** | 0.50 |
|
||||
| 0 | `vllm-gen` | 38.3 GiB | 21.97 GiB | 268,205 | 262144 | **1.02×** | 0.38 |
|
||||
| 1 | `vllm-meromero-rp` | 37.9 GiB | 19.51 GiB | 266,334 | 262144 | **1.02×** | 0.40 |
|
||||
| 1 | `vllm-erp-seat` | 26.3 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.30 |
|
||||
| 1 | `vllm-reward` | 9.0 GiB | 4.41 GiB | 26,224 | 16384 | **1.60×** | 0.10 |
|
||||
| 1 | `vllm-coder` | 6.0 GiB | 2.98 GiB | 112,624 | 8192 | **13.75×** | 0.06 |
|
||||
| 1 | `vllm-embed` | 3.4 GiB | 1.12 GiB | 10,272 | 8192 | **1.25×** | 0.03 |
|
||||
| 1 | `vllm-rerank-a3` | 2.1 GiB | 1.06 GiB | — | 8192 | — | 0.03 |
|
||||
| 2 | `vllm-flash-next` | 94.7 GiB | 79.44 GiB | 344,155 | 262144 | **1.31×** | 0.96 |
|
||||
|
||||
## Seats
|
||||
**Concurrency** = KV tokens ÷ context: how many full-length requests fit at
|
||||
once. Below ~1.0× the seat cannot hold even one conversation at its declared
|
||||
context.
|
||||
|
||||
| container | GPU | port | served-model-name | ctx | util | seqs |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `vllm-mog-sec` | 0 | 8019 | `mog-sec-27b`, `mog-sec-27b-thinking` | 163840 | 0.50 | 16 |
|
||||
| `vllm-gen` | 0 | 8015 | `qwen3.8-27b-uncensored`, `-thinking` | 262144 | 0.38 | 16 |
|
||||
| `vllm-erp-seat` | 1 | 8021 | `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | 262144 | 0.30 | 32 |
|
||||
| `vllm-gemma4-charrp` | 1 | 8016 | `char-rp`, `char-rp-thinking` | 262144 | 0.27 | 32 |
|
||||
| `vllm-reward` | 1 | 8003 | `Skywork/Skywork-Reward-V2-Llama-3.x` | 16384 | 0.10 | — |
|
||||
| `vllm-coder` | 1 | 8020 | `qwen2.5-coder-1.5b` | 8192 | 0.06 | 32 |
|
||||
| `vllm-embed` | 1 | 8001 | `Qwen/Qwen3-Embedding-0.6B` | 8192 | 0.03 | — |
|
||||
| `vllm-rerank-a3` | 1 | 8013 | `BAAI/bge-reranker-v2-m3` | 8192 | 0.03 | — |
|
||||
| `vllm-flash-next` | 2 | 8022 | `qwen3.8-flash-next-uncensored`, `-thinking` | 262144 | 0.96 | 16 |
|
||||
## Lineage and quantization
|
||||
|
||||
## Lineage — read from each model's own `config.json`
|
||||
### `vllm-gen` — GPU 0
|
||||
|
||||
| seat | on-disk path | architecture | layers | experts | quantization |
|
||||
|---|---|---|---|---|---|
|
||||
| `mog-sec` | `/tank/aimodels/mog-sec-27b-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 MLPs (layers 0–55) + FP8 attn/lm_head/layers 56–63 |
|
||||
| `gen` | `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 + FP8, same per-layer-group scheme |
|
||||
| `flash-next` | `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` | `Qwen4ExpForConditionalGeneration` (`qwen4_exp`) | 48 | **512, 10 active** | `modelopt_fp4` (declared on the CLI; no `quantization_config` in config.json) |
|
||||
| `char-rp` | `/tank/aimodels/gemma4-26b-a4b-it-nvfp4` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
| `erp-seat` | `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
- **serves:** `qwen3.8-27b-uncensored`, `qwen3.8-27b-uncensored-thinking`
|
||||
- **model:** `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed`
|
||||
- **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers
|
||||
- **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized)
|
||||
- **speculative decoding:** `{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}`
|
||||
- **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0`
|
||||
|
||||
`mog-sec` carries `max_position_embeddings` **1,000,000** in config but is served at
|
||||
163,840 — the CLI, not the checkpoint, is the operative limit.
|
||||
### `vllm-mog-sec` — GPU 0
|
||||
|
||||
**`char-rp` and `erp-seat` are the same architecture, size and expert count** — both
|
||||
Gemma-4 26B A4B MoE at NVFP4. `erp-seat` is the uncensored/"heretic" abliteration of
|
||||
that family; `char-rp` is the stock instruct build. That is why they are co-located.
|
||||
- **serves:** `mog-sec-27b`, `mog-sec-27b-thinking`
|
||||
- **model:** `/tank/aimodels/mog-sec-27b-nvfp4-mixed`
|
||||
- **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers
|
||||
- **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized)
|
||||
- **speculative decoding:** `{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}`
|
||||
- **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013`
|
||||
- **provenance:**
|
||||
```
|
||||
model: Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (bf16, pen-test seat source)
|
||||
source_url: https://huggingface.co/Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16
|
||||
revision_pinned: deede67794b4eaaf31f016d02a0aaf71f1a303b9
|
||||
pulled_by: infra-ops (as llmuser)
|
||||
pulled_at_utc: 2026-08-21T09:10Z
|
||||
size_on_disk: 52 GB (18 shards, index total_size 55.6 GB)
|
||||
```
|
||||
|
||||
## Speculative decoding — THREE different methods in play
|
||||
### `vllm-coder` — GPU 1
|
||||
|
||||
This is the least obvious thing about the box and the easiest to get wrong.
|
||||
- **serves:** `qwen2.5-coder-1.5b`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
|
||||
| seat | method | k | drafter |
|
||||
|---|---|---|---|
|
||||
| `vllm-mog-sec` | **`dflash`** | **7** | separate model at `/drafter` |
|
||||
| `vllm-gen` | **`qwen3_5_mtp`** | 3 | self (MTP head) |
|
||||
| `vllm-flash-next` | **`mtp`** | 3 | self (MTP head) |
|
||||
| all others | none | — | — |
|
||||
### `vllm-embed` — GPU 1
|
||||
|
||||
⚠ The method string is **architecture-specific and not interchangeable**:
|
||||
`qwen3_5_mtp` for the qwen3_5 seats, plain `mtp` for `qwen4_exp`. vLLM warns that
|
||||
`qwen4_exp_mtp` is deprecated and rewrites it to `mtp` at boot. Copying a
|
||||
`--speculative-config` between seats without changing `method` will not work.
|
||||
- **serves:** `Qwen/Qwen3-Embedding-0.6B`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
|
||||
`flash-next` gained MTP on **2026-09-13** — see `services/flash-next-mtp-bench/`
|
||||
for the measurement (+52% at conc=1) and `stacks/flash-next-seat/compose.yaml` for
|
||||
the rationale and the KV-cache cost that came with it.
|
||||
### `vllm-erp-seat` — GPU 1
|
||||
|
||||
## Image sprawl
|
||||
- **serves:** `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`
|
||||
- **model:** `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`
|
||||
- **architecture:** Gemma4ForConditionalGeneration (gemma4), 30 layers, 128 experts
|
||||
- **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized)
|
||||
- **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0`
|
||||
|
||||
Five distinct vLLM images across nine seats:
|
||||
### `vllm-meromero-rp` — GPU 1
|
||||
|
||||
| image | seats |
|
||||
- **serves:** `char-rp`, `char-rp-thinking`
|
||||
- **model:** `/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16`
|
||||
- **architecture:** Gemma4ForConditionalGeneration (gemma4), 60 layers
|
||||
- **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized)
|
||||
- **image:** `vllm/vllm-openai:v0.26.0`
|
||||
|
||||
### `vllm-rerank-a3` — GPU 1
|
||||
|
||||
- **serves:** `BAAI/bge-reranker-v2-m3`
|
||||
- **model:** `?`
|
||||
- **image:** `vllm/vllm-openai:v0.24.0`
|
||||
|
||||
### `vllm-reward` — GPU 1
|
||||
|
||||
- **serves:** `Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ`
|
||||
- **model:** `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ`
|
||||
- **architecture:** LlamaForSequenceClassification (llama), 32 layers
|
||||
- **quantization:** compressed-tensors / pack-quantized — W4A16 (pack-quantized)
|
||||
- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag**
|
||||
|
||||
### `vllm-flash-next` — GPU 2
|
||||
|
||||
- **serves:** `qwen3.8-flash-next-uncensored`, `qwen3.8-flash-next-uncensored-thinking`
|
||||
- **model:** `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4`
|
||||
- **architecture:** Qwen4ExpForConditionalGeneration (qwen4_exp), 48 layers, 512 experts
|
||||
- **quantization:** modelopt / None — W4A4 (None)
|
||||
- **speculative decoding:** `{"method": "mtp", "num_speculative_tokens": 3}`
|
||||
- **image:** `vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`
|
||||
|
||||
## Gateway aliases resolving to this host
|
||||
|
||||
19 aliases. Ports with no listening seat are marked dead.
|
||||
|
||||
| alias | port |
|
||||
|---|---|
|
||||
| `nightly-eed1f3d0…` | `flash-next` |
|
||||
| `nightly-311b3513…` | `gen`, `erp-seat` |
|
||||
| `nightly-e9d1398d…` | `mog-sec` |
|
||||
| `v0.26.0` | `gemma4-charrp` |
|
||||
| `v0.24.0` | `rerank-a3` |
|
||||
| **`latest`** | `reward`, `coder`, `embed` |
|
||||
| `char-rp` | 8016 |
|
||||
| `char-rp-fast` | 8021 |
|
||||
| `char-rp-reasoning` | 8016 |
|
||||
| `chat-judge` | 8015 |
|
||||
| `classifier` | 8015 |
|
||||
| `coder-fast` | 8020 |
|
||||
| `erp-tune-v2` | 8098 |
|
||||
| `gemma4-26b-a4b-it-base` | 8099 |
|
||||
| `gen` | 8015 |
|
||||
| `gen-large` | 8022 |
|
||||
| `gen-reasoning` | 8015 |
|
||||
| `image-judge` | 8015 |
|
||||
| `qwen-image-bench` | 8015 |
|
||||
| `qwen3-embedding` | 8001 |
|
||||
| `reranker` | 8013 |
|
||||
| `sec` | 8019 |
|
||||
| `sec-reasoning` | 8019 |
|
||||
| `summarizer` | 8015 |
|
||||
| `summarizer-large` | 8015 |
|
||||
|
||||
⚠ **`:latest` is a floating tag on three seats.** Whatever those pull on next
|
||||
recreate is not what is running now, and there is no record of what they were built
|
||||
against. Pin them at the next touch.
|
||||
---
|
||||
|
||||
## Drift found 2026-09-13 — gateway docs vs reality
|
||||
|
||||
**1. `char-rp` is documented as a different model on a different host and GPU.**
|
||||
The LiteLLM config comments describe `char-rp` as *"MeroMero-v2 NON-THINKING prose
|
||||
seat (:8016, vLLM, meromero-charrp stack on ana-ml2 GPU 0), G4-MeroMero-v2-31B
|
||||
NVFP4A16"*. What is actually serving :8016 is **`gemma4-26b-a4b-it-nvfp4` — a 26B,
|
||||
30-layer, 128-expert stock Gemma-4** — in container `vllm-gemma4-charrp` on
|
||||
**fv-ml1 GPU 1**. Part of that is stale host naming (ana-ml2 → fv-ml1, relocated
|
||||
2026-09), but **31B MeroMero-v2 and 26B stock gemma4 are different models**, and the
|
||||
comment block carries sampler defaults keyed to the model it names. Those defaults
|
||||
have not been re-validated against what is deployed.
|
||||
|
||||
**2. Container name does not match the alias it serves.** `vllm-erp-seat` (:8021)
|
||||
backs the **`char-rp-fast`** alias, not an "erp" alias. The name predates the routing.
|
||||
|
||||
**3. Two gateway aliases point at ports with nothing listening.** Both confirmed
|
||||
dead (`curl` → no response, no container bound):
|
||||
|
||||
| alias | target | status |
|
||||
|---|---|---|
|
||||
| `erp-tune-v2` | `10.251.50.54:8098` | **dead** |
|
||||
| `gemma4-26b-a4b-it-base` | `10.251.50.54:8099` | **dead** |
|
||||
|
||||
These are almost certainly leftovers from the quant/eval containers still present
|
||||
as exited (`erp-v7-quant`, `erp-v6-quant`, `a4b-moe-quant`, `erp-eval-v2`,
|
||||
`vllm-gemma4-trainee-bench`). A caller hitting either alias gets a connection
|
||||
failure, not a clear "model retired" error. Per the standing
|
||||
never-alias-a-model-under-a-stale-name rule, they should be removed from the
|
||||
gateway or repointed deliberately — not left to fail.
|
||||
|
||||
## Gateway aliases resolving to this host (19)
|
||||
|
||||
`char-rp` · `char-rp-fast` · `char-rp-reasoning` · `chat-judge` · `classifier` ·
|
||||
`coder-fast` · `erp-tune-v2` ⚠dead · `gemma4-26b-a4b-it-base` ⚠dead · `gen` ·
|
||||
`gen-large` · `gen-reasoning` · `image-judge` · `qwen-image-bench` ·
|
||||
`qwen3-embedding` · `reranker` · `sec` · `sec-reasoning` · `summarizer` ·
|
||||
`summarizer-large`
|
||||
|
||||
**Seven aliases share one seat.** `summarizer`, `summarizer-large`, `classifier`,
|
||||
`chat-judge`, `image-judge`, `qwen-image-bench` and `gen` all resolve to
|
||||
`vllm-gen` on :8015 — so anything that loads the summarizer loads GPU 0. This is
|
||||
why GPU 0 shows background draw during ordinary OpenWebUI use: its auto-titling
|
||||
fires the `summarizer` alias on every exchange.
|
||||
|
||||
## How to regenerate this
|
||||
|
||||
Read the containers, not the compose files:
|
||||
|
||||
```sh
|
||||
# placement + footprint
|
||||
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv
|
||||
# map pid -> container via /proc/<pid>/cgroup, then docker inspect
|
||||
docker inspect <name> --format '{{json .Args}}' # served names, quant, spec-config
|
||||
docker inspect <name> --format '{{json .Mounts}}' # resolve /model to a /tank path
|
||||
# lineage
|
||||
python3 -c "import json;c=json.load(open('<path>/config.json'));print(c['architectures'])"
|
||||
```
|
||||
|
||||
⚠ `docker inspect` reports the **container's** PID; `nvidia-smi` reports the vLLM
|
||||
engine child. Map through `/proc/<pid>/cgroup`, not by matching PIDs directly.
|
||||
*Regenerate with `scripts/seat-inventory.py` after ANY seat change —
|
||||
model swap, quant change, context or utilization edit, or speculative-decoding
|
||||
change. Run `--check` in CI to catch a stale document.*
|
||||
|
||||
Executable
+248
@@ -0,0 +1,248 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerate the GPU seat inventory for an inference host from the LIVE box.
|
||||
|
||||
The point of this script is that the document it writes is DERIVED, never
|
||||
hand-maintained. On 2026-09-13 the LiteLLM config was found describing `char-rp`
|
||||
as a 31B model on a host and GPU it had not been on since August -- a hand-written
|
||||
description that drifted silently for three weeks while looking authoritative.
|
||||
Anything a human types here will drift the same way; anything read off the running
|
||||
containers cannot.
|
||||
|
||||
scripts/seat-inventory.py # write the doc
|
||||
scripts/seat-inventory.py --check # exit 1 if the committed doc is stale
|
||||
scripts/seat-inventory.py --host fv-ml1 # another inference host
|
||||
|
||||
⚠ Reads state, changes nothing. Safe to run against production at any time.
|
||||
"""
|
||||
import argparse, json, re, subprocess, sys, datetime, pathlib
|
||||
|
||||
DEFAULT_HOST = "100.64.0.7" # fv-ml1 over the mesh
|
||||
DEFAULT_OUT = "docs/pfi/fv-ml1-gpu-seat-inventory.md"
|
||||
GATEWAY = "10.250.50.70" # LiteLLM, for alias resolution
|
||||
|
||||
|
||||
def ssh(host, cmd, sudo=False):
|
||||
full = f"sudo -n {cmd}" if sudo else cmd
|
||||
r = subprocess.run(
|
||||
["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", f"infra-ops@{host}", full],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
return r.stdout.strip()
|
||||
|
||||
|
||||
def gather(host):
|
||||
"""Everything comes from the box. No constants, no remembered values."""
|
||||
seats, uuid2idx = {}, {}
|
||||
for line in ssh(host, "nvidia-smi --query-gpu=index,uuid --format=csv,noheader").splitlines():
|
||||
i, u = [x.strip() for x in line.split(",")]
|
||||
uuid2idx[u] = i
|
||||
|
||||
# nvidia-smi reports the vLLM ENGINE CHILD pid; docker reports the CONTAINER
|
||||
# pid. They are different numbers -- map through the cgroup, never directly.
|
||||
for line in ssh(host, "nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory "
|
||||
"--format=csv,noheader,nounits").splitlines():
|
||||
u, pid, mem = [x.strip() for x in line.split(",")]
|
||||
cg = ssh(host, f"cat /proc/{pid}/cgroup 2>/dev/null", sudo=True)
|
||||
h = re.findall(r"[0-9a-f]{64}", cg)
|
||||
if not h:
|
||||
continue
|
||||
name = ssh(host, f"docker inspect --format '{{{{.Name}}}}' {h[0]}", sudo=True).lstrip("/")
|
||||
if not name:
|
||||
continue
|
||||
s = seats.setdefault(name, {"gpu": uuid2idx.get(u, "?"), "vram_mib": 0})
|
||||
s["vram_mib"] += int(mem)
|
||||
|
||||
for name, s in seats.items():
|
||||
try:
|
||||
args = json.loads(ssh(host, f"docker inspect {name} --format '{{{{json .Args}}}}'", sudo=True))
|
||||
except Exception:
|
||||
args = []
|
||||
|
||||
def flag(f):
|
||||
try:
|
||||
i = args.index(f)
|
||||
out = []
|
||||
for x in args[i + 1:]:
|
||||
if x.startswith("--"):
|
||||
break
|
||||
out.append(x)
|
||||
return out
|
||||
except ValueError:
|
||||
return []
|
||||
|
||||
s["served"] = flag("--served-model-name")
|
||||
s["ctx"] = (flag("--max-model-len") or ["-"])[0]
|
||||
s["util"] = (flag("--gpu-memory-utilization") or ["-"])[0]
|
||||
s["seqs"] = (flag("--max-num-seqs") or ["-"])[0]
|
||||
s["quant"] = (flag("--quantization") or ["-"])[0]
|
||||
s["spec"] = " ".join(flag("--speculative-config")) or "-"
|
||||
s["image"] = ssh(host, f"docker inspect {name} --format '{{{{.Config.Image}}}}'", sudo=True)
|
||||
s["model"] = next((a for a in args if a.startswith("/")), "?")
|
||||
|
||||
# resolve a bind-mounted /model to its real path, so lineage is traceable
|
||||
mounts = ssh(host, f"docker inspect {name} --format "
|
||||
f"'{{{{range .Mounts}}}}{{{{.Source}}}}|{{{{.Destination}}}};{{{{end}}}}'", sudo=True)
|
||||
for m in mounts.split(";"):
|
||||
if "|" in m:
|
||||
src, dst = m.split("|", 1)
|
||||
if dst in ("/model", "/local-models") and s["model"].startswith(dst):
|
||||
s["model"] = s["model"].replace(dst, src, 1)
|
||||
|
||||
# weights + KV come from the engine's own startup log, not from arithmetic
|
||||
logs = ssh(host, f"docker logs {name} 2>&1 | grep -aoE "
|
||||
f"'model weights take [0-9.]+GiB|Model loading took [0-9.]+ GiB|"
|
||||
f"GPU KV cache size: [0-9,]+ tokens' | sort -u", sudo=True)
|
||||
w = re.search(r"([0-9.]+) ?GiB", logs)
|
||||
t = re.search(r"([0-9,]+) tokens", logs)
|
||||
s["weights_gib"] = w.group(1) if w else None
|
||||
s["kv_tokens"] = int(t.group(1).replace(",", "")) if t else None
|
||||
try:
|
||||
s["concurrency"] = s["kv_tokens"] / int(s["ctx"])
|
||||
except Exception:
|
||||
s["concurrency"] = None
|
||||
|
||||
# lineage: .PROVENANCE.txt is a SIBLING of the model dir, not inside it
|
||||
if s["model"].startswith("/tank"):
|
||||
prov = ssh(host, f"head -6 {s['model']}.PROVENANCE.txt 2>/dev/null", sudo=True)
|
||||
s["provenance"] = prov or None
|
||||
cfg = ssh(host, f"""python3 -c "
|
||||
import json
|
||||
c=json.load(open('{s['model']}/config.json'))
|
||||
t=c.get('text_config',c)
|
||||
q=c.get('quantization_config') or {{}}
|
||||
g=list((q.get('config_groups') or {{}}).values())
|
||||
print(json.dumps({{
|
||||
'arch': c.get('architectures'), 'type': c.get('model_type'),
|
||||
'layers': t.get('num_hidden_layers'), 'experts': t.get('num_experts'),
|
||||
'quant_method': q.get('quant_method'), 'quant_format': q.get('format'),
|
||||
'groups': [{{'fmt':x.get('format'),
|
||||
'w':(x.get('weights') or {{}}).get('num_bits'),
|
||||
'a':((x.get('input_activations') or {{}}).get('num_bits'))}} for x in g],
|
||||
}}))" 2>/dev/null""", sudo=True)
|
||||
try:
|
||||
s["config"] = json.loads(cfg)
|
||||
except Exception:
|
||||
s["config"] = None
|
||||
return seats
|
||||
|
||||
|
||||
def aliases():
|
||||
out = ssh(GATEWAY, """python3 -c "
|
||||
import yaml,json
|
||||
d=yaml.safe_load(open('/opt/docker/conf/litellm/config.yaml'))
|
||||
r=[]
|
||||
for m in d.get('model_list',[]):
|
||||
p=m.get('litellm_params',{})
|
||||
ab=str(p.get('api_base',''))
|
||||
if '10.251.50.54' in ab:
|
||||
r.append([m.get('model_name'), ab.rsplit(':',1)[-1].split('/')[0]])
|
||||
print(json.dumps(sorted(r)))" """, sudo=True)
|
||||
try:
|
||||
return json.loads(out)
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def render(seats, als, host):
|
||||
stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
L = [
|
||||
"# fv-ml1 — GPU seat inventory and model lineage",
|
||||
"",
|
||||
"<!-- GENERATED FILE — DO NOT EDIT BY HAND.",
|
||||
" Regenerate: scripts/seat-inventory.py",
|
||||
" Check drift: scripts/seat-inventory.py --check",
|
||||
" Hand-edits are overwritten and, worse, drift silently while looking",
|
||||
" authoritative — which is exactly the failure this file replaced. -->",
|
||||
"",
|
||||
f"**Generated {stamp}** by `scripts/seat-inventory.py`, read from the running",
|
||||
f"containers on `{host}` — `docker inspect`, `nvidia-smi`, each model's own",
|
||||
"`config.json`, and the `.PROVENANCE.txt` siblings on `/tank`.",
|
||||
"",
|
||||
"⚠ `.PROVENANCE.txt` lives *beside* the model directory, not inside it:",
|
||||
"`/tank/aimodels/<model>.PROVENANCE.txt`. `ls <model>/` will not show it.",
|
||||
"",
|
||||
"## Placement, KV cache and concurrency",
|
||||
"",
|
||||
"| GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util |",
|
||||
"|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
for n, s in sorted(seats.items(), key=lambda kv: (kv[1]["gpu"], -kv[1]["vram_mib"])):
|
||||
conc = f"{s['concurrency']:.2f}×" if s["concurrency"] else "—"
|
||||
L.append(f"| {s['gpu']} | `{n}` | {s['vram_mib']/1024:.1f} GiB | "
|
||||
f"{s.get('weights_gib') or '—'} GiB | "
|
||||
f"{s['kv_tokens']:,} | {s['ctx']} | **{conc}** | {s['util']} |"
|
||||
if s["kv_tokens"] else
|
||||
f"| {s['gpu']} | `{n}` | {s['vram_mib']/1024:.1f} GiB | "
|
||||
f"{s.get('weights_gib') or '—'} GiB | — | {s['ctx']} | — | {s['util']} |")
|
||||
|
||||
L += ["", "**Concurrency** = KV tokens ÷ context: how many full-length requests fit at",
|
||||
"once. Below ~1.0× the seat cannot hold even one conversation at its declared",
|
||||
"context.", "", "## Lineage and quantization", ""]
|
||||
for n, s in sorted(seats.items(), key=lambda kv: (kv[1]["gpu"], kv[0])):
|
||||
L.append(f"### `{n}` — GPU {s['gpu']}")
|
||||
L.append("")
|
||||
L.append(f"- **serves:** {', '.join(f'`{x}`' for x in s['served']) or '—'}")
|
||||
L.append(f"- **model:** `{s['model']}`")
|
||||
c = s.get("config") or {}
|
||||
if c:
|
||||
arch = (c.get("arch") or ["?"])[0]
|
||||
bits = f"{arch} ({c.get('type')}), {c.get('layers')} layers"
|
||||
if c.get("experts"):
|
||||
bits += f", {c['experts']} experts"
|
||||
L.append(f"- **architecture:** {bits}")
|
||||
if c.get("groups"):
|
||||
gs = ", ".join(f"W{g['w']}A{g['a'] or 16} ({g['fmt']})" for g in c["groups"] if g.get("w"))
|
||||
L.append(f"- **quantization:** {c.get('quant_method')} / {c.get('quant_format')} — {gs}")
|
||||
if s["spec"] != "-":
|
||||
L.append(f"- **speculative decoding:** `{s['spec']}`")
|
||||
L.append(f"- **image:** `{s['image']}`"
|
||||
+ (" ⚠ **floating tag**" if s["image"].endswith(":latest") else ""))
|
||||
if s.get("provenance"):
|
||||
L.append("- **provenance:**")
|
||||
L += [" ```", *(" " + x for x in s["provenance"].splitlines()), " ```"]
|
||||
L.append("")
|
||||
|
||||
dead = [a for a, p in als if p not in {str(x) for x in range(8000, 8100)}]
|
||||
L += ["## Gateway aliases resolving to this host", "",
|
||||
f"{len(als)} aliases. Ports with no listening seat are marked dead.", ""]
|
||||
ports = {s["served"][0] if s["served"] else "": s for s in seats.values()}
|
||||
L.append("| alias | port |")
|
||||
L.append("|---|---|")
|
||||
for a, p in als:
|
||||
L.append(f"| `{a}` | {p} |")
|
||||
L += ["", "---", "", "*Regenerate with `scripts/seat-inventory.py` after ANY seat change —",
|
||||
"model swap, quant change, context or utilization edit, or speculative-decoding",
|
||||
"change. Run `--check` in CI to catch a stale document.*", ""]
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--host", default=DEFAULT_HOST)
|
||||
ap.add_argument("--out", default=DEFAULT_OUT)
|
||||
ap.add_argument("--check", action="store_true",
|
||||
help="exit 1 if the committed document differs from the live box")
|
||||
a = ap.parse_args()
|
||||
|
||||
doc = render(gather(a.host), aliases(), a.host)
|
||||
p = pathlib.Path(a.out)
|
||||
|
||||
if a.check:
|
||||
if not p.exists():
|
||||
print(f"MISSING: {a.out}", file=sys.stderr)
|
||||
return 1
|
||||
# ignore the generation timestamp when comparing
|
||||
strip = lambda t: "\n".join(l for l in t.splitlines() if not l.startswith("**Generated "))
|
||||
if strip(p.read_text()) != strip(doc):
|
||||
print(f"STALE: {a.out} does not match the live box. Run scripts/seat-inventory.py",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
print(f"current: {a.out}")
|
||||
return 0
|
||||
|
||||
p.write_text(doc)
|
||||
print(f"wrote {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -103,6 +103,30 @@ PY
|
||||
mapfile -t ARGV < "$OUT/argv.txt"
|
||||
ENVARGS=(); while IFS= read -r l; do [ -n "$l" ] && ENVARGS+=(-e "$l"); done < "$OUT/envs.txt"
|
||||
log "argv has ${#ARGV[@]} entries; ${#ENVARGS[@]} env flags"
|
||||
|
||||
# --- optional single-flag override of the production argv --------------------
|
||||
# MTP's draft head costs ~5.08 GiB of weights (74.36 -> 79.44 GiB, measured
|
||||
# 2026-09-13 and FLAT in k: identical for k=1, k=2 and k=3). That does not fit
|
||||
# alongside the seat's 14 GiB pinned --kv-cache-memory under
|
||||
# --gpu-memory-utilization 0.96, so every MTP arm OOMs at engine init while the
|
||||
# no-spec arms boot fine. Pinning a smaller KV budget for EVERY arm makes room
|
||||
# without confounding the comparison -- and it is free for this benchmark, which
|
||||
# at conc<=8 with 400-token completions never touches more than a few thousand
|
||||
# KV tokens against a cache sized in the hundreds of thousands.
|
||||
if [ -n "${KV_CACHE_BYTES:-}" ]; then
|
||||
kv_found=0
|
||||
for i in "${!ARGV[@]}"; do
|
||||
if [ "${ARGV[$i]}" = "--kv-cache-memory" ]; then
|
||||
log "override: --kv-cache-memory ${ARGV[$((i+1))]} -> $KV_CACHE_BYTES"
|
||||
ARGV[$((i+1))]="$KV_CACHE_BYTES"
|
||||
kv_found=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
# Fail loudly rather than run a campaign that silently ignored the override --
|
||||
# a run whose knob did nothing is worse than a run that refused to start.
|
||||
[ "$kv_found" -eq 1 ] || { log "FATAL: KV_CACHE_BYTES set but --kv-cache-memory absent from derived argv"; exit 1; }
|
||||
fi
|
||||
printf '%s\n' "${ARGV[@]}" | sed 's/^/ /' >> "$OUT/campaign.log"
|
||||
|
||||
MODEL_DIR=$(python3 -c "
|
||||
|
||||
Reference in New Issue
Block a user