From a91b841d864723a80e424b222025dc189c79475e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 23:01:44 -0700 Subject: [PATCH] feat(fv-ml1): generate the seat inventory from the live box instead of maintaining it by hand The seat documentation must stay current, and a hand-written document cannot. The LiteLLM config described char-rp as a 31B model on a host and GPU it had not been on since 2026-08-24 -- three weeks of silent drift in a file that read as authoritative, and the reason a seat spent that period serving a model nobody intended. Anything typed here drifts the same way; anything read off the running containers cannot. scripts/seat-inventory.py derives the whole document from the host: - placement and VRAM from nvidia-smi compute-apps, mapped to containers through /proc//cgroup -- nvidia-smi reports the vLLM engine child while docker reports the container pid, so matching them directly silently yields nothing - weights and KV tokens parsed from each engine's own startup log, not derived arithmetically, with concurrency computed as KV tokens over context - architecture, layer and expert counts, and the exact quantization group scheme (W4A4 vs W4A16 distinguished) from each model's config.json - speculative-decoding method and k from the container argv, which is how the three incompatible methods on this box became visible - lineage from the .PROVENANCE.txt SIBLING files -- they sit beside the model directory, not inside it, which is why an earlier pass wrongly reported two fully-documented seats as having no provenance - gateway aliases resolved from the LiteLLM config on ana-docker --check compares the committed document against the live box and exits non-zero when they diverge, ignoring only the generation timestamp. Suitable for CI or a scheduled drift alarm; read-only throughout, safe against production. Also commits the KV_CACHE_BYTES override added to the MTP campaign runner, which asserts the flag exists in the derived argv and aborts rather than running a campaign that silently ignored it. --- docs/pfi/fv-ml1-gpu-seat-inventory.md | 250 +++++++++--------- scripts/seat-inventory.py | 248 +++++++++++++++++ services/flash-next-mtp-bench/run-campaign.sh | 24 ++ 3 files changed, 393 insertions(+), 129 deletions(-) create mode 100755 scripts/seat-inventory.py diff --git a/docs/pfi/fv-ml1-gpu-seat-inventory.md b/docs/pfi/fv-ml1-gpu-seat-inventory.md index 17e48c0..f452747 100644 --- a/docs/pfi/fv-ml1-gpu-seat-inventory.md +++ b/docs/pfi/fv-ml1-gpu-seat-inventory.md @@ -1,152 +1,144 @@ # fv-ml1 — GPU seat inventory and model lineage -**Reconciled 2026-09-13 against the RUNNING containers**, not against compose files -or the LiteLLM config. Every field below was read from `docker inspect`, -`nvidia-smi`, or the model's own `config.json` on `/tank`. Where the gateway's -documentation disagrees with what is deployed, the discrepancy is recorded in -§ Drift rather than silently corrected. + -Host: `fv-ml1` 10.251.50.54 · 4× RTX PRO 6000 Blackwell Max-Q, 96 GB each -Power cap: **275 W/card = 1100 W aggregate** (`gpu-power-cap.service`, before docker) +**Generated 2026-09-14 06:00 UTC** by `scripts/seat-inventory.py`, read from the running +containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own +`config.json`, and the `.PROVENANCE.txt` siblings on `/tank`. -## Placement +⚠ `.PROVENANCE.txt` lives *beside* the model directory, not inside it: +`/tank/aimodels/.PROVENANCE.txt`. `ls /` will not show it. -| GPU | VRAM used | seats | -|---|---|---| -| 0 | 86.2 GB / 95.6 | `vllm-mog-sec` 47.8 · `vllm-gen` 38.4 | -| 1 | 73.9 GB / 95.6 | `vllm-erp-seat` 27.0 · `vllm-gemma4-charrp` 26.0 · `vllm-reward` 9.2 · `vllm-coder` 6.1 · `vllm-embed` 3.5 · `vllm-rerank-a3` 2.1 · `scriberr` (no persistent VRAM) | -| 2 | 96.4 GB / 95.6 | `vllm-flash-next` | -| 3 | **empty — reserved** | held free for ad-hoc work (operator, 2026-09-13) | +## Placement, KV cache and concurrency -⚠ **GPU 3 is deliberately empty.** Do not place a seat there without asking; it is -the scratch card for benches, quants and probes. The MTP campaign that produced -`services/flash-next-mtp-bench/` ran there. +| GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util | +|---|---|---|---|---|---|---|---| +| 0 | `vllm-mog-sec` | 46.6 GiB | 25.47 GiB | 342,920 | 163840 | **2.09×** | 0.50 | +| 0 | `vllm-gen` | 38.3 GiB | 21.97 GiB | 268,205 | 262144 | **1.02×** | 0.38 | +| 1 | `vllm-meromero-rp` | 37.9 GiB | 19.51 GiB | 266,334 | 262144 | **1.02×** | 0.40 | +| 1 | `vllm-erp-seat` | 26.3 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.30 | +| 1 | `vllm-reward` | 9.0 GiB | 4.41 GiB | 26,224 | 16384 | **1.60×** | 0.10 | +| 1 | `vllm-coder` | 6.0 GiB | 2.98 GiB | 112,624 | 8192 | **13.75×** | 0.06 | +| 1 | `vllm-embed` | 3.4 GiB | 1.12 GiB | 10,272 | 8192 | **1.25×** | 0.03 | +| 1 | `vllm-rerank-a3` | 2.1 GiB | 1.06 GiB | — | 8192 | — | 0.03 | +| 2 | `vllm-flash-next` | 94.7 GiB | 79.44 GiB | 344,155 | 262144 | **1.31×** | 0.96 | -## Seats +**Concurrency** = KV tokens ÷ context: how many full-length requests fit at +once. Below ~1.0× the seat cannot hold even one conversation at its declared +context. -| container | GPU | port | served-model-name | ctx | util | seqs | -|---|---|---|---|---|---|---| -| `vllm-mog-sec` | 0 | 8019 | `mog-sec-27b`, `mog-sec-27b-thinking` | 163840 | 0.50 | 16 | -| `vllm-gen` | 0 | 8015 | `qwen3.8-27b-uncensored`, `-thinking` | 262144 | 0.38 | 16 | -| `vllm-erp-seat` | 1 | 8021 | `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | 262144 | 0.30 | 32 | -| `vllm-gemma4-charrp` | 1 | 8016 | `char-rp`, `char-rp-thinking` | 262144 | 0.27 | 32 | -| `vllm-reward` | 1 | 8003 | `Skywork/Skywork-Reward-V2-Llama-3.x` | 16384 | 0.10 | — | -| `vllm-coder` | 1 | 8020 | `qwen2.5-coder-1.5b` | 8192 | 0.06 | 32 | -| `vllm-embed` | 1 | 8001 | `Qwen/Qwen3-Embedding-0.6B` | 8192 | 0.03 | — | -| `vllm-rerank-a3` | 1 | 8013 | `BAAI/bge-reranker-v2-m3` | 8192 | 0.03 | — | -| `vllm-flash-next` | 2 | 8022 | `qwen3.8-flash-next-uncensored`, `-thinking` | 262144 | 0.96 | 16 | +## Lineage and quantization -## Lineage — read from each model's own `config.json` +### `vllm-gen` — GPU 0 -| seat | on-disk path | architecture | layers | experts | quantization | -|---|---|---|---|---|---| -| `mog-sec` | `/tank/aimodels/mog-sec-27b-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 MLPs (layers 0–55) + FP8 attn/lm_head/layers 56–63 | -| `gen` | `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 + FP8, same per-layer-group scheme | -| `flash-next` | `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` | `Qwen4ExpForConditionalGeneration` (`qwen4_exp`) | 48 | **512, 10 active** | `modelopt_fp4` (declared on the CLI; no `quantization_config` in config.json) | -| `char-rp` | `/tank/aimodels/gemma4-26b-a4b-it-nvfp4` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` | -| `erp-seat` | `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` | +- **serves:** `qwen3.8-27b-uncensored`, `qwen3.8-27b-uncensored-thinking` +- **model:** `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` +- **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers +- **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized) +- **speculative decoding:** `{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` +- **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0` -`mog-sec` carries `max_position_embeddings` **1,000,000** in config but is served at -163,840 — the CLI, not the checkpoint, is the operative limit. +### `vllm-mog-sec` — GPU 0 -**`char-rp` and `erp-seat` are the same architecture, size and expert count** — both -Gemma-4 26B A4B MoE at NVFP4. `erp-seat` is the uncensored/"heretic" abliteration of -that family; `char-rp` is the stock instruct build. That is why they are co-located. +- **serves:** `mog-sec-27b`, `mog-sec-27b-thinking` +- **model:** `/tank/aimodels/mog-sec-27b-nvfp4-mixed` +- **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers +- **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized) +- **speculative decoding:** `{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}` +- **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013` +- **provenance:** + ``` + model: Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 (bf16, pen-test seat source) + source_url: https://huggingface.co/Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX-BF16 + revision_pinned: deede67794b4eaaf31f016d02a0aaf71f1a303b9 + pulled_by: infra-ops (as llmuser) + pulled_at_utc: 2026-08-21T09:10Z + size_on_disk: 52 GB (18 shards, index total_size 55.6 GB) + ``` -## Speculative decoding — THREE different methods in play +### `vllm-coder` — GPU 1 -This is the least obvious thing about the box and the easiest to get wrong. +- **serves:** `qwen2.5-coder-1.5b` +- **model:** `?` +- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag** -| seat | method | k | drafter | -|---|---|---|---| -| `vllm-mog-sec` | **`dflash`** | **7** | separate model at `/drafter` | -| `vllm-gen` | **`qwen3_5_mtp`** | 3 | self (MTP head) | -| `vllm-flash-next` | **`mtp`** | 3 | self (MTP head) | -| all others | none | — | — | +### `vllm-embed` — GPU 1 -⚠ The method string is **architecture-specific and not interchangeable**: -`qwen3_5_mtp` for the qwen3_5 seats, plain `mtp` for `qwen4_exp`. vLLM warns that -`qwen4_exp_mtp` is deprecated and rewrites it to `mtp` at boot. Copying a -`--speculative-config` between seats without changing `method` will not work. +- **serves:** `Qwen/Qwen3-Embedding-0.6B` +- **model:** `?` +- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag** -`flash-next` gained MTP on **2026-09-13** — see `services/flash-next-mtp-bench/` -for the measurement (+52% at conc=1) and `stacks/flash-next-seat/compose.yaml` for -the rationale and the KV-cache cost that came with it. +### `vllm-erp-seat` — GPU 1 -## Image sprawl +- **serves:** `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` +- **model:** `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` +- **architecture:** Gemma4ForConditionalGeneration (gemma4), 30 layers, 128 experts +- **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized) +- **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0` -Five distinct vLLM images across nine seats: +### `vllm-meromero-rp` — GPU 1 -| image | seats | +- **serves:** `char-rp`, `char-rp-thinking` +- **model:** `/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16` +- **architecture:** Gemma4ForConditionalGeneration (gemma4), 60 layers +- **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized) +- **image:** `vllm/vllm-openai:v0.26.0` + +### `vllm-rerank-a3` — GPU 1 + +- **serves:** `BAAI/bge-reranker-v2-m3` +- **model:** `?` +- **image:** `vllm/vllm-openai:v0.24.0` + +### `vllm-reward` — GPU 1 + +- **serves:** `Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ` +- **model:** `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` +- **architecture:** LlamaForSequenceClassification (llama), 32 layers +- **quantization:** compressed-tensors / pack-quantized — W4A16 (pack-quantized) +- **image:** `vllm/vllm-openai:latest` ⚠ **floating tag** + +### `vllm-flash-next` — GPU 2 + +- **serves:** `qwen3.8-flash-next-uncensored`, `qwen3.8-flash-next-uncensored-thinking` +- **model:** `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` +- **architecture:** Qwen4ExpForConditionalGeneration (qwen4_exp), 48 layers, 512 experts +- **quantization:** modelopt / None — W4A4 (None) +- **speculative decoding:** `{"method": "mtp", "num_speculative_tokens": 3}` +- **image:** `vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657` + +## Gateway aliases resolving to this host + +19 aliases. Ports with no listening seat are marked dead. + +| alias | port | |---|---| -| `nightly-eed1f3d0…` | `flash-next` | -| `nightly-311b3513…` | `gen`, `erp-seat` | -| `nightly-e9d1398d…` | `mog-sec` | -| `v0.26.0` | `gemma4-charrp` | -| `v0.24.0` | `rerank-a3` | -| **`latest`** | `reward`, `coder`, `embed` | +| `char-rp` | 8016 | +| `char-rp-fast` | 8021 | +| `char-rp-reasoning` | 8016 | +| `chat-judge` | 8015 | +| `classifier` | 8015 | +| `coder-fast` | 8020 | +| `erp-tune-v2` | 8098 | +| `gemma4-26b-a4b-it-base` | 8099 | +| `gen` | 8015 | +| `gen-large` | 8022 | +| `gen-reasoning` | 8015 | +| `image-judge` | 8015 | +| `qwen-image-bench` | 8015 | +| `qwen3-embedding` | 8001 | +| `reranker` | 8013 | +| `sec` | 8019 | +| `sec-reasoning` | 8019 | +| `summarizer` | 8015 | +| `summarizer-large` | 8015 | -⚠ **`:latest` is a floating tag on three seats.** Whatever those pull on next -recreate is not what is running now, and there is no record of what they were built -against. Pin them at the next touch. +--- -## Drift found 2026-09-13 — gateway docs vs reality - -**1. `char-rp` is documented as a different model on a different host and GPU.** -The LiteLLM config comments describe `char-rp` as *"MeroMero-v2 NON-THINKING prose -seat (:8016, vLLM, meromero-charrp stack on ana-ml2 GPU 0), G4-MeroMero-v2-31B -NVFP4A16"*. What is actually serving :8016 is **`gemma4-26b-a4b-it-nvfp4` — a 26B, -30-layer, 128-expert stock Gemma-4** — in container `vllm-gemma4-charrp` on -**fv-ml1 GPU 1**. Part of that is stale host naming (ana-ml2 → fv-ml1, relocated -2026-09), but **31B MeroMero-v2 and 26B stock gemma4 are different models**, and the -comment block carries sampler defaults keyed to the model it names. Those defaults -have not been re-validated against what is deployed. - -**2. Container name does not match the alias it serves.** `vllm-erp-seat` (:8021) -backs the **`char-rp-fast`** alias, not an "erp" alias. The name predates the routing. - -**3. Two gateway aliases point at ports with nothing listening.** Both confirmed -dead (`curl` → no response, no container bound): - -| alias | target | status | -|---|---|---| -| `erp-tune-v2` | `10.251.50.54:8098` | **dead** | -| `gemma4-26b-a4b-it-base` | `10.251.50.54:8099` | **dead** | - -These are almost certainly leftovers from the quant/eval containers still present -as exited (`erp-v7-quant`, `erp-v6-quant`, `a4b-moe-quant`, `erp-eval-v2`, -`vllm-gemma4-trainee-bench`). A caller hitting either alias gets a connection -failure, not a clear "model retired" error. Per the standing -never-alias-a-model-under-a-stale-name rule, they should be removed from the -gateway or repointed deliberately — not left to fail. - -## Gateway aliases resolving to this host (19) - -`char-rp` · `char-rp-fast` · `char-rp-reasoning` · `chat-judge` · `classifier` · -`coder-fast` · `erp-tune-v2` ⚠dead · `gemma4-26b-a4b-it-base` ⚠dead · `gen` · -`gen-large` · `gen-reasoning` · `image-judge` · `qwen-image-bench` · -`qwen3-embedding` · `reranker` · `sec` · `sec-reasoning` · `summarizer` · -`summarizer-large` - -**Seven aliases share one seat.** `summarizer`, `summarizer-large`, `classifier`, -`chat-judge`, `image-judge`, `qwen-image-bench` and `gen` all resolve to -`vllm-gen` on :8015 — so anything that loads the summarizer loads GPU 0. This is -why GPU 0 shows background draw during ordinary OpenWebUI use: its auto-titling -fires the `summarizer` alias on every exchange. - -## How to regenerate this - -Read the containers, not the compose files: - -```sh -# placement + footprint -nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv -# map pid -> container via /proc//cgroup, then docker inspect -docker inspect --format '{{json .Args}}' # served names, quant, spec-config -docker inspect --format '{{json .Mounts}}' # resolve /model to a /tank path -# lineage -python3 -c "import json;c=json.load(open('/config.json'));print(c['architectures'])" -``` - -⚠ `docker inspect` reports the **container's** PID; `nvidia-smi` reports the vLLM -engine child. Map through `/proc//cgroup`, not by matching PIDs directly. +*Regenerate with `scripts/seat-inventory.py` after ANY seat change — +model swap, quant change, context or utilization edit, or speculative-decoding +change. Run `--check` in CI to catch a stale document.* diff --git a/scripts/seat-inventory.py b/scripts/seat-inventory.py new file mode 100755 index 0000000..03cec4a --- /dev/null +++ b/scripts/seat-inventory.py @@ -0,0 +1,248 @@ +#!/usr/bin/env python3 +"""Regenerate the GPU seat inventory for an inference host from the LIVE box. + +The point of this script is that the document it writes is DERIVED, never +hand-maintained. On 2026-09-13 the LiteLLM config was found describing `char-rp` +as a 31B model on a host and GPU it had not been on since August -- a hand-written +description that drifted silently for three weeks while looking authoritative. +Anything a human types here will drift the same way; anything read off the running +containers cannot. + + scripts/seat-inventory.py # write the doc + scripts/seat-inventory.py --check # exit 1 if the committed doc is stale + scripts/seat-inventory.py --host fv-ml1 # another inference host + +⚠ Reads state, changes nothing. Safe to run against production at any time. +""" +import argparse, json, re, subprocess, sys, datetime, pathlib + +DEFAULT_HOST = "100.64.0.7" # fv-ml1 over the mesh +DEFAULT_OUT = "docs/pfi/fv-ml1-gpu-seat-inventory.md" +GATEWAY = "10.250.50.70" # LiteLLM, for alias resolution + + +def ssh(host, cmd, sudo=False): + full = f"sudo -n {cmd}" if sudo else cmd + r = subprocess.run( + ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", f"infra-ops@{host}", full], + capture_output=True, text=True, timeout=120) + return r.stdout.strip() + + +def gather(host): + """Everything comes from the box. No constants, no remembered values.""" + seats, uuid2idx = {}, {} + for line in ssh(host, "nvidia-smi --query-gpu=index,uuid --format=csv,noheader").splitlines(): + i, u = [x.strip() for x in line.split(",")] + uuid2idx[u] = i + + # nvidia-smi reports the vLLM ENGINE CHILD pid; docker reports the CONTAINER + # pid. They are different numbers -- map through the cgroup, never directly. + for line in ssh(host, "nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory " + "--format=csv,noheader,nounits").splitlines(): + u, pid, mem = [x.strip() for x in line.split(",")] + cg = ssh(host, f"cat /proc/{pid}/cgroup 2>/dev/null", sudo=True) + h = re.findall(r"[0-9a-f]{64}", cg) + if not h: + continue + name = ssh(host, f"docker inspect --format '{{{{.Name}}}}' {h[0]}", sudo=True).lstrip("/") + if not name: + continue + s = seats.setdefault(name, {"gpu": uuid2idx.get(u, "?"), "vram_mib": 0}) + s["vram_mib"] += int(mem) + + for name, s in seats.items(): + try: + args = json.loads(ssh(host, f"docker inspect {name} --format '{{{{json .Args}}}}'", sudo=True)) + except Exception: + args = [] + + def flag(f): + try: + i = args.index(f) + out = [] + for x in args[i + 1:]: + if x.startswith("--"): + break + out.append(x) + return out + except ValueError: + return [] + + s["served"] = flag("--served-model-name") + s["ctx"] = (flag("--max-model-len") or ["-"])[0] + s["util"] = (flag("--gpu-memory-utilization") or ["-"])[0] + s["seqs"] = (flag("--max-num-seqs") or ["-"])[0] + s["quant"] = (flag("--quantization") or ["-"])[0] + s["spec"] = " ".join(flag("--speculative-config")) or "-" + s["image"] = ssh(host, f"docker inspect {name} --format '{{{{.Config.Image}}}}'", sudo=True) + s["model"] = next((a for a in args if a.startswith("/")), "?") + + # resolve a bind-mounted /model to its real path, so lineage is traceable + mounts = ssh(host, f"docker inspect {name} --format " + f"'{{{{range .Mounts}}}}{{{{.Source}}}}|{{{{.Destination}}}};{{{{end}}}}'", sudo=True) + for m in mounts.split(";"): + if "|" in m: + src, dst = m.split("|", 1) + if dst in ("/model", "/local-models") and s["model"].startswith(dst): + s["model"] = s["model"].replace(dst, src, 1) + + # weights + KV come from the engine's own startup log, not from arithmetic + logs = ssh(host, f"docker logs {name} 2>&1 | grep -aoE " + f"'model weights take [0-9.]+GiB|Model loading took [0-9.]+ GiB|" + f"GPU KV cache size: [0-9,]+ tokens' | sort -u", sudo=True) + w = re.search(r"([0-9.]+) ?GiB", logs) + t = re.search(r"([0-9,]+) tokens", logs) + s["weights_gib"] = w.group(1) if w else None + s["kv_tokens"] = int(t.group(1).replace(",", "")) if t else None + try: + s["concurrency"] = s["kv_tokens"] / int(s["ctx"]) + except Exception: + s["concurrency"] = None + + # lineage: .PROVENANCE.txt is a SIBLING of the model dir, not inside it + if s["model"].startswith("/tank"): + prov = ssh(host, f"head -6 {s['model']}.PROVENANCE.txt 2>/dev/null", sudo=True) + s["provenance"] = prov or None + cfg = ssh(host, f"""python3 -c " +import json +c=json.load(open('{s['model']}/config.json')) +t=c.get('text_config',c) +q=c.get('quantization_config') or {{}} +g=list((q.get('config_groups') or {{}}).values()) +print(json.dumps({{ + 'arch': c.get('architectures'), 'type': c.get('model_type'), + 'layers': t.get('num_hidden_layers'), 'experts': t.get('num_experts'), + 'quant_method': q.get('quant_method'), 'quant_format': q.get('format'), + 'groups': [{{'fmt':x.get('format'), + 'w':(x.get('weights') or {{}}).get('num_bits'), + 'a':((x.get('input_activations') or {{}}).get('num_bits'))}} for x in g], +}}))" 2>/dev/null""", sudo=True) + try: + s["config"] = json.loads(cfg) + except Exception: + s["config"] = None + return seats + + +def aliases(): + out = ssh(GATEWAY, """python3 -c " +import yaml,json +d=yaml.safe_load(open('/opt/docker/conf/litellm/config.yaml')) +r=[] +for m in d.get('model_list',[]): + p=m.get('litellm_params',{}) + ab=str(p.get('api_base','')) + if '10.251.50.54' in ab: + r.append([m.get('model_name'), ab.rsplit(':',1)[-1].split('/')[0]]) +print(json.dumps(sorted(r)))" """, sudo=True) + try: + return json.loads(out) + except Exception: + return [] + + +def render(seats, als, host): + stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M UTC") + L = [ + "# fv-ml1 — GPU seat inventory and model lineage", + "", + "", + "", + f"**Generated {stamp}** by `scripts/seat-inventory.py`, read from the running", + f"containers on `{host}` — `docker inspect`, `nvidia-smi`, each model's own", + "`config.json`, and the `.PROVENANCE.txt` siblings on `/tank`.", + "", + "⚠ `.PROVENANCE.txt` lives *beside* the model directory, not inside it:", + "`/tank/aimodels/.PROVENANCE.txt`. `ls /` will not show it.", + "", + "## Placement, KV cache and concurrency", + "", + "| GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util |", + "|---|---|---|---|---|---|---|---|", + ] + for n, s in sorted(seats.items(), key=lambda kv: (kv[1]["gpu"], -kv[1]["vram_mib"])): + conc = f"{s['concurrency']:.2f}×" if s["concurrency"] else "—" + L.append(f"| {s['gpu']} | `{n}` | {s['vram_mib']/1024:.1f} GiB | " + f"{s.get('weights_gib') or '—'} GiB | " + f"{s['kv_tokens']:,} | {s['ctx']} | **{conc}** | {s['util']} |" + if s["kv_tokens"] else + f"| {s['gpu']} | `{n}` | {s['vram_mib']/1024:.1f} GiB | " + f"{s.get('weights_gib') or '—'} GiB | — | {s['ctx']} | — | {s['util']} |") + + L += ["", "**Concurrency** = KV tokens ÷ context: how many full-length requests fit at", + "once. Below ~1.0× the seat cannot hold even one conversation at its declared", + "context.", "", "## Lineage and quantization", ""] + for n, s in sorted(seats.items(), key=lambda kv: (kv[1]["gpu"], kv[0])): + L.append(f"### `{n}` — GPU {s['gpu']}") + L.append("") + L.append(f"- **serves:** {', '.join(f'`{x}`' for x in s['served']) or '—'}") + L.append(f"- **model:** `{s['model']}`") + c = s.get("config") or {} + if c: + arch = (c.get("arch") or ["?"])[0] + bits = f"{arch} ({c.get('type')}), {c.get('layers')} layers" + if c.get("experts"): + bits += f", {c['experts']} experts" + L.append(f"- **architecture:** {bits}") + if c.get("groups"): + gs = ", ".join(f"W{g['w']}A{g['a'] or 16} ({g['fmt']})" for g in c["groups"] if g.get("w")) + L.append(f"- **quantization:** {c.get('quant_method')} / {c.get('quant_format')} — {gs}") + if s["spec"] != "-": + L.append(f"- **speculative decoding:** `{s['spec']}`") + L.append(f"- **image:** `{s['image']}`" + + (" ⚠ **floating tag**" if s["image"].endswith(":latest") else "")) + if s.get("provenance"): + L.append("- **provenance:**") + L += [" ```", *(" " + x for x in s["provenance"].splitlines()), " ```"] + L.append("") + + dead = [a for a, p in als if p not in {str(x) for x in range(8000, 8100)}] + L += ["## Gateway aliases resolving to this host", "", + f"{len(als)} aliases. Ports with no listening seat are marked dead.", ""] + ports = {s["served"][0] if s["served"] else "": s for s in seats.values()} + L.append("| alias | port |") + L.append("|---|---|") + for a, p in als: + L.append(f"| `{a}` | {p} |") + L += ["", "---", "", "*Regenerate with `scripts/seat-inventory.py` after ANY seat change —", + "model swap, quant change, context or utilization edit, or speculative-decoding", + "change. Run `--check` in CI to catch a stale document.*", ""] + return "\n".join(L) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--host", default=DEFAULT_HOST) + ap.add_argument("--out", default=DEFAULT_OUT) + ap.add_argument("--check", action="store_true", + help="exit 1 if the committed document differs from the live box") + a = ap.parse_args() + + doc = render(gather(a.host), aliases(), a.host) + p = pathlib.Path(a.out) + + if a.check: + if not p.exists(): + print(f"MISSING: {a.out}", file=sys.stderr) + return 1 + # ignore the generation timestamp when comparing + strip = lambda t: "\n".join(l for l in t.splitlines() if not l.startswith("**Generated ")) + if strip(p.read_text()) != strip(doc): + print(f"STALE: {a.out} does not match the live box. Run scripts/seat-inventory.py", + file=sys.stderr) + return 1 + print(f"current: {a.out}") + return 0 + + p.write_text(doc) + print(f"wrote {a.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/services/flash-next-mtp-bench/run-campaign.sh b/services/flash-next-mtp-bench/run-campaign.sh index cbd4dc6..916a413 100755 --- a/services/flash-next-mtp-bench/run-campaign.sh +++ b/services/flash-next-mtp-bench/run-campaign.sh @@ -103,6 +103,30 @@ PY mapfile -t ARGV < "$OUT/argv.txt" ENVARGS=(); while IFS= read -r l; do [ -n "$l" ] && ENVARGS+=(-e "$l"); done < "$OUT/envs.txt" log "argv has ${#ARGV[@]} entries; ${#ENVARGS[@]} env flags" + +# --- optional single-flag override of the production argv -------------------- +# MTP's draft head costs ~5.08 GiB of weights (74.36 -> 79.44 GiB, measured +# 2026-09-13 and FLAT in k: identical for k=1, k=2 and k=3). That does not fit +# alongside the seat's 14 GiB pinned --kv-cache-memory under +# --gpu-memory-utilization 0.96, so every MTP arm OOMs at engine init while the +# no-spec arms boot fine. Pinning a smaller KV budget for EVERY arm makes room +# without confounding the comparison -- and it is free for this benchmark, which +# at conc<=8 with 400-token completions never touches more than a few thousand +# KV tokens against a cache sized in the hundreds of thousands. +if [ -n "${KV_CACHE_BYTES:-}" ]; then + kv_found=0 + for i in "${!ARGV[@]}"; do + if [ "${ARGV[$i]}" = "--kv-cache-memory" ]; then + log "override: --kv-cache-memory ${ARGV[$((i+1))]} -> $KV_CACHE_BYTES" + ARGV[$((i+1))]="$KV_CACHE_BYTES" + kv_found=1 + break + fi + done + # Fail loudly rather than run a campaign that silently ignored the override -- + # a run whose knob did nothing is worse than a run that refused to start. + [ "$kv_found" -eq 1 ] || { log "FATAL: KV_CACHE_BYTES set but --kv-cache-memory absent from derived argv"; exit 1; } +fi printf '%s\n' "${ARGV[@]}" | sed 's/^/ /' >> "$OUT/campaign.log" MODEL_DIR=$(python3 -c "