From 2d83a895c16ded1d8b40629bd041f80100facf7e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 22:28:58 -0700 Subject: [PATCH] docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production). --- docs/pfi/fv-ml1-gpu-seat-inventory.md | 152 ++++++++++++++++++++++++++ stacks/flash-next-seat/.env.example | 9 +- stacks/flash-next-seat/compose.yaml | 42 +++++-- 3 files changed, 194 insertions(+), 9 deletions(-) create mode 100644 docs/pfi/fv-ml1-gpu-seat-inventory.md diff --git a/docs/pfi/fv-ml1-gpu-seat-inventory.md b/docs/pfi/fv-ml1-gpu-seat-inventory.md new file mode 100644 index 0000000..17e48c0 --- /dev/null +++ b/docs/pfi/fv-ml1-gpu-seat-inventory.md @@ -0,0 +1,152 @@ +# fv-ml1 — GPU seat inventory and model lineage + +**Reconciled 2026-09-13 against the RUNNING containers**, not against compose files +or the LiteLLM config. Every field below was read from `docker inspect`, +`nvidia-smi`, or the model's own `config.json` on `/tank`. Where the gateway's +documentation disagrees with what is deployed, the discrepancy is recorded in +§ Drift rather than silently corrected. + +Host: `fv-ml1` 10.251.50.54 · 4× RTX PRO 6000 Blackwell Max-Q, 96 GB each +Power cap: **275 W/card = 1100 W aggregate** (`gpu-power-cap.service`, before docker) + +## Placement + +| GPU | VRAM used | seats | +|---|---|---| +| 0 | 86.2 GB / 95.6 | `vllm-mog-sec` 47.8 · `vllm-gen` 38.4 | +| 1 | 73.9 GB / 95.6 | `vllm-erp-seat` 27.0 · `vllm-gemma4-charrp` 26.0 · `vllm-reward` 9.2 · `vllm-coder` 6.1 · `vllm-embed` 3.5 · `vllm-rerank-a3` 2.1 · `scriberr` (no persistent VRAM) | +| 2 | 96.4 GB / 95.6 | `vllm-flash-next` | +| 3 | **empty — reserved** | held free for ad-hoc work (operator, 2026-09-13) | + +⚠ **GPU 3 is deliberately empty.** Do not place a seat there without asking; it is +the scratch card for benches, quants and probes. The MTP campaign that produced +`services/flash-next-mtp-bench/` ran there. + +## Seats + +| container | GPU | port | served-model-name | ctx | util | seqs | +|---|---|---|---|---|---|---| +| `vllm-mog-sec` | 0 | 8019 | `mog-sec-27b`, `mog-sec-27b-thinking` | 163840 | 0.50 | 16 | +| `vllm-gen` | 0 | 8015 | `qwen3.8-27b-uncensored`, `-thinking` | 262144 | 0.38 | 16 | +| `vllm-erp-seat` | 1 | 8021 | `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | 262144 | 0.30 | 32 | +| `vllm-gemma4-charrp` | 1 | 8016 | `char-rp`, `char-rp-thinking` | 262144 | 0.27 | 32 | +| `vllm-reward` | 1 | 8003 | `Skywork/Skywork-Reward-V2-Llama-3.x` | 16384 | 0.10 | — | +| `vllm-coder` | 1 | 8020 | `qwen2.5-coder-1.5b` | 8192 | 0.06 | 32 | +| `vllm-embed` | 1 | 8001 | `Qwen/Qwen3-Embedding-0.6B` | 8192 | 0.03 | — | +| `vllm-rerank-a3` | 1 | 8013 | `BAAI/bge-reranker-v2-m3` | 8192 | 0.03 | — | +| `vllm-flash-next` | 2 | 8022 | `qwen3.8-flash-next-uncensored`, `-thinking` | 262144 | 0.96 | 16 | + +## Lineage — read from each model's own `config.json` + +| seat | on-disk path | architecture | layers | experts | quantization | +|---|---|---|---|---|---| +| `mog-sec` | `/tank/aimodels/mog-sec-27b-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 MLPs (layers 0–55) + FP8 attn/lm_head/layers 56–63 | +| `gen` | `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 + FP8, same per-layer-group scheme | +| `flash-next` | `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` | `Qwen4ExpForConditionalGeneration` (`qwen4_exp`) | 48 | **512, 10 active** | `modelopt_fp4` (declared on the CLI; no `quantization_config` in config.json) | +| `char-rp` | `/tank/aimodels/gemma4-26b-a4b-it-nvfp4` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` | +| `erp-seat` | `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` | + +`mog-sec` carries `max_position_embeddings` **1,000,000** in config but is served at +163,840 — the CLI, not the checkpoint, is the operative limit. + +**`char-rp` and `erp-seat` are the same architecture, size and expert count** — both +Gemma-4 26B A4B MoE at NVFP4. `erp-seat` is the uncensored/"heretic" abliteration of +that family; `char-rp` is the stock instruct build. That is why they are co-located. + +## Speculative decoding — THREE different methods in play + +This is the least obvious thing about the box and the easiest to get wrong. + +| seat | method | k | drafter | +|---|---|---|---| +| `vllm-mog-sec` | **`dflash`** | **7** | separate model at `/drafter` | +| `vllm-gen` | **`qwen3_5_mtp`** | 3 | self (MTP head) | +| `vllm-flash-next` | **`mtp`** | 3 | self (MTP head) | +| all others | none | — | — | + +⚠ The method string is **architecture-specific and not interchangeable**: +`qwen3_5_mtp` for the qwen3_5 seats, plain `mtp` for `qwen4_exp`. vLLM warns that +`qwen4_exp_mtp` is deprecated and rewrites it to `mtp` at boot. Copying a +`--speculative-config` between seats without changing `method` will not work. + +`flash-next` gained MTP on **2026-09-13** — see `services/flash-next-mtp-bench/` +for the measurement (+52% at conc=1) and `stacks/flash-next-seat/compose.yaml` for +the rationale and the KV-cache cost that came with it. + +## Image sprawl + +Five distinct vLLM images across nine seats: + +| image | seats | +|---|---| +| `nightly-eed1f3d0…` | `flash-next` | +| `nightly-311b3513…` | `gen`, `erp-seat` | +| `nightly-e9d1398d…` | `mog-sec` | +| `v0.26.0` | `gemma4-charrp` | +| `v0.24.0` | `rerank-a3` | +| **`latest`** | `reward`, `coder`, `embed` | + +⚠ **`:latest` is a floating tag on three seats.** Whatever those pull on next +recreate is not what is running now, and there is no record of what they were built +against. Pin them at the next touch. + +## Drift found 2026-09-13 — gateway docs vs reality + +**1. `char-rp` is documented as a different model on a different host and GPU.** +The LiteLLM config comments describe `char-rp` as *"MeroMero-v2 NON-THINKING prose +seat (:8016, vLLM, meromero-charrp stack on ana-ml2 GPU 0), G4-MeroMero-v2-31B +NVFP4A16"*. What is actually serving :8016 is **`gemma4-26b-a4b-it-nvfp4` — a 26B, +30-layer, 128-expert stock Gemma-4** — in container `vllm-gemma4-charrp` on +**fv-ml1 GPU 1**. Part of that is stale host naming (ana-ml2 → fv-ml1, relocated +2026-09), but **31B MeroMero-v2 and 26B stock gemma4 are different models**, and the +comment block carries sampler defaults keyed to the model it names. Those defaults +have not been re-validated against what is deployed. + +**2. Container name does not match the alias it serves.** `vllm-erp-seat` (:8021) +backs the **`char-rp-fast`** alias, not an "erp" alias. The name predates the routing. + +**3. Two gateway aliases point at ports with nothing listening.** Both confirmed +dead (`curl` → no response, no container bound): + +| alias | target | status | +|---|---|---| +| `erp-tune-v2` | `10.251.50.54:8098` | **dead** | +| `gemma4-26b-a4b-it-base` | `10.251.50.54:8099` | **dead** | + +These are almost certainly leftovers from the quant/eval containers still present +as exited (`erp-v7-quant`, `erp-v6-quant`, `a4b-moe-quant`, `erp-eval-v2`, +`vllm-gemma4-trainee-bench`). A caller hitting either alias gets a connection +failure, not a clear "model retired" error. Per the standing +never-alias-a-model-under-a-stale-name rule, they should be removed from the +gateway or repointed deliberately — not left to fail. + +## Gateway aliases resolving to this host (19) + +`char-rp` · `char-rp-fast` · `char-rp-reasoning` · `chat-judge` · `classifier` · +`coder-fast` · `erp-tune-v2` ⚠dead · `gemma4-26b-a4b-it-base` ⚠dead · `gen` · +`gen-large` · `gen-reasoning` · `image-judge` · `qwen-image-bench` · +`qwen3-embedding` · `reranker` · `sec` · `sec-reasoning` · `summarizer` · +`summarizer-large` + +**Seven aliases share one seat.** `summarizer`, `summarizer-large`, `classifier`, +`chat-judge`, `image-judge`, `qwen-image-bench` and `gen` all resolve to +`vllm-gen` on :8015 — so anything that loads the summarizer loads GPU 0. This is +why GPU 0 shows background draw during ordinary OpenWebUI use: its auto-titling +fires the `summarizer` alias on every exchange. + +## How to regenerate this + +Read the containers, not the compose files: + +```sh +# placement + footprint +nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv +# map pid -> container via /proc//cgroup, then docker inspect +docker inspect --format '{{json .Args}}' # served names, quant, spec-config +docker inspect --format '{{json .Mounts}}' # resolve /model to a /tank path +# lineage +python3 -c "import json;c=json.load(open('/config.json'));print(c['architectures'])" +``` + +⚠ `docker inspect` reports the **container's** PID; `nvidia-smi` reports the vLLM +engine child. Map through `/proc//cgroup`, not by matching PIDs directly. diff --git a/stacks/flash-next-seat/.env.example b/stacks/flash-next-seat/.env.example index 3892dfb..1a893f0 100644 --- a/stacks/flash-next-seat/.env.example +++ b/stacks/flash-next-seat/.env.example @@ -54,7 +54,14 @@ FN_GPU_MEM_UTIL=0.96 # Re-raising requires re-running the deep probe and reading the allocator log. # VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB # still free on the card at peak. -FN_KV_CACHE_MEMORY=15032385536 +# ⚠ MTP ON (k=3) REQUIRES A SMALLER KV POOL THAN THE NO-SPEC SEAT. +# Measured 2026-09-13: the draft head adds ~5.08 GiB of weights and raises +# per-token KV cost ~16%. At the old 15032385536 (14 GiB) the engine OOMs at +# init with MTP enabled -- 79.44 GiB weights + 14 GiB KV = 93.44 against a +# 91.17 GiB budget. 8589934592 (8 GiB) is PROVEN to boot; 10737418240 (10 GiB) +# leaves ~1.73 GiB for activations and is the current setting. If the seat +# OOMs at init, drop to 8589934592. +FN_KV_CACHE_MEMORY=10737418240 # FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K # tokens, so a single max-length request fits with ~2.4x concurrency to spare. diff --git a/stacks/flash-next-seat/compose.yaml b/stacks/flash-next-seat/compose.yaml index ed6117e..6c3d290 100644 --- a/stacks/flash-next-seat/compose.yaml +++ b/stacks/flash-next-seat/compose.yaml @@ -221,14 +221,40 @@ services: # the mog-sec seat uses. - --tool-call-parser - ${FN_TOOL_CALL_PARSER:-qwen3_xml} - # ── NO --speculative-config LINE, ON PURPOSE ───────────────────────────── - # Speculative decoding is absent rather than disabled-by-a-flag, because - # there is no honest "off" value for --speculative-config and a placeholder - # flag in this slot is the kind of thing that gets copied forward and - # misread. To turn MTP on, add these two lines here and nothing else: - # - --speculative-config - # - '{"method": "mtp", "num_speculative_tokens": 3}' - # Read the MTP warning in the header first; the measured default is off. + # ── MTP ON, k=3 — MEASURED 2026-09-13, reversing the previous default ──── + # This seat shipped with speculative decoding absent, citing vLLM's recipe: + # on 4xH100 at TP=4 that recipe measured MTP WORSE at every concurrency + # (8-36% lower throughput, ~36% acceptance) and said don't default it on. + # + # That recipe does not describe this machine, and the campaign in + # services/flash-next-mtp-bench/ measured close to its inverse on 1x RTX + # PRO 6000 Blackwell at TP=1 (5 arms x 3 reps x conc 1/4/8, drift-bracketed + # by a no-spec arm at each end, both within noise): + # + # vs no-spec floor conc=1 conc=4 conc=8 acceptance + # k=1 +29.6% +40.7% +27.2% ~73% + # k=2 +42.3% +52.4% +37.7% ~59% + # k=3 +52.1% +51.3% +33.5% ~48% + # + # k=3 is set because this seat is single-user interactive, where conc=1 + # dominates and k=3 measured best (+52.1%, and the k3-over-k2 margin at + # conc=1 cleared its own noise floor). For a batch-heavy seat prefer k=2: + # it is best or statistically tied everywhere, and k=3's conc=4/8 figures + # sit BELOW k=2 (within noise, so not a proven regression, but no gain). + # + # ⚠ TWO COSTS, both measured, neither optional: + # 1. MTP adds ~5.08 GiB of draft-head weights (74.36 -> 79.44 GiB) and + # raises per-token KV cost ~16% (26.24 -> 30.47 KiB/token). The seat's + # previous 14 GiB FN_KV_CACHE_MEMORY OOMs at engine init with MTP on. + # FN_KV_CACHE_MEMORY must come down; see .env.example. + # 2. Run-to-run variance grows sharply with k (within-arm spread at conc=4: + # 5.69 no-spec -> 34.61 k1 -> 57.06 k2 -> 76.90 k3). Throughput is + # higher and LESS predictable. That is a real operational property. + # + # To turn MTP back off, delete these two lines — do not add a placeholder + # value, there is no honest "off" for --speculative-config. + - --speculative-config + - '{"method": "mtp", "num_speculative_tokens": 3}' deploy: resources: reservations: