docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers
Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
This commit is contained in:
@@ -0,0 +1,152 @@
|
||||
# fv-ml1 — GPU seat inventory and model lineage
|
||||
|
||||
**Reconciled 2026-09-13 against the RUNNING containers**, not against compose files
|
||||
or the LiteLLM config. Every field below was read from `docker inspect`,
|
||||
`nvidia-smi`, or the model's own `config.json` on `/tank`. Where the gateway's
|
||||
documentation disagrees with what is deployed, the discrepancy is recorded in
|
||||
§ Drift rather than silently corrected.
|
||||
|
||||
Host: `fv-ml1` 10.251.50.54 · 4× RTX PRO 6000 Blackwell Max-Q, 96 GB each
|
||||
Power cap: **275 W/card = 1100 W aggregate** (`gpu-power-cap.service`, before docker)
|
||||
|
||||
## Placement
|
||||
|
||||
| GPU | VRAM used | seats |
|
||||
|---|---|---|
|
||||
| 0 | 86.2 GB / 95.6 | `vllm-mog-sec` 47.8 · `vllm-gen` 38.4 |
|
||||
| 1 | 73.9 GB / 95.6 | `vllm-erp-seat` 27.0 · `vllm-gemma4-charrp` 26.0 · `vllm-reward` 9.2 · `vllm-coder` 6.1 · `vllm-embed` 3.5 · `vllm-rerank-a3` 2.1 · `scriberr` (no persistent VRAM) |
|
||||
| 2 | 96.4 GB / 95.6 | `vllm-flash-next` |
|
||||
| 3 | **empty — reserved** | held free for ad-hoc work (operator, 2026-09-13) |
|
||||
|
||||
⚠ **GPU 3 is deliberately empty.** Do not place a seat there without asking; it is
|
||||
the scratch card for benches, quants and probes. The MTP campaign that produced
|
||||
`services/flash-next-mtp-bench/` ran there.
|
||||
|
||||
## Seats
|
||||
|
||||
| container | GPU | port | served-model-name | ctx | util | seqs |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `vllm-mog-sec` | 0 | 8019 | `mog-sec-27b`, `mog-sec-27b-thinking` | 163840 | 0.50 | 16 |
|
||||
| `vllm-gen` | 0 | 8015 | `qwen3.8-27b-uncensored`, `-thinking` | 262144 | 0.38 | 16 |
|
||||
| `vllm-erp-seat` | 1 | 8021 | `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | 262144 | 0.30 | 32 |
|
||||
| `vllm-gemma4-charrp` | 1 | 8016 | `char-rp`, `char-rp-thinking` | 262144 | 0.27 | 32 |
|
||||
| `vllm-reward` | 1 | 8003 | `Skywork/Skywork-Reward-V2-Llama-3.x` | 16384 | 0.10 | — |
|
||||
| `vllm-coder` | 1 | 8020 | `qwen2.5-coder-1.5b` | 8192 | 0.06 | 32 |
|
||||
| `vllm-embed` | 1 | 8001 | `Qwen/Qwen3-Embedding-0.6B` | 8192 | 0.03 | — |
|
||||
| `vllm-rerank-a3` | 1 | 8013 | `BAAI/bge-reranker-v2-m3` | 8192 | 0.03 | — |
|
||||
| `vllm-flash-next` | 2 | 8022 | `qwen3.8-flash-next-uncensored`, `-thinking` | 262144 | 0.96 | 16 |
|
||||
|
||||
## Lineage — read from each model's own `config.json`
|
||||
|
||||
| seat | on-disk path | architecture | layers | experts | quantization |
|
||||
|---|---|---|---|---|---|
|
||||
| `mog-sec` | `/tank/aimodels/mog-sec-27b-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 MLPs (layers 0–55) + FP8 attn/lm_head/layers 56–63 |
|
||||
| `gen` | `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 + FP8, same per-layer-group scheme |
|
||||
| `flash-next` | `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` | `Qwen4ExpForConditionalGeneration` (`qwen4_exp`) | 48 | **512, 10 active** | `modelopt_fp4` (declared on the CLI; no `quantization_config` in config.json) |
|
||||
| `char-rp` | `/tank/aimodels/gemma4-26b-a4b-it-nvfp4` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
| `erp-seat` | `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
|
||||
`mog-sec` carries `max_position_embeddings` **1,000,000** in config but is served at
|
||||
163,840 — the CLI, not the checkpoint, is the operative limit.
|
||||
|
||||
**`char-rp` and `erp-seat` are the same architecture, size and expert count** — both
|
||||
Gemma-4 26B A4B MoE at NVFP4. `erp-seat` is the uncensored/"heretic" abliteration of
|
||||
that family; `char-rp` is the stock instruct build. That is why they are co-located.
|
||||
|
||||
## Speculative decoding — THREE different methods in play
|
||||
|
||||
This is the least obvious thing about the box and the easiest to get wrong.
|
||||
|
||||
| seat | method | k | drafter |
|
||||
|---|---|---|---|
|
||||
| `vllm-mog-sec` | **`dflash`** | **7** | separate model at `/drafter` |
|
||||
| `vllm-gen` | **`qwen3_5_mtp`** | 3 | self (MTP head) |
|
||||
| `vllm-flash-next` | **`mtp`** | 3 | self (MTP head) |
|
||||
| all others | none | — | — |
|
||||
|
||||
⚠ The method string is **architecture-specific and not interchangeable**:
|
||||
`qwen3_5_mtp` for the qwen3_5 seats, plain `mtp` for `qwen4_exp`. vLLM warns that
|
||||
`qwen4_exp_mtp` is deprecated and rewrites it to `mtp` at boot. Copying a
|
||||
`--speculative-config` between seats without changing `method` will not work.
|
||||
|
||||
`flash-next` gained MTP on **2026-09-13** — see `services/flash-next-mtp-bench/`
|
||||
for the measurement (+52% at conc=1) and `stacks/flash-next-seat/compose.yaml` for
|
||||
the rationale and the KV-cache cost that came with it.
|
||||
|
||||
## Image sprawl
|
||||
|
||||
Five distinct vLLM images across nine seats:
|
||||
|
||||
| image | seats |
|
||||
|---|---|
|
||||
| `nightly-eed1f3d0…` | `flash-next` |
|
||||
| `nightly-311b3513…` | `gen`, `erp-seat` |
|
||||
| `nightly-e9d1398d…` | `mog-sec` |
|
||||
| `v0.26.0` | `gemma4-charrp` |
|
||||
| `v0.24.0` | `rerank-a3` |
|
||||
| **`latest`** | `reward`, `coder`, `embed` |
|
||||
|
||||
⚠ **`:latest` is a floating tag on three seats.** Whatever those pull on next
|
||||
recreate is not what is running now, and there is no record of what they were built
|
||||
against. Pin them at the next touch.
|
||||
|
||||
## Drift found 2026-09-13 — gateway docs vs reality
|
||||
|
||||
**1. `char-rp` is documented as a different model on a different host and GPU.**
|
||||
The LiteLLM config comments describe `char-rp` as *"MeroMero-v2 NON-THINKING prose
|
||||
seat (:8016, vLLM, meromero-charrp stack on ana-ml2 GPU 0), G4-MeroMero-v2-31B
|
||||
NVFP4A16"*. What is actually serving :8016 is **`gemma4-26b-a4b-it-nvfp4` — a 26B,
|
||||
30-layer, 128-expert stock Gemma-4** — in container `vllm-gemma4-charrp` on
|
||||
**fv-ml1 GPU 1**. Part of that is stale host naming (ana-ml2 → fv-ml1, relocated
|
||||
2026-09), but **31B MeroMero-v2 and 26B stock gemma4 are different models**, and the
|
||||
comment block carries sampler defaults keyed to the model it names. Those defaults
|
||||
have not been re-validated against what is deployed.
|
||||
|
||||
**2. Container name does not match the alias it serves.** `vllm-erp-seat` (:8021)
|
||||
backs the **`char-rp-fast`** alias, not an "erp" alias. The name predates the routing.
|
||||
|
||||
**3. Two gateway aliases point at ports with nothing listening.** Both confirmed
|
||||
dead (`curl` → no response, no container bound):
|
||||
|
||||
| alias | target | status |
|
||||
|---|---|---|
|
||||
| `erp-tune-v2` | `10.251.50.54:8098` | **dead** |
|
||||
| `gemma4-26b-a4b-it-base` | `10.251.50.54:8099` | **dead** |
|
||||
|
||||
These are almost certainly leftovers from the quant/eval containers still present
|
||||
as exited (`erp-v7-quant`, `erp-v6-quant`, `a4b-moe-quant`, `erp-eval-v2`,
|
||||
`vllm-gemma4-trainee-bench`). A caller hitting either alias gets a connection
|
||||
failure, not a clear "model retired" error. Per the standing
|
||||
never-alias-a-model-under-a-stale-name rule, they should be removed from the
|
||||
gateway or repointed deliberately — not left to fail.
|
||||
|
||||
## Gateway aliases resolving to this host (19)
|
||||
|
||||
`char-rp` · `char-rp-fast` · `char-rp-reasoning` · `chat-judge` · `classifier` ·
|
||||
`coder-fast` · `erp-tune-v2` ⚠dead · `gemma4-26b-a4b-it-base` ⚠dead · `gen` ·
|
||||
`gen-large` · `gen-reasoning` · `image-judge` · `qwen-image-bench` ·
|
||||
`qwen3-embedding` · `reranker` · `sec` · `sec-reasoning` · `summarizer` ·
|
||||
`summarizer-large`
|
||||
|
||||
**Seven aliases share one seat.** `summarizer`, `summarizer-large`, `classifier`,
|
||||
`chat-judge`, `image-judge`, `qwen-image-bench` and `gen` all resolve to
|
||||
`vllm-gen` on :8015 — so anything that loads the summarizer loads GPU 0. This is
|
||||
why GPU 0 shows background draw during ordinary OpenWebUI use: its auto-titling
|
||||
fires the `summarizer` alias on every exchange.
|
||||
|
||||
## How to regenerate this
|
||||
|
||||
Read the containers, not the compose files:
|
||||
|
||||
```sh
|
||||
# placement + footprint
|
||||
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv
|
||||
# map pid -> container via /proc/<pid>/cgroup, then docker inspect
|
||||
docker inspect <name> --format '{{json .Args}}' # served names, quant, spec-config
|
||||
docker inspect <name> --format '{{json .Mounts}}' # resolve /model to a /tank path
|
||||
# lineage
|
||||
python3 -c "import json;c=json.load(open('<path>/config.json'));print(c['architectures'])"
|
||||
```
|
||||
|
||||
⚠ `docker inspect` reports the **container's** PID; `nvidia-smi` reports the vLLM
|
||||
engine child. Map through `/proc/<pid>/cgroup`, not by matching PIDs directly.
|
||||
@@ -54,7 +54,14 @@ FN_GPU_MEM_UTIL=0.96
|
||||
# Re-raising requires re-running the deep probe and reading the allocator log.
|
||||
# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB
|
||||
# still free on the card at peak.
|
||||
FN_KV_CACHE_MEMORY=15032385536
|
||||
# ⚠ MTP ON (k=3) REQUIRES A SMALLER KV POOL THAN THE NO-SPEC SEAT.
|
||||
# Measured 2026-09-13: the draft head adds ~5.08 GiB of weights and raises
|
||||
# per-token KV cost ~16%. At the old 15032385536 (14 GiB) the engine OOMs at
|
||||
# init with MTP enabled -- 79.44 GiB weights + 14 GiB KV = 93.44 against a
|
||||
# 91.17 GiB budget. 8589934592 (8 GiB) is PROVEN to boot; 10737418240 (10 GiB)
|
||||
# leaves ~1.73 GiB for activations and is the current setting. If the seat
|
||||
# OOMs at init, drop to 8589934592.
|
||||
FN_KV_CACHE_MEMORY=10737418240
|
||||
|
||||
# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K
|
||||
# tokens, so a single max-length request fits with ~2.4x concurrency to spare.
|
||||
|
||||
@@ -221,14 +221,40 @@ services:
|
||||
# the mog-sec seat uses.
|
||||
- --tool-call-parser
|
||||
- ${FN_TOOL_CALL_PARSER:-qwen3_xml}
|
||||
# ── NO --speculative-config LINE, ON PURPOSE ─────────────────────────────
|
||||
# Speculative decoding is absent rather than disabled-by-a-flag, because
|
||||
# there is no honest "off" value for --speculative-config and a placeholder
|
||||
# flag in this slot is the kind of thing that gets copied forward and
|
||||
# misread. To turn MTP on, add these two lines here and nothing else:
|
||||
# - --speculative-config
|
||||
# - '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
# Read the MTP warning in the header first; the measured default is off.
|
||||
# ── MTP ON, k=3 — MEASURED 2026-09-13, reversing the previous default ────
|
||||
# This seat shipped with speculative decoding absent, citing vLLM's recipe:
|
||||
# on 4xH100 at TP=4 that recipe measured MTP WORSE at every concurrency
|
||||
# (8-36% lower throughput, ~36% acceptance) and said don't default it on.
|
||||
#
|
||||
# That recipe does not describe this machine, and the campaign in
|
||||
# services/flash-next-mtp-bench/ measured close to its inverse on 1x RTX
|
||||
# PRO 6000 Blackwell at TP=1 (5 arms x 3 reps x conc 1/4/8, drift-bracketed
|
||||
# by a no-spec arm at each end, both within noise):
|
||||
#
|
||||
# vs no-spec floor conc=1 conc=4 conc=8 acceptance
|
||||
# k=1 +29.6% +40.7% +27.2% ~73%
|
||||
# k=2 +42.3% +52.4% +37.7% ~59%
|
||||
# k=3 +52.1% +51.3% +33.5% ~48%
|
||||
#
|
||||
# k=3 is set because this seat is single-user interactive, where conc=1
|
||||
# dominates and k=3 measured best (+52.1%, and the k3-over-k2 margin at
|
||||
# conc=1 cleared its own noise floor). For a batch-heavy seat prefer k=2:
|
||||
# it is best or statistically tied everywhere, and k=3's conc=4/8 figures
|
||||
# sit BELOW k=2 (within noise, so not a proven regression, but no gain).
|
||||
#
|
||||
# ⚠ TWO COSTS, both measured, neither optional:
|
||||
# 1. MTP adds ~5.08 GiB of draft-head weights (74.36 -> 79.44 GiB) and
|
||||
# raises per-token KV cost ~16% (26.24 -> 30.47 KiB/token). The seat's
|
||||
# previous 14 GiB FN_KV_CACHE_MEMORY OOMs at engine init with MTP on.
|
||||
# FN_KV_CACHE_MEMORY must come down; see .env.example.
|
||||
# 2. Run-to-run variance grows sharply with k (within-arm spread at conc=4:
|
||||
# 5.69 no-spec -> 34.61 k1 -> 57.06 k2 -> 76.90 k3). Throughput is
|
||||
# higher and LESS predictable. That is a real operational property.
|
||||
#
|
||||
# To turn MTP back off, delete these two lines — do not add a placeholder
|
||||
# value, there is no honest "off" for --speculative-config.
|
||||
- --speculative-config
|
||||
- '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
Reference in New Issue
Block a user