docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers
Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
This commit is contained in:
@@ -0,0 +1,152 @@
|
||||
# fv-ml1 — GPU seat inventory and model lineage
|
||||
|
||||
**Reconciled 2026-09-13 against the RUNNING containers**, not against compose files
|
||||
or the LiteLLM config. Every field below was read from `docker inspect`,
|
||||
`nvidia-smi`, or the model's own `config.json` on `/tank`. Where the gateway's
|
||||
documentation disagrees with what is deployed, the discrepancy is recorded in
|
||||
§ Drift rather than silently corrected.
|
||||
|
||||
Host: `fv-ml1` 10.251.50.54 · 4× RTX PRO 6000 Blackwell Max-Q, 96 GB each
|
||||
Power cap: **275 W/card = 1100 W aggregate** (`gpu-power-cap.service`, before docker)
|
||||
|
||||
## Placement
|
||||
|
||||
| GPU | VRAM used | seats |
|
||||
|---|---|---|
|
||||
| 0 | 86.2 GB / 95.6 | `vllm-mog-sec` 47.8 · `vllm-gen` 38.4 |
|
||||
| 1 | 73.9 GB / 95.6 | `vllm-erp-seat` 27.0 · `vllm-gemma4-charrp` 26.0 · `vllm-reward` 9.2 · `vllm-coder` 6.1 · `vllm-embed` 3.5 · `vllm-rerank-a3` 2.1 · `scriberr` (no persistent VRAM) |
|
||||
| 2 | 96.4 GB / 95.6 | `vllm-flash-next` |
|
||||
| 3 | **empty — reserved** | held free for ad-hoc work (operator, 2026-09-13) |
|
||||
|
||||
⚠ **GPU 3 is deliberately empty.** Do not place a seat there without asking; it is
|
||||
the scratch card for benches, quants and probes. The MTP campaign that produced
|
||||
`services/flash-next-mtp-bench/` ran there.
|
||||
|
||||
## Seats
|
||||
|
||||
| container | GPU | port | served-model-name | ctx | util | seqs |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `vllm-mog-sec` | 0 | 8019 | `mog-sec-27b`, `mog-sec-27b-thinking` | 163840 | 0.50 | 16 |
|
||||
| `vllm-gen` | 0 | 8015 | `qwen3.8-27b-uncensored`, `-thinking` | 262144 | 0.38 | 16 |
|
||||
| `vllm-erp-seat` | 1 | 8021 | `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | 262144 | 0.30 | 32 |
|
||||
| `vllm-gemma4-charrp` | 1 | 8016 | `char-rp`, `char-rp-thinking` | 262144 | 0.27 | 32 |
|
||||
| `vllm-reward` | 1 | 8003 | `Skywork/Skywork-Reward-V2-Llama-3.x` | 16384 | 0.10 | — |
|
||||
| `vllm-coder` | 1 | 8020 | `qwen2.5-coder-1.5b` | 8192 | 0.06 | 32 |
|
||||
| `vllm-embed` | 1 | 8001 | `Qwen/Qwen3-Embedding-0.6B` | 8192 | 0.03 | — |
|
||||
| `vllm-rerank-a3` | 1 | 8013 | `BAAI/bge-reranker-v2-m3` | 8192 | 0.03 | — |
|
||||
| `vllm-flash-next` | 2 | 8022 | `qwen3.8-flash-next-uncensored`, `-thinking` | 262144 | 0.96 | 16 |
|
||||
|
||||
## Lineage — read from each model's own `config.json`
|
||||
|
||||
| seat | on-disk path | architecture | layers | experts | quantization |
|
||||
|---|---|---|---|---|---|
|
||||
| `mog-sec` | `/tank/aimodels/mog-sec-27b-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 MLPs (layers 0–55) + FP8 attn/lm_head/layers 56–63 |
|
||||
| `gen` | `/tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed` | `Qwen3_5ForConditionalGeneration` (`qwen3_5`) | 64 | — | **mixed-precision**: NVFP4 + FP8, same per-layer-group scheme |
|
||||
| `flash-next` | `/tank/aimodels/qwen38-flash-next-abliterated-nvfp4` | `Qwen4ExpForConditionalGeneration` (`qwen4_exp`) | 48 | **512, 10 active** | `modelopt_fp4` (declared on the CLI; no `quantization_config` in config.json) |
|
||||
| `char-rp` | `/tank/aimodels/gemma4-26b-a4b-it-nvfp4` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
| `erp-seat` | `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `Gemma4ForConditionalGeneration` (`gemma4`) | 30 | 128 | `nvfp4-pack-quantized` |
|
||||
|
||||
`mog-sec` carries `max_position_embeddings` **1,000,000** in config but is served at
|
||||
163,840 — the CLI, not the checkpoint, is the operative limit.
|
||||
|
||||
**`char-rp` and `erp-seat` are the same architecture, size and expert count** — both
|
||||
Gemma-4 26B A4B MoE at NVFP4. `erp-seat` is the uncensored/"heretic" abliteration of
|
||||
that family; `char-rp` is the stock instruct build. That is why they are co-located.
|
||||
|
||||
## Speculative decoding — THREE different methods in play
|
||||
|
||||
This is the least obvious thing about the box and the easiest to get wrong.
|
||||
|
||||
| seat | method | k | drafter |
|
||||
|---|---|---|---|
|
||||
| `vllm-mog-sec` | **`dflash`** | **7** | separate model at `/drafter` |
|
||||
| `vllm-gen` | **`qwen3_5_mtp`** | 3 | self (MTP head) |
|
||||
| `vllm-flash-next` | **`mtp`** | 3 | self (MTP head) |
|
||||
| all others | none | — | — |
|
||||
|
||||
⚠ The method string is **architecture-specific and not interchangeable**:
|
||||
`qwen3_5_mtp` for the qwen3_5 seats, plain `mtp` for `qwen4_exp`. vLLM warns that
|
||||
`qwen4_exp_mtp` is deprecated and rewrites it to `mtp` at boot. Copying a
|
||||
`--speculative-config` between seats without changing `method` will not work.
|
||||
|
||||
`flash-next` gained MTP on **2026-09-13** — see `services/flash-next-mtp-bench/`
|
||||
for the measurement (+52% at conc=1) and `stacks/flash-next-seat/compose.yaml` for
|
||||
the rationale and the KV-cache cost that came with it.
|
||||
|
||||
## Image sprawl
|
||||
|
||||
Five distinct vLLM images across nine seats:
|
||||
|
||||
| image | seats |
|
||||
|---|---|
|
||||
| `nightly-eed1f3d0…` | `flash-next` |
|
||||
| `nightly-311b3513…` | `gen`, `erp-seat` |
|
||||
| `nightly-e9d1398d…` | `mog-sec` |
|
||||
| `v0.26.0` | `gemma4-charrp` |
|
||||
| `v0.24.0` | `rerank-a3` |
|
||||
| **`latest`** | `reward`, `coder`, `embed` |
|
||||
|
||||
⚠ **`:latest` is a floating tag on three seats.** Whatever those pull on next
|
||||
recreate is not what is running now, and there is no record of what they were built
|
||||
against. Pin them at the next touch.
|
||||
|
||||
## Drift found 2026-09-13 — gateway docs vs reality
|
||||
|
||||
**1. `char-rp` is documented as a different model on a different host and GPU.**
|
||||
The LiteLLM config comments describe `char-rp` as *"MeroMero-v2 NON-THINKING prose
|
||||
seat (:8016, vLLM, meromero-charrp stack on ana-ml2 GPU 0), G4-MeroMero-v2-31B
|
||||
NVFP4A16"*. What is actually serving :8016 is **`gemma4-26b-a4b-it-nvfp4` — a 26B,
|
||||
30-layer, 128-expert stock Gemma-4** — in container `vllm-gemma4-charrp` on
|
||||
**fv-ml1 GPU 1**. Part of that is stale host naming (ana-ml2 → fv-ml1, relocated
|
||||
2026-09), but **31B MeroMero-v2 and 26B stock gemma4 are different models**, and the
|
||||
comment block carries sampler defaults keyed to the model it names. Those defaults
|
||||
have not been re-validated against what is deployed.
|
||||
|
||||
**2. Container name does not match the alias it serves.** `vllm-erp-seat` (:8021)
|
||||
backs the **`char-rp-fast`** alias, not an "erp" alias. The name predates the routing.
|
||||
|
||||
**3. Two gateway aliases point at ports with nothing listening.** Both confirmed
|
||||
dead (`curl` → no response, no container bound):
|
||||
|
||||
| alias | target | status |
|
||||
|---|---|---|
|
||||
| `erp-tune-v2` | `10.251.50.54:8098` | **dead** |
|
||||
| `gemma4-26b-a4b-it-base` | `10.251.50.54:8099` | **dead** |
|
||||
|
||||
These are almost certainly leftovers from the quant/eval containers still present
|
||||
as exited (`erp-v7-quant`, `erp-v6-quant`, `a4b-moe-quant`, `erp-eval-v2`,
|
||||
`vllm-gemma4-trainee-bench`). A caller hitting either alias gets a connection
|
||||
failure, not a clear "model retired" error. Per the standing
|
||||
never-alias-a-model-under-a-stale-name rule, they should be removed from the
|
||||
gateway or repointed deliberately — not left to fail.
|
||||
|
||||
## Gateway aliases resolving to this host (19)
|
||||
|
||||
`char-rp` · `char-rp-fast` · `char-rp-reasoning` · `chat-judge` · `classifier` ·
|
||||
`coder-fast` · `erp-tune-v2` ⚠dead · `gemma4-26b-a4b-it-base` ⚠dead · `gen` ·
|
||||
`gen-large` · `gen-reasoning` · `image-judge` · `qwen-image-bench` ·
|
||||
`qwen3-embedding` · `reranker` · `sec` · `sec-reasoning` · `summarizer` ·
|
||||
`summarizer-large`
|
||||
|
||||
**Seven aliases share one seat.** `summarizer`, `summarizer-large`, `classifier`,
|
||||
`chat-judge`, `image-judge`, `qwen-image-bench` and `gen` all resolve to
|
||||
`vllm-gen` on :8015 — so anything that loads the summarizer loads GPU 0. This is
|
||||
why GPU 0 shows background draw during ordinary OpenWebUI use: its auto-titling
|
||||
fires the `summarizer` alias on every exchange.
|
||||
|
||||
## How to regenerate this
|
||||
|
||||
Read the containers, not the compose files:
|
||||
|
||||
```sh
|
||||
# placement + footprint
|
||||
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv
|
||||
# map pid -> container via /proc/<pid>/cgroup, then docker inspect
|
||||
docker inspect <name> --format '{{json .Args}}' # served names, quant, spec-config
|
||||
docker inspect <name> --format '{{json .Mounts}}' # resolve /model to a /tank path
|
||||
# lineage
|
||||
python3 -c "import json;c=json.load(open('<path>/config.json'));print(c['architectures'])"
|
||||
```
|
||||
|
||||
⚠ `docker inspect` reports the **container's** PID; `nvidia-smi` reports the vLLM
|
||||
engine child. Map through `/proc/<pid>/cgroup`, not by matching PIDs directly.
|
||||
Reference in New Issue
Block a user