docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers

Read from docker inspect, nvidia-smi and each model's own config.json rather
than from compose files or the gateway config, so the document records what is
deployed rather than what was intended.

Findings worth naming:

- char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on
  ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer
  128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it,
  but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment
  block carries sampler defaults keyed to the model it names.
- Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base
  (:8099). Nothing is listening on either; callers get a connection failure
  rather than a clear retirement error.
- Three different speculative-decoding methods coexist -- dflash k=7 with a
  separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next.
  The method string is architecture-specific; copying a --speculative-config
  between seats without changing it will not work.
- Three seats run the floating :latest tag, so what they would pull on recreate
  is not what is running and was never recorded.
- Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw
  during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn.

Also commits the flash-next-seat MTP k=3 change made today: compose gains the
speculative-config with its measured rationale, and .env.example documents the
KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
This commit is contained in:
vh
2026-09-13 22:28:58 -07:00
parent f9a8f176ca
commit 2d83a895c1
3 changed files with 194 additions and 9 deletions
+8 -1
View File
@@ -54,7 +54,14 @@ FN_GPU_MEM_UTIL=0.96
# Re-raising requires re-running the deep probe and reading the allocator log.
# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB
# still free on the card at peak.
FN_KV_CACHE_MEMORY=15032385536
# ⚠ MTP ON (k=3) REQUIRES A SMALLER KV POOL THAN THE NO-SPEC SEAT.
# Measured 2026-09-13: the draft head adds ~5.08 GiB of weights and raises
# per-token KV cost ~16%. At the old 15032385536 (14 GiB) the engine OOMs at
# init with MTP enabled -- 79.44 GiB weights + 14 GiB KV = 93.44 against a
# 91.17 GiB budget. 8589934592 (8 GiB) is PROVEN to boot; 10737418240 (10 GiB)
# leaves ~1.73 GiB for activations and is the current setting. If the seat
# OOMs at init, drop to 8589934592.
FN_KV_CACHE_MEMORY=10737418240
# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K
# tokens, so a single max-length request fits with ~2.4x concurrency to spare.
+34 -8
View File
@@ -221,14 +221,40 @@ services:
# the mog-sec seat uses.
- --tool-call-parser
- ${FN_TOOL_CALL_PARSER:-qwen3_xml}
# ── NO --speculative-config LINE, ON PURPOSE ─────────────────────────────
# Speculative decoding is absent rather than disabled-by-a-flag, because
# there is no honest "off" value for --speculative-config and a placeholder
# flag in this slot is the kind of thing that gets copied forward and
# misread. To turn MTP on, add these two lines here and nothing else:
# - --speculative-config
# - '{"method": "mtp", "num_speculative_tokens": 3}'
# Read the MTP warning in the header first; the measured default is off.
# ── MTP ON, k=3 — MEASURED 2026-09-13, reversing the previous default ────
# This seat shipped with speculative decoding absent, citing vLLM's recipe:
# on 4xH100 at TP=4 that recipe measured MTP WORSE at every concurrency
# (8-36% lower throughput, ~36% acceptance) and said don't default it on.
#
# That recipe does not describe this machine, and the campaign in
# services/flash-next-mtp-bench/ measured close to its inverse on 1x RTX
# PRO 6000 Blackwell at TP=1 (5 arms x 3 reps x conc 1/4/8, drift-bracketed
# by a no-spec arm at each end, both within noise):
#
# vs no-spec floor conc=1 conc=4 conc=8 acceptance
# k=1 +29.6% +40.7% +27.2% ~73%
# k=2 +42.3% +52.4% +37.7% ~59%
# k=3 +52.1% +51.3% +33.5% ~48%
#
# k=3 is set because this seat is single-user interactive, where conc=1
# dominates and k=3 measured best (+52.1%, and the k3-over-k2 margin at
# conc=1 cleared its own noise floor). For a batch-heavy seat prefer k=2:
# it is best or statistically tied everywhere, and k=3's conc=4/8 figures
# sit BELOW k=2 (within noise, so not a proven regression, but no gain).
#
# ⚠ TWO COSTS, both measured, neither optional:
# 1. MTP adds ~5.08 GiB of draft-head weights (74.36 -> 79.44 GiB) and
# raises per-token KV cost ~16% (26.24 -> 30.47 KiB/token). The seat's
# previous 14 GiB FN_KV_CACHE_MEMORY OOMs at engine init with MTP on.
# FN_KV_CACHE_MEMORY must come down; see .env.example.
# 2. Run-to-run variance grows sharply with k (within-arm spread at conc=4:
# 5.69 no-spec -> 34.61 k1 -> 57.06 k2 -> 76.90 k3). Throughput is
# higher and LESS predictable. That is a real operational property.
#
# To turn MTP back off, delete these two lines — do not add a placeholder
# value, there is no honest "off" for --speculative-config.
- --speculative-config
- '{"method": "mtp", "num_speculative_tokens": 3}'
deploy:
resources:
reservations: