docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers
Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
This commit is contained in:
@@ -221,14 +221,40 @@ services:
|
||||
# the mog-sec seat uses.
|
||||
- --tool-call-parser
|
||||
- ${FN_TOOL_CALL_PARSER:-qwen3_xml}
|
||||
# ── NO --speculative-config LINE, ON PURPOSE ─────────────────────────────
|
||||
# Speculative decoding is absent rather than disabled-by-a-flag, because
|
||||
# there is no honest "off" value for --speculative-config and a placeholder
|
||||
# flag in this slot is the kind of thing that gets copied forward and
|
||||
# misread. To turn MTP on, add these two lines here and nothing else:
|
||||
# - --speculative-config
|
||||
# - '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
# Read the MTP warning in the header first; the measured default is off.
|
||||
# ── MTP ON, k=3 — MEASURED 2026-09-13, reversing the previous default ────
|
||||
# This seat shipped with speculative decoding absent, citing vLLM's recipe:
|
||||
# on 4xH100 at TP=4 that recipe measured MTP WORSE at every concurrency
|
||||
# (8-36% lower throughput, ~36% acceptance) and said don't default it on.
|
||||
#
|
||||
# That recipe does not describe this machine, and the campaign in
|
||||
# services/flash-next-mtp-bench/ measured close to its inverse on 1x RTX
|
||||
# PRO 6000 Blackwell at TP=1 (5 arms x 3 reps x conc 1/4/8, drift-bracketed
|
||||
# by a no-spec arm at each end, both within noise):
|
||||
#
|
||||
# vs no-spec floor conc=1 conc=4 conc=8 acceptance
|
||||
# k=1 +29.6% +40.7% +27.2% ~73%
|
||||
# k=2 +42.3% +52.4% +37.7% ~59%
|
||||
# k=3 +52.1% +51.3% +33.5% ~48%
|
||||
#
|
||||
# k=3 is set because this seat is single-user interactive, where conc=1
|
||||
# dominates and k=3 measured best (+52.1%, and the k3-over-k2 margin at
|
||||
# conc=1 cleared its own noise floor). For a batch-heavy seat prefer k=2:
|
||||
# it is best or statistically tied everywhere, and k=3's conc=4/8 figures
|
||||
# sit BELOW k=2 (within noise, so not a proven regression, but no gain).
|
||||
#
|
||||
# ⚠ TWO COSTS, both measured, neither optional:
|
||||
# 1. MTP adds ~5.08 GiB of draft-head weights (74.36 -> 79.44 GiB) and
|
||||
# raises per-token KV cost ~16% (26.24 -> 30.47 KiB/token). The seat's
|
||||
# previous 14 GiB FN_KV_CACHE_MEMORY OOMs at engine init with MTP on.
|
||||
# FN_KV_CACHE_MEMORY must come down; see .env.example.
|
||||
# 2. Run-to-run variance grows sharply with k (within-arm spread at conc=4:
|
||||
# 5.69 no-spec -> 34.61 k1 -> 57.06 k2 -> 76.90 k3). Throughput is
|
||||
# higher and LESS predictable. That is a real operational property.
|
||||
#
|
||||
# To turn MTP back off, delete these two lines — do not add a placeholder
|
||||
# value, there is no honest "off" for --speculative-config.
|
||||
- --speculative-config
|
||||
- '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
Reference in New Issue
Block a user