docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers

Read from docker inspect, nvidia-smi and each model's own config.json rather
than from compose files or the gateway config, so the document records what is
deployed rather than what was intended.

Findings worth naming:

- char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on
  ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer
  128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it,
  but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment
  block carries sampler defaults keyed to the model it names.
- Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base
  (:8099). Nothing is listening on either; callers get a connection failure
  rather than a clear retirement error.
- Three different speculative-decoding methods coexist -- dflash k=7 with a
  separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next.
  The method string is architecture-specific; copying a --speculative-config
  between seats without changing it will not work.
- Three seats run the floating :latest tag, so what they would pull on recreate
  is not what is running and was never recorded.
- Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw
  during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn.

Also commits the flash-next-seat MTP k=3 change made today: compose gains the
speculative-config with its measured rationale, and .env.example documents the
KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
This commit is contained in:
vh
2026-09-13 22:28:58 -07:00
parent f9a8f176ca
commit 2d83a895c1
3 changed files with 194 additions and 9 deletions
+8 -1
View File
@@ -54,7 +54,14 @@ FN_GPU_MEM_UTIL=0.96
# Re-raising requires re-running the deep probe and reading the allocator log.
# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB
# still free on the card at peak.
FN_KV_CACHE_MEMORY=15032385536
# ⚠ MTP ON (k=3) REQUIRES A SMALLER KV POOL THAN THE NO-SPEC SEAT.
# Measured 2026-09-13: the draft head adds ~5.08 GiB of weights and raises
# per-token KV cost ~16%. At the old 15032385536 (14 GiB) the engine OOMs at
# init with MTP enabled -- 79.44 GiB weights + 14 GiB KV = 93.44 against a
# 91.17 GiB budget. 8589934592 (8 GiB) is PROVEN to boot; 10737418240 (10 GiB)
# leaves ~1.73 GiB for activations and is the current setting. If the seat
# OOMs at init, drop to 8589934592.
FN_KV_CACHE_MEMORY=10737418240
# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K
# tokens, so a single max-length request fits with ~2.4x concurrency to spare.