docs(fv-ml1): reconcile GPU seat inventory and model lineage against running containers
Read from docker inspect, nvidia-smi and each model's own config.json rather than from compose files or the gateway config, so the document records what is deployed rather than what was intended. Findings worth naming: - char-rp drift. The LiteLLM config documents char-rp as G4-MeroMero-v2-31B on ana-ml2 GPU 0; :8016 actually serves gemma4-26b-a4b-it-nvfp4, a 26B 30-layer 128-expert stock Gemma-4, on fv-ml1 GPU 1. Host renaming explains part of it, but 31B MeroMero-v2 and 26B stock gemma4 are different models and the comment block carries sampler defaults keyed to the model it names. - Two gateway aliases are dead: erp-tune-v2 (:8098) and gemma4-26b-a4b-it-base (:8099). Nothing is listening on either; callers get a connection failure rather than a clear retirement error. - Three different speculative-decoding methods coexist -- dflash k=7 with a separate drafter on mog-sec, qwen3_5_mtp k=3 on gen, mtp k=3 on flash-next. The method string is architecture-specific; copying a --speculative-config between seats without changing it will not work. - Three seats run the floating :latest tag, so what they would pull on recreate is not what is running and was never recorded. - Seven aliases resolve to vllm-gen on :8015, which is why GPU 0 shows draw during ordinary OpenWebUI use -- its auto-titling fires summarizer per turn. Also commits the flash-next-seat MTP k=3 change made today: compose gains the speculative-config with its measured rationale, and .env.example documents the KV-cache reduction MTP requires (14 GiB OOMs, 10 GiB verified in production).
This commit is contained in:
@@ -54,7 +54,14 @@ FN_GPU_MEM_UTIL=0.96
|
||||
# Re-raising requires re-running the deep probe and reading the allocator log.
|
||||
# VERIFIED at 14.00 GiB: 0 OOM warnings, 155,557-token prefill in 14.2 s, 2,085 MiB
|
||||
# still free on the card at peak.
|
||||
FN_KV_CACHE_MEMORY=15032385536
|
||||
# ⚠ MTP ON (k=3) REQUIRES A SMALLER KV POOL THAN THE NO-SPEC SEAT.
|
||||
# Measured 2026-09-13: the draft head adds ~5.08 GiB of weights and raises
|
||||
# per-token KV cost ~16%. At the old 15032385536 (14 GiB) the engine OOMs at
|
||||
# init with MTP enabled -- 79.44 GiB weights + 14 GiB KV = 93.44 against a
|
||||
# 91.17 GiB budget. 8589934592 (8 GiB) is PROVEN to boot; 10737418240 (10 GiB)
|
||||
# leaves ~1.73 GiB for activations and is the current setting. If the seat
|
||||
# OOMs at init, drop to 8589934592.
|
||||
FN_KV_CACHE_MEMORY=10737418240
|
||||
|
||||
# FULL NATIVE 262,144 (operator-directed 2026-09-13). The KV pool holds ~641K
|
||||
# tokens, so a single max-length request fits with ~2.4x concurrency to spare.
|
||||
|
||||
Reference in New Issue
Block a user