diff --git a/stacks/mog-sec/.env.example b/stacks/mog-sec/.env.example index 3269abe..2f5a992 100644 --- a/stacks/mog-sec/.env.example +++ b/stacks/mog-sec/.env.example @@ -1,8 +1,10 @@ # mog-sec — pen-test seat (ana-ml2 GPU1, :8019). Copy to .env on the host. # # Values below are the configuration VALIDATED 2026-08-22: DFlash2 speculative -# decoding on a newer vLLM, 480K context, 2048x2048 vision. Promoted from a -# standalone experimental container after real-use testing. +# decoding on a newer vLLM, 420K context, 2048x2048 vision. Promoted from a +# standalone experimental container after real-use testing, then re-tuned down +# from 0.55/480K after that combination crashed the engine in production (see +# the memory notes below — it was a headroom failure, not a context failure). # ── Image ─────────────────────────────────────────────────────────────────── # Contains DFlash2 (#52816) AND GDN spec-decode fix #53077. Verified by @@ -39,15 +41,26 @@ MOG_SPEC_CONFIG={"method": "dflash", "model": "/drafter", "num_speculative_token # The model carries a complete YaRN config (rope_type yarn, factor 4.0, # original_max_position_embeddings 262144, max_position_embeddings 1000000), # so context is a KV-MEMORY choice, not a model limit. -# ⚠ max-model-len must stay UNDER the KV pool or vLLM refuses to start. -# At 0.55 the pool is ~526,617 tokens -> 480,000 gives 1.10x concurrency. -# 262144 instead would give ~1.86x. Straight trade: context vs concurrency. -MOG_MAX_MODEL_LEN=480000 -# ⚠ 0.55 IS THE STABLE CEILING while GPU1's other tenants are up. 0.58 sized a -# bigger pool and then OOM'd during CUDA graph capture (process reached -# 57.49 GiB against ~57.6 free). Real 1M context needs ~49 GiB of KV and so -# requires evicting most of GPU1 — a fleet decision, not a flag. -MOG_GPU_MEM_UTIL=0.55 +# ⚠ max-model-len must stay UNDER the KV pool or vLLM refuses to start — and +# ⭐ THE POOL VARIES BOOT TO BOOT. Measured at 0.52 on two consecutive boots: +# 425,663 then 453,600 tokens, a 6.6% swing. The pool is sized from FREE +# memory at startup, which depends on what GPU1's other five vLLM tenants +# happen to hold at that instant. 420,000 sits under the LOWER observation, +# so it starts reliably in both conditions. Sizing to the higher one would +# boot today and refuse tomorrow. +# Concurrency ~1.0-1.08x: one request at FULL length. Shorter requests still +# batch normally — vLLM schedules by tokens, not by max-model-len. +MOG_MAX_MODEL_LEN=420000 +# ⚠ 0.55 CRASHED IN PRODUCTION. It booted fine and then died at RUNTIME on an +# ordinary ~20k-token request: torch.OutOfMemoryError allocating 614 MiB in +# the GDN causal_conv1d path, with 443 MiB free. gpu-memory-utilization is +# NOT a hard reservation — weights and KV are reserved, but ACTIVATION grows +# on demand past the startup profile (profiled 3.62 GiB on dummy data; the +# process sat ~1 GiB over its whole budget even at idle). With six vLLM +# containers sharing GPU1 there was no slack left to grow into. +# 0.52 leaves ~7.7 GiB free vs ~4.6 GiB at 0.55. Higher needs fewer tenants. +# (0.58 failed even earlier — OOM during CUDA graph capture at startup.) +MOG_GPU_MEM_UTIL=0.52 MOG_MAX_NUM_SEQS=16 MOG_MAX_NUM_BATCHED_TOKENS=16384 MOG_KV_CACHE_DTYPE=fp8