fix(mog-sec): retune to 0.52/420K after a runtime OOM at 0.55/480K

The 0.55 configuration booted cleanly, passed every startup check, and then
killed the engine on an ordinary request of roughly 20k tokens:
torch.OutOfMemoryError allocating 614 MiB in the GDN causal_conv1d path with
443 MiB free. The container self-recovered via restart: unless-stopped.

Context length was not the cause. gpu-memory-utilization is not a hard
reservation. Weights and KV are reserved up front, but activation memory is
allocated on demand and the profile that sizes it uses dummy data at
max_num_batched_tokens. The profile predicted 3.62 GiB; the process sat about
1 GiB over its entire budget even at idle. With six vLLM containers sharing
GPU1 there was no slack left to grow into, and a modest allocation lost the
race. 0.52 leaves roughly 7.7 GiB free against 4.6 GiB at 0.55.

Records a measurement that governs how max-model-len can be chosen at all:
the KV pool varies between boots. Two consecutive boots at 0.52 produced
425,663 and 453,600 tokens, a 6.6 percent swing, because the pool is sized
from free memory at startup and that depends on what the other tenants hold
at that instant. 420,000 sits under the lower observation so the seat starts
in both conditions. Sizing to the higher one would boot today and refuse
tomorrow, which is why the value was measured across boots rather than
estimated from one.

Concurrency of about 1.0 to 1.08x means one request at full length. Shorter
requests still batch normally, since vLLM schedules by tokens rather than by
max-model-len.

stacks/mog-sec/.env.example

@@ -1,8 +1,10 @@
 # mog-sec — pen-test seat (ana-ml2 GPU1, :8019). Copy to .env on the host.
 #
 # Values below are the configuration VALIDATED 2026-08-22: DFlash2 speculative
-# decoding on a newer vLLM, 480K context, 2048x2048 vision. Promoted from a
-# standalone experimental container after real-use testing.
+# decoding on a newer vLLM, 420K context, 2048x2048 vision. Promoted from a
+# standalone experimental container after real-use testing, then re-tuned down
+# from 0.55/480K after that combination crashed the engine in production (see
+# the memory notes below — it was a headroom failure, not a context failure).
 
 # ── Image ───────────────────────────────────────────────────────────────────
 # Contains DFlash2 (#52816) AND GDN spec-decode fix #53077. Verified by
@@ -39,15 +41,26 @@ MOG_SPEC_CONFIG={"method": "dflash", "model": "/drafter", "num_speculative_token
 # The model carries a complete YaRN config (rope_type yarn, factor 4.0,
 # original_max_position_embeddings 262144, max_position_embeddings 1000000),
 # so context is a KV-MEMORY choice, not a model limit.
-# ⚠ max-model-len must stay UNDER the KV pool or vLLM refuses to start.
-#   At 0.55 the pool is ~526,617 tokens -> 480,000 gives 1.10x concurrency.
-#   262144 instead would give ~1.86x. Straight trade: context vs concurrency.
-MOG_MAX_MODEL_LEN=480000
-# ⚠ 0.55 IS THE STABLE CEILING while GPU1's other tenants are up. 0.58 sized a
-#   bigger pool and then OOM'd during CUDA graph capture (process reached
-#   57.49 GiB against ~57.6 free). Real 1M context needs ~49 GiB of KV and so
-#   requires evicting most of GPU1 — a fleet decision, not a flag.
-MOG_GPU_MEM_UTIL=0.55
+# ⚠ max-model-len must stay UNDER the KV pool or vLLM refuses to start — and
+#   ⭐ THE POOL VARIES BOOT TO BOOT. Measured at 0.52 on two consecutive boots:
+#   425,663 then 453,600 tokens, a 6.6% swing. The pool is sized from FREE
+#   memory at startup, which depends on what GPU1's other five vLLM tenants
+#   happen to hold at that instant. 420,000 sits under the LOWER observation,
+#   so it starts reliably in both conditions. Sizing to the higher one would
+#   boot today and refuse tomorrow.
+#   Concurrency ~1.0-1.08x: one request at FULL length. Shorter requests still
+#   batch normally — vLLM schedules by tokens, not by max-model-len.
+MOG_MAX_MODEL_LEN=420000
+# ⚠ 0.55 CRASHED IN PRODUCTION. It booted fine and then died at RUNTIME on an
+#   ordinary ~20k-token request: torch.OutOfMemoryError allocating 614 MiB in
+#   the GDN causal_conv1d path, with 443 MiB free. gpu-memory-utilization is
+#   NOT a hard reservation — weights and KV are reserved, but ACTIVATION grows
+#   on demand past the startup profile (profiled 3.62 GiB on dummy data; the
+#   process sat ~1 GiB over its whole budget even at idle). With six vLLM
+#   containers sharing GPU1 there was no slack left to grow into.
+#   0.52 leaves ~7.7 GiB free vs ~4.6 GiB at 0.55. Higher needs fewer tenants.
+#   (0.58 failed even earlier — OOM during CUDA graph capture at startup.)
+MOG_GPU_MEM_UTIL=0.52
 MOG_MAX_NUM_SEQS=16
 MOG_MAX_NUM_BATCHED_TOKENS=16384
 MOG_KV_CACHE_DTYPE=fp8