diff --git a/persistent-memory.md b/persistent-memory.md index bd82b40..495d5c6 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,7 +206,7 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions -- `[2026-09-10]` **`sec`/`sec-reasoning` (mog-sec, ana-ml2 GPU0 `:8019`) crashed on a CUDA OOM and auto-recovered in 3 s — GPU0 runs at ~95% and has ~4.6 GiB of true headroom.** First and only fatal error since the container was created 2026-08-28 (`RestartCount=1`), so this is rare, not chronic. Trigger: a request already **151,728 tokens deep** scheduling a further 15,696-token chunk needed a ~1.04 GiB allocation with ~600 MB free; EngineCore took a fatal error, the API server exited 0, and `restart: unless-stopped` brought it back (`manualRestart=false` in the dockerd journal — nobody restarted it by hand). **NOT caused by the same-day char-rp-fast seat swap**: all of that was on GPU1 (`erp-seat` is pinned `device_ids: ["1"]`) and GPU0 read 93,257 MiB both before the work started and after it finished. ⚠ **The two GPU0 seats reserve 0.38 (`gen`) + 0.52 (`mog-sec`) = 0.90 of the card**, and vLLM's utilization figure covers weights + KV pool but NOT all transient activation memory — long-context prefill chunks with speculative decoding live in what is left. Lever if it recurs: drop `mog-sec`'s `--gpu-memory-utilization` 0.52 → ~0.50, or pin its KV in bytes the way `erp-seat` does so the reservation stops being ratio-dependent. **Do NOT reach for `num_speculative_tokens`** — see the trap below. +- `[2026-09-10]` **GPU0 headroom FIXED: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` / `MOG_MAX_MODEL_LEN=393216` (was 0.52 / 420000), after `sec`/`sec-reasoning` crash-bounced TWICE in ten minutes.** ⚠ **I initially called it "rare, not chronic" off a `RestartCount=1` and recommended doing nothing — the operator pushed back and was right; it recurred inside ten minutes** (fatal errors 20:20:30Z and 20:30:02Z). Cause: `mog-sec` (0.52) + `gen` (0.38) reserve 0.90 of the card, leaving ~4.6 GiB, and vLLM's utilization figure covers weights + KV pool but **NOT** transient activation memory — a request ~151,700 tokens deep scheduling a ~15,700-token chunk asked for ~1.04 GiB with ~600 MB free. ⚠⚠ **0.50 and 420000 are MUTUALLY EXCLUSIVE and dropping util alone makes it crash-loop at STARTUP instead**: one 420k sequence needs 17.88 GiB of KV, the 0.50 pool is 17.4–17.5 GiB, so vLLM refuses to start outright. ⚠ **Do NOT pin `max_model_len` to vLLM's suggested maximum** — it estimated 406,352 on one boot and 409,840 on the next (available-KV drifts ~0.1 GiB per boot), so the edge value fails on an unlucky boot; 393,216 sits 3% under the lower estimate and leaves ~0.7 GiB of the pool unspent, which is the transient headroom the change exists to buy. Verified after: KV 405,612 tokens, concurrency 1.03x, both aliases 200 through the gateway. Backup `.env.bak-util052-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. - `[2026-09-10]` ⚠ **Near-miss on measurement discipline, worth keeping as a specimen.** The crash window logged `Avg Draft acceptance rate: 17.6%` and per-position rates of 0.049/0.024/0.015 for draft positions 5–7, which reads as an obvious "cut `num_speculative_tokens` 7 → 3, it is buying nothing." Across **180 samples** of the same counter over the container's life the real distribution is **median acceptance length 3.12 of 7 (range 1.83–6.75)** and **median draft acceptance 30.4% (range 11.9–82.1%)** — the crash window was near the *minimum*, not the norm, and cutting to 3 would cap the workloads that were accepting nearly the full 7-wide draft. **The n=1 window pointed the opposite way from the n=180 distribution.** Same session that wrote "a positive control is only worth what it can distinguish"; the lesson generalises to log lines. - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` diff --git a/stacks/mog-sec/compose.yaml b/stacks/mog-sec/compose.yaml index 7bd9428..608fedb 100644 --- a/stacks/mog-sec/compose.yaml +++ b/stacks/mog-sec/compose.yaml @@ -23,6 +23,37 @@ # ships a deployment kit for — neither is our vLLM serving surface. 262K is the honest # native ceiling here; a real 1M seat would be a separate SGLang project. # +# ⚠⚠ GPU0 HEADROOM — the 2026-09-10 crash pair, and why the live .env now reads +# MOG_GPU_MEM_UTIL=0.50 / MOG_MAX_MODEL_LEN=393216 rather than 0.52 / 420000. +# +# At util 0.52 this seat and `gen` (0.38) together reserve 0.90 of the card, leaving +# ~4.6 GiB. vLLM's utilization figure covers weights and the KV pool but NOT all +# transient activation memory, and a long-context prefill chunk with the 7-wide dflash +# drafter lives in what is left. Twice in ten minutes (20:20:30Z and 20:30:02Z) a request +# ~151,700 tokens deep scheduling a further ~15,700-token chunk asked for ~1.04 GiB with +# ~600 MB free, EngineCore took a fatal error, and `restart: unless-stopped` bounced the +# seat. Each bounce is a hard 500 to every in-flight caller. +# +# ⚠ 0.50 AND 420000 ARE MUTUALLY EXCLUSIVE — dropping util alone does NOT work and the +# seat will crash-loop at startup instead of at runtime. A single 420,000-token sequence +# needs 17.88 GiB of KV; at 0.50 the pool is 17.4-17.5 GiB, so vLLM refuses: +# "To serve at least one request with the model's max seq len (420000), 17.88 GiB KV +# cache is needed, which is larger than the available KV cache memory (17.41 GiB)." +# The context length and the crash were directly coupled: 420k was only reachable at the +# utilization that left no transient headroom. +# +# ⚠ DO NOT pin max_model_len to vLLM's suggested maximum. It estimated 406,352 on one +# boot and 409,840 on the next -- the available-KV figure drifts ~0.1 GiB boot to boot, so +# the edge value fails to start on an unlucky one. 393,216 (384k) sits 3% under the lower +# estimate and leaves ~0.7 GiB of the pool unspent, which IS the transient headroom this +# change exists to buy. Measured after: KV 405,612 tokens, concurrency 1.03x at 393,216. +# +# ⚠ NOT the lever: `num_speculative_tokens`. The crash window logged 17.6% draft +# acceptance with positions 5-7 at 1.5-4.9%, which reads as an obvious cut from 7 to 3. +# Across 180 samples of that counter the median acceptance LENGTH is 3.12 of 7 (range +# 1.83-6.75) and median draft acceptance is 30.4% (range 11.9-82.1%). The crash window sat +# near the minimum; cutting to 3 would cap the workloads accepting nearly the full draft. +# # Two served-names (base + `-thinking`): LiteLLM keys deployments by (model, api_base), # so mog-sec and mog-sec-reasoning use distinct names to avoid the shared-config # enable_thinking clobber. Same pinned nightly as the gen seat (carries the #51113