Record the mog-sec GPU0 OOM crash and the spec-decode n=1 near-miss

sec/sec-reasoning on ana-ml2 :8019 took a CUDA OOM on GPU0 at 13:20 PT and
auto-recovered in three seconds. First fatal error since the container was
created on 2026-08-28. A request already 151,728 tokens deep scheduled a further
15,696-token chunk, needed a 1.04 GB allocation, and found about 600 MB free.
EngineCore died, the API server exited 0, and restart: unless-stopped brought it
back; the dockerd journal records manualRestart=false, so nothing restarted it by
hand.

Not caused by the same-day char-rp-fast swap. That work was entirely on GPU1 --
erp-seat is pinned device_ids ["1"] -- and GPU0 read 93,257 MiB both before it
started and after it finished.

The standing condition is that GPU0's two seats reserve 0.38 plus 0.52 of the
card between them, leaving about 4.6 GiB, and vLLM's utilization figure covers
weights and the KV pool but not all transient activation memory. Long-context
prefill chunks with speculative decoding live in what is left.

Also recorded as a specimen: the crash window logged a 17.6% draft acceptance
rate with positions 5 through 7 accepting 1.5 to 4.9 percent, which reads as an
obvious case for cutting num_speculative_tokens from 7 to 3. Across 180 samples
of the same counter the median acceptance length is 3.12 of 7 (range 1.83 to
6.75) and the median draft acceptance is 30.4% (range 11.9 to 82.1). The crash
window sat near the minimum. The single window pointed the opposite way from the
distribution, and acting on it would have capped the workloads that were
accepting nearly the full draft.

No configuration changed. One self-healing failure in thirteen days does not
justify shrinking a KV pool on a seat configured for 420k context.

persistent-memory.md

@@ -206,6 +206,9 @@ _As of 2026-09-10 10:25 PT._
 
 ## Recent decisions
 
+- `[2026-09-10]` **`sec`/`sec-reasoning` (mog-sec, ana-ml2 GPU0 `:8019`) crashed on a CUDA OOM and auto-recovered in 3 s — GPU0 runs at ~95% and has ~4.6 GiB of true headroom.** First and only fatal error since the container was created 2026-08-28 (`RestartCount=1`), so this is rare, not chronic. Trigger: a request already **151,728 tokens deep** scheduling a further 15,696-token chunk needed a ~1.04 GiB allocation with ~600 MB free; EngineCore took a fatal error, the API server exited 0, and `restart: unless-stopped` brought it back (`manualRestart=false` in the dockerd journal — nobody restarted it by hand). **NOT caused by the same-day char-rp-fast seat swap**: all of that was on GPU1 (`erp-seat` is pinned `device_ids: ["1"]`) and GPU0 read 93,257 MiB both before the work started and after it finished. ⚠ **The two GPU0 seats reserve 0.38 (`gen`) + 0.52 (`mog-sec`) = 0.90 of the card**, and vLLM's utilization figure covers weights + KV pool but NOT all transient activation memory — long-context prefill chunks with speculative decoding live in what is left. Lever if it recurs: drop `mog-sec`'s `--gpu-memory-utilization` 0.52 → ~0.50, or pin its KV in bytes the way `erp-seat` does so the reservation stops being ratio-dependent. **Do NOT reach for `num_speculative_tokens`** — see the trap below.
+- `[2026-09-10]` ⚠ **Near-miss on measurement discipline, worth keeping as a specimen.** The crash window logged `Avg Draft acceptance rate: 17.6%` and per-position rates of 0.049/0.024/0.015 for draft positions 5–7, which reads as an obvious "cut `num_speculative_tokens` 7 → 3, it is buying nothing." Across **180 samples** of the same counter over the container's life the real distribution is **median acceptance length 3.12 of 7 (range 1.83–6.75)** and **median draft acceptance 30.4% (range 11.9–82.1%)** — the crash window was near the *minimum*, not the norm, and cutting to 3 would cap the workloads that were accepting nearly the full 7-wide draft. **The n=1 window pointed the opposite way from the n=180 distribution.** Same session that wrote "a positive control is only worth what it can distinguish"; the lesson generalises to log lines.
+
 - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md`
 - `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`.
 - `[2026-09-10]` **MeroMero A4B swapped onto the `erp-seat` seat as `char-rp-fast`; `Pfish-6` alias removed.** The A4B's FIRST quant used the dense recipe and 4-bit-quantized all 30 MoE routers — it passed its healthcheck and answered every request with the full token count decoding to the empty string, NaN logits the only tell. Re-quantized with the MoE recipe; live and verified (prose, vision, tool call, finite logprobs). Durable lesson: **a positive control must match the ARCHITECTURE CLASS** — the broken A4B was diffed against a good *dense* quant, which has no routers, so the clean result was meaningless. → playbook §3.15, §4.4