diff --git a/persistent-memory.md b/persistent-memory.md index 8a11870..bd82b40 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,6 +206,9 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions +- `[2026-09-10]` **`sec`/`sec-reasoning` (mog-sec, ana-ml2 GPU0 `:8019`) crashed on a CUDA OOM and auto-recovered in 3 s — GPU0 runs at ~95% and has ~4.6 GiB of true headroom.** First and only fatal error since the container was created 2026-08-28 (`RestartCount=1`), so this is rare, not chronic. Trigger: a request already **151,728 tokens deep** scheduling a further 15,696-token chunk needed a ~1.04 GiB allocation with ~600 MB free; EngineCore took a fatal error, the API server exited 0, and `restart: unless-stopped` brought it back (`manualRestart=false` in the dockerd journal — nobody restarted it by hand). **NOT caused by the same-day char-rp-fast seat swap**: all of that was on GPU1 (`erp-seat` is pinned `device_ids: ["1"]`) and GPU0 read 93,257 MiB both before the work started and after it finished. ⚠ **The two GPU0 seats reserve 0.38 (`gen`) + 0.52 (`mog-sec`) = 0.90 of the card**, and vLLM's utilization figure covers weights + KV pool but NOT all transient activation memory — long-context prefill chunks with speculative decoding live in what is left. Lever if it recurs: drop `mog-sec`'s `--gpu-memory-utilization` 0.52 → ~0.50, or pin its KV in bytes the way `erp-seat` does so the reservation stops being ratio-dependent. **Do NOT reach for `num_speculative_tokens`** — see the trap below. +- `[2026-09-10]` ⚠ **Near-miss on measurement discipline, worth keeping as a specimen.** The crash window logged `Avg Draft acceptance rate: 17.6%` and per-position rates of 0.049/0.024/0.015 for draft positions 5–7, which reads as an obvious "cut `num_speculative_tokens` 7 → 3, it is buying nothing." Across **180 samples** of the same counter over the container's life the real distribution is **median acceptance length 3.12 of 7 (range 1.83–6.75)** and **median draft acceptance 30.4% (range 11.9–82.1%)** — the crash window was near the *minimum*, not the norm, and cutting to 3 would cap the workloads that were accepting nearly the full 7-wide draft. **The n=1 window pointed the opposite way from the n=180 distribution.** Same session that wrote "a positive control is only worth what it can distinguish"; the lesson generalises to log lines. + - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` - `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`. - `[2026-09-10]` **MeroMero A4B swapped onto the `erp-seat` seat as `char-rp-fast`; `Pfish-6` alias removed.** The A4B's FIRST quant used the dense recipe and 4-bit-quantized all 30 MoE routers — it passed its healthcheck and answered every request with the full token count decoding to the empty string, NaN logits the only tell. Re-quantized with the MoE recipe; live and verified (prose, vision, tool call, finite logprobs). Durable lesson: **a positive control must match the ARCHITECTURE CLASS** — the broken A4B was diffed against a good *dense* quant, which has no routers, so the clean result was meaningless. → playbook §3.15, §4.4