diff --git a/persistent-memory.md b/persistent-memory.md index 495d5c6..15a9153 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,7 +206,9 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions -- `[2026-09-10]` **GPU0 headroom FIXED: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` / `MOG_MAX_MODEL_LEN=393216` (was 0.52 / 420000), after `sec`/`sec-reasoning` crash-bounced TWICE in ten minutes.** ⚠ **I initially called it "rare, not chronic" off a `RestartCount=1` and recommended doing nothing — the operator pushed back and was right; it recurred inside ten minutes** (fatal errors 20:20:30Z and 20:30:02Z). Cause: `mog-sec` (0.52) + `gen` (0.38) reserve 0.90 of the card, leaving ~4.6 GiB, and vLLM's utilization figure covers weights + KV pool but **NOT** transient activation memory — a request ~151,700 tokens deep scheduling a ~15,700-token chunk asked for ~1.04 GiB with ~600 MB free. ⚠⚠ **0.50 and 420000 are MUTUALLY EXCLUSIVE and dropping util alone makes it crash-loop at STARTUP instead**: one 420k sequence needs 17.88 GiB of KV, the 0.50 pool is 17.4–17.5 GiB, so vLLM refuses to start outright. ⚠ **Do NOT pin `max_model_len` to vLLM's suggested maximum** — it estimated 406,352 on one boot and 409,840 on the next (available-KV drifts ~0.1 GiB per boot), so the edge value fails on an unlucky boot; 393,216 sits 3% under the lower estimate and leaves ~0.7 GiB of the pool unspent, which is the transient headroom the change exists to buy. Verified after: KV 405,612 tokens, concurrency 1.03x, both aliases 200 through the gateway. Backup `.env.bak-util052-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. +- `[2026-09-10]` **BabyBronte H02 adapter: the VOICE transferred, the SENSE did not — operator's read, "it's all nonsense, but it sounds like Brontë's nonsense."** Eyeball A/B (NOT the adjudication; nothing here feeds the frozen rule), 9 arbitrary prompts on a deliberate difficulty gradient × 2 arms × 2 seeds, booth at `http://10.100.10.50:8090/b/babybronte-voice/`. Measured across the 18 pairs: **curly quotes 1/18 base → 18/18 tuned**, **math/worksheet collapse 3/18 base → 0/18 tuned**. Given *"The self-checkout machine refused her coupon"* the base 0.6B produced a **quadratic-formula worksheet**; the tuned arm wrote a clerk refusing a customer in Victorian retrospective first person. This is the expected and informative result for the smallest rung — **voice is separable from coherence at 0.6B**, which is the premise the whole lightweight-adapter regime rests on, and the 1.7B/4B rungs are where sense should return. The 1-epoch loss was still descending at step 169 (undertrained, not overfit), so the incoherence is carrier capacity, not training. ⚠ **Corpus-prep defect found: the tuned output is hard-wrapped at ~70 chars** (median mid-length-line ratio 0.85 vs base 0.00) — the Gutenberg source kept its original line breaks and the adapter learned the typography along with the voice. Unwrap to flowing paragraphs before any real use or the next rung learns it too. + +- `[2026-09-10]` **GPU0 SETTLED after THREE crashes and two failed attempts: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` + **`MOG_KV_CACHE_MEMORY=17697765376` (KV PINNED IN BYTES)** + `MOG_MAX_MODEL_LEN=327680` (was 0.52 / ratio-inferred KV / 420000).** ⚠ **I got the severity call wrong first**: off a `RestartCount=1` I said "rare, not chronic, do nothing"; the operator pushed back and it crashed twice more (20:20:30Z, 20:30:02Z at the old settings, then 21:46:35Z after the first fix). ⚠ **Dropping utilization alone is NOT the fix and fails in a worse way** — at 0.50 with 420k the seat crash-loops at STARTUP, because one 420k sequence needs 17.88 GiB of KV and the 0.50 pool is 17.4–17.5. ⚠⚠ **THE REAL FINDING: `--gpu-memory-utilization` does NOT bound actual usage.** It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line proved mog-sec was running **0.9 GiB OVER** its 47.48 GiB reservation (26.44 consumed + 3.53 peak act + 0.89 CUDAGraph + 17.52 KV = 48.38), and **`gen` was over by 0.33 on the same card** — that overage came out of the shared slack and is what kept OOMing. Fix is the one vLLM printed itself: pin KV in bytes at its recommended 16.48 GiB, same discipline `erp-seat` already uses. **The KV pin and the context length are COUPLED** — 16.48 GiB yields ~383,730 tokens, so 393,216 falls under the 1.0x floor and pinning KV while keeping 384k was never available. Verified after: `reserved 16.48 GiB ... as specified by kv_cache_memory_bytes config and skipped memory profiling`, KV 375,901 tokens, **concurrency 1.15x** (was 1.03x), GPU0 90,561 MiB (~750 MiB reclaimed), both aliases 200 through the gateway. Backups `.env.bak-util052-20260910` and `.env.bak-384k-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. Cost: 320k context instead of 420k, on a seat whose crashes happened at ~151k. - `[2026-09-10]` ⚠ **Near-miss on measurement discipline, worth keeping as a specimen.** The crash window logged `Avg Draft acceptance rate: 17.6%` and per-position rates of 0.049/0.024/0.015 for draft positions 5–7, which reads as an obvious "cut `num_speculative_tokens` 7 → 3, it is buying nothing." Across **180 samples** of the same counter over the container's life the real distribution is **median acceptance length 3.12 of 7 (range 1.83–6.75)** and **median draft acceptance 30.4% (range 11.9–82.1%)** — the crash window was near the *minimum*, not the norm, and cutting to 3 would cap the workloads that were accepting nearly the full 7-wide draft. **The n=1 window pointed the opposite way from the n=180 distribution.** Same session that wrote "a positive control is only worth what it can distinguish"; the lesson generalises to log lines. - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` diff --git a/stacks/mog-sec/compose.yaml b/stacks/mog-sec/compose.yaml index 608fedb..57f57d2 100644 --- a/stacks/mog-sec/compose.yaml +++ b/stacks/mog-sec/compose.yaml @@ -99,7 +99,26 @@ services: - ${MOG_QUANT:-compressed-tensors} - --gpu-memory-utilization - ${MOG_GPU_MEM_UTIL:-0.44} + # ⚠ KV PINNED IN BYTES, added 2026-09-10 after the crash pair above. The + # utilization ratio does NOT bound actual usage -- it sizes the KV calculation, + # but peak activation is measured at profiling time and real long-context work + # exceeds the profile. vLLM's own budget line proved this seat was running 0.9 GiB + # OVER its 47.48 GiB reservation at util 0.50 (26.44 consumed + 3.53 peak activation + # + 0.89 CUDAGraph + 17.52 KV = 48.38), and `gen` was over by 0.33 on the same card. + # That overage came out of the shared card's slack, which is what kept OOMing. + # + # 17,697,765,376 B = 16.48 GiB is vLLM's OWN recommended figure from that line + # ("Replace gpu_memory_utilization config with --kv-cache-memory=17697765376 to fit + # into requested memory"), not a value anyone here invented. Same discipline as + # stacks/erp-seat: an explicit figure is reproducible, a ratio silently yields a + # different cache depending on what else is resident at start time. + - --kv-cache-memory + - ${MOG_KV_CACHE_MEMORY:-17697765376} - --max-model-len + # ⚠ 320k, NOT 384k, and the two settings are coupled -- 16.48 GiB of KV yields about + # 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM + # refuses to START rather than crashing later. Pinning the KV and keeping 384k was + # never an available combination. 327,680 leaves ~1.17x. - ${MOG_MAX_MODEL_LEN:-262144} - --max-num-seqs - ${MOG_MAX_NUM_SEQS:-16}