From 8842ffe1fe904c7d345fe831ab3e93712e9c9c36 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 10 Sep 2026 15:19:47 -0700 Subject: [PATCH] Pin mog-sec's KV in bytes at 16.48 GiB and take it to 320k context Operator: "yes, pin the kv and take it to 320k." The real finding, which took three crashes and two failed attempts to reach: --gpu-memory-utilization does not bound actual usage. It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line showed mog-sec running 0.9 GiB over its 47.48 GiB reservation -- 26.44 consumed plus 3.53 peak activation plus 0.89 CUDAGraph plus 17.52 KV equals 48.38 -- and gen was over by 0.33 on the same card. That overage came out of the shared card's slack, which is what kept OOMing after the utilization drop. The fix is the one vLLM printed itself: --kv-cache-memory=17697765376, its own recommended figure to fit inside the requested budget. Same discipline erp-seat already uses, and for the same stated reason -- an explicit figure is reproducible where a ratio silently yields a different cache depending on what else is resident at start time. The KV pin and the context length are coupled. 16.48 GiB yields about 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM refuses to start rather than crashing later; pinning the KV while keeping 384k was never an available combination. 327,680 leaves 1.15x, up from 1.03x. Verified: the engine now logs "reserved 16.48 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling", KV 375,901 tokens, GPU0 down to 90,561 MiB from 91,313, RestartCount 0, and both sec and sec-reasoning return 200 through the gateway. Also records the BabyBronte eyeball A/B, whose result is the operator's own: the voice transferred and the sense did not. Curly quotes went 1 of 18 to 18 of 18 and worksheet collapse 3 of 18 to 0 of 18 between arms. That voice is separable from coherence at 0.6B is the premise the lightweight-adapter regime rests on, so this is the informative outcome rather than a disappointing one. A corpus-prep defect surfaced with it: the tuned output is hard-wrapped at about 70 characters because the Gutenberg source kept its line breaks and the adapter learned the typography too. Cost: 320k of context instead of 420k, on a seat whose crashes happened at 151k. --- persistent-memory.md | 4 +++- stacks/mog-sec/compose.yaml | 19 +++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index 495d5c6..15a9153 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -206,7 +206,9 @@ _As of 2026-09-10 10:25 PT._ ## Recent decisions -- `[2026-09-10]` **GPU0 headroom FIXED: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` / `MOG_MAX_MODEL_LEN=393216` (was 0.52 / 420000), after `sec`/`sec-reasoning` crash-bounced TWICE in ten minutes.** ⚠ **I initially called it "rare, not chronic" off a `RestartCount=1` and recommended doing nothing — the operator pushed back and was right; it recurred inside ten minutes** (fatal errors 20:20:30Z and 20:30:02Z). Cause: `mog-sec` (0.52) + `gen` (0.38) reserve 0.90 of the card, leaving ~4.6 GiB, and vLLM's utilization figure covers weights + KV pool but **NOT** transient activation memory — a request ~151,700 tokens deep scheduling a ~15,700-token chunk asked for ~1.04 GiB with ~600 MB free. ⚠⚠ **0.50 and 420000 are MUTUALLY EXCLUSIVE and dropping util alone makes it crash-loop at STARTUP instead**: one 420k sequence needs 17.88 GiB of KV, the 0.50 pool is 17.4–17.5 GiB, so vLLM refuses to start outright. ⚠ **Do NOT pin `max_model_len` to vLLM's suggested maximum** — it estimated 406,352 on one boot and 409,840 on the next (available-KV drifts ~0.1 GiB per boot), so the edge value fails on an unlucky boot; 393,216 sits 3% under the lower estimate and leaves ~0.7 GiB of the pool unspent, which is the transient headroom the change exists to buy. Verified after: KV 405,612 tokens, concurrency 1.03x, both aliases 200 through the gateway. Backup `.env.bak-util052-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. +- `[2026-09-10]` **BabyBronte H02 adapter: the VOICE transferred, the SENSE did not — operator's read, "it's all nonsense, but it sounds like Brontë's nonsense."** Eyeball A/B (NOT the adjudication; nothing here feeds the frozen rule), 9 arbitrary prompts on a deliberate difficulty gradient × 2 arms × 2 seeds, booth at `http://10.100.10.50:8090/b/babybronte-voice/`. Measured across the 18 pairs: **curly quotes 1/18 base → 18/18 tuned**, **math/worksheet collapse 3/18 base → 0/18 tuned**. Given *"The self-checkout machine refused her coupon"* the base 0.6B produced a **quadratic-formula worksheet**; the tuned arm wrote a clerk refusing a customer in Victorian retrospective first person. This is the expected and informative result for the smallest rung — **voice is separable from coherence at 0.6B**, which is the premise the whole lightweight-adapter regime rests on, and the 1.7B/4B rungs are where sense should return. The 1-epoch loss was still descending at step 169 (undertrained, not overfit), so the incoherence is carrier capacity, not training. ⚠ **Corpus-prep defect found: the tuned output is hard-wrapped at ~70 chars** (median mid-length-line ratio 0.85 vs base 0.00) — the Gutenberg source kept its original line breaks and the adapter learned the typography along with the voice. Unwrap to flowing paragraphs before any real use or the next rung learns it too. + +- `[2026-09-10]` **GPU0 SETTLED after THREE crashes and two failed attempts: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` + **`MOG_KV_CACHE_MEMORY=17697765376` (KV PINNED IN BYTES)** + `MOG_MAX_MODEL_LEN=327680` (was 0.52 / ratio-inferred KV / 420000).** ⚠ **I got the severity call wrong first**: off a `RestartCount=1` I said "rare, not chronic, do nothing"; the operator pushed back and it crashed twice more (20:20:30Z, 20:30:02Z at the old settings, then 21:46:35Z after the first fix). ⚠ **Dropping utilization alone is NOT the fix and fails in a worse way** — at 0.50 with 420k the seat crash-loops at STARTUP, because one 420k sequence needs 17.88 GiB of KV and the 0.50 pool is 17.4–17.5. ⚠⚠ **THE REAL FINDING: `--gpu-memory-utilization` does NOT bound actual usage.** It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line proved mog-sec was running **0.9 GiB OVER** its 47.48 GiB reservation (26.44 consumed + 3.53 peak act + 0.89 CUDAGraph + 17.52 KV = 48.38), and **`gen` was over by 0.33 on the same card** — that overage came out of the shared slack and is what kept OOMing. Fix is the one vLLM printed itself: pin KV in bytes at its recommended 16.48 GiB, same discipline `erp-seat` already uses. **The KV pin and the context length are COUPLED** — 16.48 GiB yields ~383,730 tokens, so 393,216 falls under the 1.0x floor and pinning KV while keeping 384k was never available. Verified after: `reserved 16.48 GiB ... as specified by kv_cache_memory_bytes config and skipped memory profiling`, KV 375,901 tokens, **concurrency 1.15x** (was 1.03x), GPU0 90,561 MiB (~750 MiB reclaimed), both aliases 200 through the gateway. Backups `.env.bak-util052-20260910` and `.env.bak-384k-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. Cost: 320k context instead of 420k, on a seat whose crashes happened at ~151k. - `[2026-09-10]` ⚠ **Near-miss on measurement discipline, worth keeping as a specimen.** The crash window logged `Avg Draft acceptance rate: 17.6%` and per-position rates of 0.049/0.024/0.015 for draft positions 5–7, which reads as an obvious "cut `num_speculative_tokens` 7 → 3, it is buying nothing." Across **180 samples** of the same counter over the container's life the real distribution is **median acceptance length 3.12 of 7 (range 1.83–6.75)** and **median draft acceptance 30.4% (range 11.9–82.1%)** — the crash window was near the *minimum*, not the norm, and cutting to 3 would cap the workloads that were accepting nearly the full 7-wide draft. **The n=1 window pointed the opposite way from the n=180 distribution.** Same session that wrote "a positive control is only worth what it can distinguish"; the lesson generalises to log lines. - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` diff --git a/stacks/mog-sec/compose.yaml b/stacks/mog-sec/compose.yaml index 608fedb..57f57d2 100644 --- a/stacks/mog-sec/compose.yaml +++ b/stacks/mog-sec/compose.yaml @@ -99,7 +99,26 @@ services: - ${MOG_QUANT:-compressed-tensors} - --gpu-memory-utilization - ${MOG_GPU_MEM_UTIL:-0.44} + # ⚠ KV PINNED IN BYTES, added 2026-09-10 after the crash pair above. The + # utilization ratio does NOT bound actual usage -- it sizes the KV calculation, + # but peak activation is measured at profiling time and real long-context work + # exceeds the profile. vLLM's own budget line proved this seat was running 0.9 GiB + # OVER its 47.48 GiB reservation at util 0.50 (26.44 consumed + 3.53 peak activation + # + 0.89 CUDAGraph + 17.52 KV = 48.38), and `gen` was over by 0.33 on the same card. + # That overage came out of the shared card's slack, which is what kept OOMing. + # + # 17,697,765,376 B = 16.48 GiB is vLLM's OWN recommended figure from that line + # ("Replace gpu_memory_utilization config with --kv-cache-memory=17697765376 to fit + # into requested memory"), not a value anyone here invented. Same discipline as + # stacks/erp-seat: an explicit figure is reproducible, a ratio silently yields a + # different cache depending on what else is resident at start time. + - --kv-cache-memory + - ${MOG_KV_CACHE_MEMORY:-17697765376} - --max-model-len + # ⚠ 320k, NOT 384k, and the two settings are coupled -- 16.48 GiB of KV yields about + # 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM + # refuses to START rather than crashing later. Pinning the KV and keeping 384k was + # never an available combination. 327,680 leaves ~1.17x. - ${MOG_MAX_MODEL_LEN:-262144} - --max-num-seqs - ${MOG_MAX_NUM_SEQS:-16}