Set mog-sec's context to a measured ceiling so it refuses instead of dying

Operator: "loading up the context killed sec again." That reproducer is what
finally made the failure legible, and it showed the previous four fixes had all
been aimed at the wrong quantity.

What the KV pool can hold and what the card can process at depth are different
numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and
dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were
governed by the transient needed to process a prefill chunk against a quarter
million tokens of resident KV, which scales with depth and not with pool size.
Each change helped and none fixed it.

Bisected against the real reproducer, with a non-repeating prompt because prefix
caching would let a repeated one hash to cached blocks and never prefill deep:

  113,247 prompt tokens  SURVIVED (27 s)
  200,088 prompt tokens  SURVIVED (174 s)
  ~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted

The sustainable ceiling therefore sits between 200k and 285k with gen idle, and
gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin
under the proven-good depth rather than sitting at the measured edge.

The ceiling's purpose is the refusal. Verified after the change: a 149,073-token
request serves in 41 s, and requests at both 200k and the ~285k depth that killed
the engine now return a clean 400 naming the limit in under a second with the seat
untouched. A seat that refuses what it cannot serve is strictly better than one
that dies trying.

Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was
never actually deliverable on a shared card; it had simply not been exercised at
depth until today.

The probe is committed rather than described, so the ceiling can be re-measured
when the card's tenancy changes.

persistent-memory.md

@@ -210,7 +210,7 @@ _As of 2026-09-10 10:25 PT._
 
 - `[2026-09-10]` **BabyBronte H02 adapter: the VOICE transferred, the SENSE did not — operator's read, "it's all nonsense, but it sounds like Brontë's nonsense."** Eyeball A/B (NOT the adjudication; nothing here feeds the frozen rule), 9 arbitrary prompts on a deliberate difficulty gradient × 2 arms × 2 seeds, booth at `http://10.100.10.50:8090/b/babybronte-voice/`. Measured across the 18 pairs: **curly quotes 1/18 base → 18/18 tuned**, **math/worksheet collapse 3/18 base → 0/18 tuned**. Given *"The self-checkout machine refused her coupon"* the base 0.6B produced a **quadratic-formula worksheet**; the tuned arm wrote a clerk refusing a customer in Victorian retrospective first person. This is the expected and informative result for the smallest rung — **voice is separable from coherence at 0.6B**, which is the premise the whole lightweight-adapter regime rests on, and the 1.7B/4B rungs are where sense should return. The 1-epoch loss was still descending at step 169 (undertrained, not overfit), so the incoherence is carrier capacity, not training. ⚠ **Corpus-prep defect found: the tuned output is hard-wrapped at ~70 chars** (median mid-length-line ratio 0.85 vs base 0.00) — the Gutenberg source kept its original line breaks and the adapter learned the typography along with the voice. Unwrap to flowing paragraphs before any real use or the next rung learns it too.
 
-- `[2026-09-10]` **GPU0 SETTLED after THREE crashes and two failed attempts: mog-sec now runs `MOG_GPU_MEM_UTIL=0.50` + **`MOG_KV_CACHE_MEMORY=17697765376` (KV PINNED IN BYTES)** + `MOG_MAX_MODEL_LEN=327680` (was 0.52 / ratio-inferred KV / 420000).** ⚠ **I got the severity call wrong first**: off a `RestartCount=1` I said "rare, not chronic, do nothing"; the operator pushed back and it crashed twice more (20:20:30Z, 20:30:02Z at the old settings, then 21:46:35Z after the first fix). ⚠ **Dropping utilization alone is NOT the fix and fails in a worse way** — at 0.50 with 420k the seat crash-loops at STARTUP, because one 420k sequence needs 17.88 GiB of KV and the 0.50 pool is 17.4–17.5. ⚠⚠ **THE REAL FINDING: `--gpu-memory-utilization` does NOT bound actual usage.** It sizes the KV calculation, but peak activation is measured at profiling time and real long-context work exceeds the profile. vLLM's own budget line proved mog-sec was running **0.9 GiB OVER** its 47.48 GiB reservation (26.44 consumed + 3.53 peak act + 0.89 CUDAGraph + 17.52 KV = 48.38), and **`gen` was over by 0.33 on the same card** — that overage came out of the shared slack and is what kept OOMing. Fix is the one vLLM printed itself: pin KV in bytes at its recommended 16.48 GiB, same discipline `erp-seat` already uses. **The KV pin and the context length are COUPLED** — 16.48 GiB yields ~383,730 tokens, so 393,216 falls under the 1.0x floor and pinning KV while keeping 384k was never available. Verified after: `reserved 16.48 GiB ... as specified by kv_cache_memory_bytes config and skipped memory profiling`, KV 375,901 tokens, **concurrency 1.15x** (was 1.03x), GPU0 90,561 MiB (~750 MiB reclaimed), both aliases 200 through the gateway. Backups `.env.bak-util052-20260910` and `.env.bak-384k-20260910`; full reasoning in `stacks/mog-sec/compose.yaml`'s header. Cost: 320k context instead of 420k, on a seat whose crashes happened at ~151k.
+- `[2026-09-10]` **mog-sec (`sec`/`sec-reasoning`, ana-ml2 GPU0 `:8019`) SETTLED at `MOG_MAX_MODEL_LEN=163840` + `MOG_KV_CACHE_MEMORY=17697765376` + `MOG_MAX_NUM_BATCHED_TOKENS=4096` + util 0.50, after FIVE crashes and four wrong fixes.** ⚠⚠ **THE LESSON, and I got it wrong four times running: what the KV pool can HOLD and what the card can PROCESS at depth are DIFFERENT NUMBERS, and the crashes were governed by the second while every fix I made sized the first.** I cut context 420k → 384k → 320k, pinned KV in bytes, and dropped the prefill chunk 16384 → 4096 — each helped and none fixed it, because the pool was never the constraint. ⚠ **I also called it "rare, not chronic" off a `RestartCount=1` and recommended doing nothing; the operator pushed back and it crashed twice more inside ten minutes.** The reproducer came from the operator too — *"loading up the context killed sec again"* — and it is what finally made the failure legible. **Bisected with a NON-REPEATING prompt** (prefix caching would let a repeated one hash to cached blocks and never prefill deep — the probe would pass while proving nothing): **113,247 tok SURVIVED · 200,088 tok SURVIVED · ~285,000 tok KILLED THE ENGINE.** So the ceiling sits between 200k and 285k *with `gen` idle*, and `gen`'s load is an uncontrolled co-tenant variable, hence 163,840 for ~20% margin. ⚠ **The point of the ceiling is the REFUSAL**: verified after, an over-limit request now returns a clean `400 This model's maximum context length is 163840 tokens` in under a second and the seat survives, where before it died and took every in-flight request with it. **A seat that refuses what it cannot serve beats one that dies trying.** Concurrency 1.03x → **2.09x**; 149,073-token request served in 41 s. ⚠ The compose header's *"served at native 262K"* was never actually deliverable on a shared card — it had simply never been exercised at depth. Probe committed at `services/mog-sec-tuning/deep_ctx_probe.py`; backups `.env.bak-{util052,384k,batched16384}-20260910`.
 - `[2026-09-10]` ⚠ **Near-miss on measurement discipline, worth keeping as a specimen.** The crash window logged `Avg Draft acceptance rate: 17.6%` and per-position rates of 0.049/0.024/0.015 for draft positions 5–7, which reads as an obvious "cut `num_speculative_tokens` 7 → 3, it is buying nothing." Across **180 samples** of the same counter over the container's life the real distribution is **median acceptance length 3.12 of 7 (range 1.83–6.75)** and **median draft acceptance 30.4% (range 11.9–82.1%)** — the crash window was near the *minimum*, not the norm, and cutting to 3 would cap the workloads that were accepting nearly the full 7-wide draft. **The n=1 window pointed the opposite way from the n=180 distribution.** Same session that wrote "a positive control is only worth what it can distinguish"; the lesson generalises to log lines.
 
 - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md`

services/mog-sec-tuning/deep_ctx_probe.py

@@ -0,0 +1,79 @@
+"""Reproduce the operator's kill: fill the context and see whether the seat survives.
+
+"Loading up the context killed sec again" is a reproducer, and a config change that
+has not been run against the reproducer is a hope rather than a fix. The crash dumps
+put the failures at num_computed_tokens 151,728 and then 266,832, so the probe walks
+UP through those depths and reports which one, if any, takes the seat down.
+
+⚠ The text must be NON-REPEATING. Prefix caching is on, so a prompt built by repeating
+a paragraph would hash to cached blocks after the first occurrence and never actually
+prefill deep -- the probe would pass while proving nothing. Every word here comes from
+a seeded RNG over a large vocabulary, so no block repeats and every token is real work.
+
+Reports the seat's restart count before and after, because the failure mode is the
+ENGINE dying: a request can return a 500 while the seat stays up, and it can also
+succeed while the seat is already restarting from someone else's request. The restart
+count is what distinguishes them.
+"""
+import json
+import random
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+
+SEAT = "http://10.250.50.54:8019/v1/chat/completions"
+HOST = "infra-ops@10.250.50.54"
+WORDS = [f"{a}{b}" for a in
+         "ash birch cedar dale elm fern gale hollow iron juniper kestrel larch marsh "
+         "north oak pike quarry rowan slate thorn upland vale willow yarrow".split()
+         for b in ("", "wood", "field", "stone", "water", "ridge", "moor", "gate",
+                   "hill", "brook", "fell", "reach")]
+
+
+def restarts():
+    out = subprocess.run(
+        ["ssh", "-o", "ConnectTimeout=10", HOST,
+         "docker inspect vllm-mog-sec --format '{{.RestartCount}}'"],
+        capture_output=True, text=True, timeout=40)
+    return out.stdout.strip() or "?"
+
+
+def build(n_words, seed):
+    r = random.Random(seed)
+    return " ".join(r.choice(WORDS) for _ in range(n_words))
+
+
+# ~1.35 Qwen tokens per word for this vocabulary; depths chosen to bracket both crashes.
+for label, n_words in [("~60k tok", 44_000), ("~150k tok (crash 1 depth)", 111_000),
+                       ("~270k tok (crash 2 depth)", 200_000)]:
+    before = restarts()
+    body = {"model": "mog-sec-27b",
+            "messages": [{"role": "user", "content":
+                          "Here is a word list. Reply with only the last word of it.\n\n"
+                          + build(n_words, hash(label) & 0xffff)}],
+            "max_tokens": 16, "temperature": 0,
+            "chat_template_kwargs": {"enable_thinking": False}}
+    payload = json.dumps(body).encode()
+    print(f"\n== {label}  ({n_words:,} words, {len(payload)/1e6:.1f} MB)  "
+          f"restarts before={before}", flush=True)
+    t0 = time.time()
+    try:
+        req = urllib.request.Request(SEAT, data=payload,
+                                     headers={"Content-Type": "application/json"})
+        d = json.load(urllib.request.urlopen(req, timeout=900))
+        pt = d["usage"]["prompt_tokens"]
+        print(f"   OK  {pt:,} prompt tokens in {time.time()-t0:.0f}s  "
+              f"answer={d['choices'][0]['message'].get('content')!r}")
+    except urllib.error.HTTPError as e:
+        print(f"   HTTP {e.code} after {time.time()-t0:.0f}s: {e.read().decode()[:180]}")
+    except Exception as e:
+        print(f"   {type(e).__name__} after {time.time()-t0:.0f}s: {str(e)[:180]}")
+    time.sleep(5)
+    after = restarts()
+    verdict = "SEAT SURVIVED" if after == before else f"*** SEAT DIED (restarts {before} -> {after})"
+    print(f"   {verdict}")
+    if after != before:
+        sys.exit(1)
+print("\nAll depths completed with no engine restart.")

stacks/mog-sec/compose.yaml

@@ -115,16 +115,48 @@ services:
       - --kv-cache-memory
       - ${MOG_KV_CACHE_MEMORY:-17697765376}
       - --max-model-len
-      # ⚠ 320k, NOT 384k, and the two settings are coupled -- 16.48 GiB of KV yields about
-      # 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM
-      # refuses to START rather than crashing later. Pinning the KV and keeping 384k was
-      # never an available combination. 327,680 leaves ~1.17x.
+      # ⚠⚠ 160k, AND THE NUMBER IS MEASURED, NOT DERIVED FROM THE KV POOL. That distinction
+      # is the whole lesson of 2026-09-10: what the KV pool can HOLD and what the card can
+      # PROCESS at depth are different numbers, and every earlier cut (420k -> 384k -> 320k)
+      # sized the first while the crashes were governed by the second. Sizing to the pool
+      # never worked because the pool was never the constraint.
+      #
+      # Bisected against the real reproducer -- a non-repeating prompt, because prefix
+      # caching would let a repeated one hash to cached blocks and never prefill deep:
+      #     113,247 prompt tokens  -> SURVIVED (27 s)
+      #     200,088 prompt tokens  -> SURVIVED (174 s)
+      #     ~285,000 prompt tokens -> ENGINE DIED, HTTP 500, container restarted
+      # So the sustainable ceiling sits between 200k and 285k with `gen` idle. 163,840
+      # leaves ~20% margin under the proven-good depth, which is deliberate: `gen` shares
+      # this card and its load is an UNCONTROLLED variable, so a value set just under the
+      # measured edge would fail whenever the neighbour happened to be busy.
+      #
+      # ⚠ THE POINT OF THE CEILING IS THE REFUSAL. Below it the seat serves; above it vLLM
+      # returns a clean 400 naming the limit instead of the engine dying and taking every
+      # in-flight request with it. A seat that refuses what it cannot serve is strictly
+      # better than one that dies trying, and that is what this number buys.
+      #
+      # ⚠ The header's "served at native 262K" was therefore never actually deliverable on
+      # a shared card. It had simply not been exercised at depth until today.
       - ${MOG_MAX_MODEL_LEN:-262144}
       - --max-num-seqs
       - ${MOG_MAX_NUM_SEQS:-16}
       - --max-num-batched-tokens
       # ⚠ Raising this costs peak-activation VRAM straight out of the KV pool
       # (measured 2026-08-22: 16384 -> 32768 cost ~3 GiB of KV for no benefit).
+      #
+      # ⚠⚠ LOWERED 16384 -> 4096 on 2026-09-10, and this is the lever that actually
+      # matches the failure. The OOMs were DEPTH-DEPENDENT -- operator: "loading up the
+      # context killed sec again" -- and the dumps confirm it: the crashes hit at
+      # num_computed_tokens 151,728 and then 266,832, each scheduling a 15,696-token
+      # chunk. A chunked-prefill step of 16,384 tokens attending over a quarter-million
+      # tokens of KV is the transient that blows the budget, and it scales with DEPTH,
+      # not with the KV pool size. Cutting context 420k -> 384k -> 320k shrank the pool
+      # and never touched that spike, which is why three cuts did not stop the crashes.
+      # Reclaim is roughly the inverse of the 2026-08-22 measurement (~3 GiB per 16k of
+      # batched tokens), so 16384 -> 4096 should return ~2.3 GiB of peak activation --
+      # more than the 0.9 GiB the seat was over by. Costs prefill throughput on long
+      # prompts (4x the chunks) and costs NO context.
       - ${MOG_MAX_NUM_BATCHED_TOKENS:-16384}
       - --trust-remote-code
       - --dtype