vh
838132cd6b
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call.
4 lines
2.1 KiB
Markdown
4 lines
2.1 KiB
Markdown
# `[2026-09-10]` mog-sec (`sec`/`sec-reasoning`, ana-ml2 GPU0 `:8019`) SETTLED at `MOG_MAX_MODEL_LEN=163840` + `MOG_KV_CACHE_ME
|
|
|
|
**mog-sec (`sec`/`sec-reasoning`, ana-ml2 GPU0 `:8019`) SETTLED at `MOG_MAX_MODEL_LEN=163840` + `MOG_KV_CACHE_MEMORY=17697765376` + `MOG_MAX_NUM_BATCHED_TOKENS=4096` + util 0.50, after FIVE crashes and four wrong fixes.** ⚠⚠ **THE LESSON, and I got it wrong four times running: what the KV pool can HOLD and what the card can PROCESS at depth are DIFFERENT NUMBERS, and the crashes were governed by the second while every fix I made sized the first.** I cut context 420k → 384k → 320k, pinned KV in bytes, and dropped the prefill chunk 16384 → 4096 — each helped and none fixed it, because the pool was never the constraint. ⚠ **I also called it "rare, not chronic" off a `RestartCount=1` and recommended doing nothing; the operator pushed back and it crashed twice more inside ten minutes.** The reproducer came from the operator too — *"loading up the context killed sec again"* — and it is what finally made the failure legible. **Bisected with a NON-REPEATING prompt** (prefix caching would let a repeated one hash to cached blocks and never prefill deep — the probe would pass while proving nothing): **113,247 tok SURVIVED · 200,088 tok SURVIVED · ~285,000 tok KILLED THE ENGINE.** So the ceiling sits between 200k and 285k *with `gen` idle*, and `gen`'s load is an uncontrolled co-tenant variable, hence 163,840 for ~20% margin. ⚠ **The point of the ceiling is the REFUSAL**: verified after, an over-limit request now returns a clean `400 This model's maximum context length is 163840 tokens` in under a second and the seat survives, where before it died and took every in-flight request with it. **A seat that refuses what it cannot serve beats one that dies trying.** Concurrency 1.03x → **2.09x**; 149,073-token request served in 41 s. ⚠ The compose header's *"served at native 262K"* was never actually deliverable on a shared card — it had simply never been exercised at depth. Probe committed at `services/mog-sec-tuning/deep_ctx_probe.py`; backups `.env.bak-{util052,384k,batched16384}-20260910`.
|