Set mog-sec's context to a measured ceiling so it refuses instead of dying

Operator: "loading up the context killed sec again." That reproducer is what
finally made the failure legible, and it showed the previous four fixes had all
been aimed at the wrong quantity.

What the KV pool can hold and what the card can process at depth are different
numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and
dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were
governed by the transient needed to process a prefill chunk against a quarter
million tokens of resident KV, which scales with depth and not with pool size.
Each change helped and none fixed it.

Bisected against the real reproducer, with a non-repeating prompt because prefix
caching would let a repeated one hash to cached blocks and never prefill deep:

  113,247 prompt tokens  SURVIVED (27 s)
  200,088 prompt tokens  SURVIVED (174 s)
  ~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted

The sustainable ceiling therefore sits between 200k and 285k with gen idle, and
gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin
under the proven-good depth rather than sitting at the measured edge.

The ceiling's purpose is the refusal. Verified after the change: a 149,073-token
request serves in 41 s, and requests at both 200k and the ~285k depth that killed
the engine now return a clean 400 naming the limit in under a second with the seat
untouched. A seat that refuses what it cannot serve is strictly better than one
that dies trying.

Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was
never actually deliverable on a shared card; it had simply not been exercised at
depth until today.

The probe is committed rather than described, so the ceiling can be re-measured
when the card's tenancy changes.
This commit is contained in:
vh
2026-09-10 16:04:16 -07:00
parent 935b35ac2e
commit b8435ceb6f
3 changed files with 116 additions and 5 deletions
+36 -4
View File
@@ -115,16 +115,48 @@ services:
- --kv-cache-memory
- ${MOG_KV_CACHE_MEMORY:-17697765376}
- --max-model-len
# ⚠ 320k, NOT 384k, and the two settings are coupled -- 16.48 GiB of KV yields about
# 383,730 tokens, so a 393,216 max_model_len falls under the 1.0x floor and vLLM
# refuses to START rather than crashing later. Pinning the KV and keeping 384k was
# never an available combination. 327,680 leaves ~1.17x.
# ⚠⚠ 160k, AND THE NUMBER IS MEASURED, NOT DERIVED FROM THE KV POOL. That distinction
# is the whole lesson of 2026-09-10: what the KV pool can HOLD and what the card can
# PROCESS at depth are different numbers, and every earlier cut (420k -> 384k -> 320k)
# sized the first while the crashes were governed by the second. Sizing to the pool
# never worked because the pool was never the constraint.
#
# Bisected against the real reproducer -- a non-repeating prompt, because prefix
# caching would let a repeated one hash to cached blocks and never prefill deep:
# 113,247 prompt tokens -> SURVIVED (27 s)
# 200,088 prompt tokens -> SURVIVED (174 s)
# ~285,000 prompt tokens -> ENGINE DIED, HTTP 500, container restarted
# So the sustainable ceiling sits between 200k and 285k with `gen` idle. 163,840
# leaves ~20% margin under the proven-good depth, which is deliberate: `gen` shares
# this card and its load is an UNCONTROLLED variable, so a value set just under the
# measured edge would fail whenever the neighbour happened to be busy.
#
# ⚠ THE POINT OF THE CEILING IS THE REFUSAL. Below it the seat serves; above it vLLM
# returns a clean 400 naming the limit instead of the engine dying and taking every
# in-flight request with it. A seat that refuses what it cannot serve is strictly
# better than one that dies trying, and that is what this number buys.
#
# ⚠ The header's "served at native 262K" was therefore never actually deliverable on
# a shared card. It had simply not been exercised at depth until today.
- ${MOG_MAX_MODEL_LEN:-262144}
- --max-num-seqs
- ${MOG_MAX_NUM_SEQS:-16}
- --max-num-batched-tokens
# ⚠ Raising this costs peak-activation VRAM straight out of the KV pool
# (measured 2026-08-22: 16384 -> 32768 cost ~3 GiB of KV for no benefit).
#
# ⚠⚠ LOWERED 16384 -> 4096 on 2026-09-10, and this is the lever that actually
# matches the failure. The OOMs were DEPTH-DEPENDENT -- operator: "loading up the
# context killed sec again" -- and the dumps confirm it: the crashes hit at
# num_computed_tokens 151,728 and then 266,832, each scheduling a 15,696-token
# chunk. A chunked-prefill step of 16,384 tokens attending over a quarter-million
# tokens of KV is the transient that blows the budget, and it scales with DEPTH,
# not with the KV pool size. Cutting context 420k -> 384k -> 320k shrank the pool
# and never touched that spike, which is why three cuts did not stop the crashes.
# Reclaim is roughly the inverse of the 2026-08-22 measurement (~3 GiB per 16k of
# batched tokens), so 16384 -> 4096 should return ~2.3 GiB of peak activation --
# more than the 0.9 GiB the seat was over by. Costs prefill throughput on long
# prompts (4x the chunks) and costs NO context.
- ${MOG_MAX_NUM_BATCHED_TOKENS:-16384}
- --trust-remote-code
- --dtype