Operator: "loading up the context killed sec again." That reproducer is what
finally made the failure legible, and it showed the previous four fixes had all
been aimed at the wrong quantity.
What the KV pool can hold and what the card can process at depth are different
numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and
dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were
governed by the transient needed to process a prefill chunk against a quarter
million tokens of resident KV, which scales with depth and not with pool size.
Each change helped and none fixed it.
Bisected against the real reproducer, with a non-repeating prompt because prefix
caching would let a repeated one hash to cached blocks and never prefill deep:
113,247 prompt tokens SURVIVED (27 s)
200,088 prompt tokens SURVIVED (174 s)
~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted
The sustainable ceiling therefore sits between 200k and 285k with gen idle, and
gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin
under the proven-good depth rather than sitting at the measured edge.
The ceiling's purpose is the refusal. Verified after the change: a 149,073-token
request serves in 41 s, and requests at both 200k and the ~285k depth that killed
the engine now return a clean 400 naming the limit in under a second with the seat
untouched. A seat that refuses what it cannot serve is strictly better than one
that dies trying.
Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was
never actually deliverable on a shared card; it had simply not been exercised at
depth until today.
The probe is committed rather than described, so the ceiling can be re-measured
when the card's tenancy changes.
vh
b8435ceb6f