vh
b8435ceb6f
Operator: "loading up the context killed sec again." That reproducer is what finally made the failure legible, and it showed the previous four fixes had all been aimed at the wrong quantity. What the KV pool can hold and what the card can process at depth are different numbers. Cutting context 420k to 384k to 320k, pinning the KV in bytes, and dropping the prefill chunk from 16384 to 4096 all sized the pool. The crashes were governed by the transient needed to process a prefill chunk against a quarter million tokens of resident KV, which scales with depth and not with pool size. Each change helped and none fixed it. Bisected against the real reproducer, with a non-repeating prompt because prefix caching would let a repeated one hash to cached blocks and never prefill deep: 113,247 prompt tokens SURVIVED (27 s) 200,088 prompt tokens SURVIVED (174 s) ~285,000 prompt tokens ENGINE DIED, HTTP 500, container restarted The sustainable ceiling therefore sits between 200k and 285k with gen idle, and gen shares the card with its load uncontrolled, so 163,840 takes about 20% margin under the proven-good depth rather than sitting at the measured edge. The ceiling's purpose is the refusal. Verified after the change: a 149,073-token request serves in 41 s, and requests at both 200k and the ~285k depth that killed the engine now return a clean 400 naming the limit in under a second with the seat untouched. A seat that refuses what it cannot serve is strictly better than one that dies trying. Concurrency went 1.03x to 2.09x. The compose header's "served at native 262K" was never actually deliverable on a shared card; it had simply not been exercised at depth until today. The probe is committed rather than described, so the ceiling can be re-measured when the card's tenancy changes.
80 lines
3.6 KiB
Python
80 lines
3.6 KiB
Python
"""Reproduce the operator's kill: fill the context and see whether the seat survives.
|
|
|
|
"Loading up the context killed sec again" is a reproducer, and a config change that
|
|
has not been run against the reproducer is a hope rather than a fix. The crash dumps
|
|
put the failures at num_computed_tokens 151,728 and then 266,832, so the probe walks
|
|
UP through those depths and reports which one, if any, takes the seat down.
|
|
|
|
⚠ The text must be NON-REPEATING. Prefix caching is on, so a prompt built by repeating
|
|
a paragraph would hash to cached blocks after the first occurrence and never actually
|
|
prefill deep -- the probe would pass while proving nothing. Every word here comes from
|
|
a seeded RNG over a large vocabulary, so no block repeats and every token is real work.
|
|
|
|
Reports the seat's restart count before and after, because the failure mode is the
|
|
ENGINE dying: a request can return a 500 while the seat stays up, and it can also
|
|
succeed while the seat is already restarting from someone else's request. The restart
|
|
count is what distinguishes them.
|
|
"""
|
|
import json
|
|
import random
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
import urllib.error
|
|
import urllib.request
|
|
|
|
SEAT = "http://10.250.50.54:8019/v1/chat/completions"
|
|
HOST = "infra-ops@10.250.50.54"
|
|
WORDS = [f"{a}{b}" for a in
|
|
"ash birch cedar dale elm fern gale hollow iron juniper kestrel larch marsh "
|
|
"north oak pike quarry rowan slate thorn upland vale willow yarrow".split()
|
|
for b in ("", "wood", "field", "stone", "water", "ridge", "moor", "gate",
|
|
"hill", "brook", "fell", "reach")]
|
|
|
|
|
|
def restarts():
|
|
out = subprocess.run(
|
|
["ssh", "-o", "ConnectTimeout=10", HOST,
|
|
"docker inspect vllm-mog-sec --format '{{.RestartCount}}'"],
|
|
capture_output=True, text=True, timeout=40)
|
|
return out.stdout.strip() or "?"
|
|
|
|
|
|
def build(n_words, seed):
|
|
r = random.Random(seed)
|
|
return " ".join(r.choice(WORDS) for _ in range(n_words))
|
|
|
|
|
|
# ~1.35 Qwen tokens per word for this vocabulary; depths chosen to bracket both crashes.
|
|
for label, n_words in [("~60k tok", 44_000), ("~150k tok (crash 1 depth)", 111_000),
|
|
("~270k tok (crash 2 depth)", 200_000)]:
|
|
before = restarts()
|
|
body = {"model": "mog-sec-27b",
|
|
"messages": [{"role": "user", "content":
|
|
"Here is a word list. Reply with only the last word of it.\n\n"
|
|
+ build(n_words, hash(label) & 0xffff)}],
|
|
"max_tokens": 16, "temperature": 0,
|
|
"chat_template_kwargs": {"enable_thinking": False}}
|
|
payload = json.dumps(body).encode()
|
|
print(f"\n== {label} ({n_words:,} words, {len(payload)/1e6:.1f} MB) "
|
|
f"restarts before={before}", flush=True)
|
|
t0 = time.time()
|
|
try:
|
|
req = urllib.request.Request(SEAT, data=payload,
|
|
headers={"Content-Type": "application/json"})
|
|
d = json.load(urllib.request.urlopen(req, timeout=900))
|
|
pt = d["usage"]["prompt_tokens"]
|
|
print(f" OK {pt:,} prompt tokens in {time.time()-t0:.0f}s "
|
|
f"answer={d['choices'][0]['message'].get('content')!r}")
|
|
except urllib.error.HTTPError as e:
|
|
print(f" HTTP {e.code} after {time.time()-t0:.0f}s: {e.read().decode()[:180]}")
|
|
except Exception as e:
|
|
print(f" {type(e).__name__} after {time.time()-t0:.0f}s: {str(e)[:180]}")
|
|
time.sleep(5)
|
|
after = restarts()
|
|
verdict = "SEAT SURVIVED" if after == before else f"*** SEAT DIED (restarts {before} -> {after})"
|
|
print(f" {verdict}")
|
|
if after != before:
|
|
sys.exit(1)
|
|
print("\nAll depths completed with no engine restart.")
|