test(flash-next-seat): depth-probe orca to 258K clean; record a blind instrument

Bisected context depth on the orcarouter checkpoint with non-repeating prompts
(unique random hex per probe, so prefix caching cannot short-circuit the
prefill). Six depths from 31,978 to 258,517 tokens, all served.

The load-bearing evidence is the engine allocator log: zero OOM, CUBLAS, or
illegal-memory entries across the run. That is the same detector that caught
the dealignai near-miss at 155K on the previous checkpoint, where it did fire.
The probe also ran under real concurrent operator load, making it a stricter
test than a solo run rather than a weaker one.

Positive control passed: a mis-sized first attempt produced a ~265K-token
prompt and got a clean 400 naming the limit instead of killing the engine, so
the probe could detect the failure mode it was looking for. Calibration for
re-runs: random hex words tokenize at 7.9 tokens/word here.

vLLM #54919 (long prefill starving decode for 3-7 minutes) did not reproduce:
258K prefilled in 28.9 s, roughly 8,900 tok/s, scaling near-linearly from 32K.

Records that the probe's memory-headroom half was BLIND and must not be reused.
It reported an identical 95,460 MiB used / 2,427 MiB free on every row across an
8x range of depths, which is the tell. Two causes: --kv-cache-memory pins the
pool and the engine logs "skipped memory profiling", so GPU usage is flat with
respect to depth; and the actual risk is a transient activation spike during
prefill, which before/after nvidia-smi bracketing structurally cannot observe.
Peak-activation headroom therefore remains unmeasured; the pass/fail result
rests on the allocator log alone.

Also qualifies the earlier 167.5 tok/s decode figure as a possibly-contended
lower bound, and records the operator's independent 140 tok/s average measured
in real use while this probe was loading the same card.
This commit is contained in:
vh
2026-09-14 03:02:55 -07:00
parent d6cc22f6a0
commit 4954cf6756
2 changed files with 45 additions and 11 deletions
@@ -92,7 +92,11 @@ both ways — as-shipped raises, with the declaration returns `Qwen4ExpPLEFp8Emb
| on card | ~75 GiB; PLE 47.7 GiB pinned host RAM |
⚠ **Do NOT read 167.5 as a win over dealignai.** dealignai's ~121 tok/s in this file came
from a different harness/prompt; cross-harness comparison is invalid. What IS established is
from a different harness/prompt; cross-harness comparison is invalid. ⚠ The 167.5 may itself
have been contended — the operator was using the seat around that window — so treat it as a
LOWER BOUND, not a clean solo figure. Operator's own session reported **140 tok/s average**
in real use while the 258K depth probe was running against the same card: an independent,
contended floor that agrees with the picture. What IS established is
that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did **not**
cost decode speed, which was the standing risk of giving up FP4 tensor-core compute.
@@ -100,7 +104,16 @@ cost decode speed, which was the standing risk of giving up FP4 tensor-core comp
- Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context
degradation axis). Needs a controlled harness + noise floor.
- Deep-prefill probe at 262K against THIS checkpoint. Startup is not a depth test.
- ✅ **Deep-prefill probe DONE 2026-09-14: clean to 258,517 tokens.** Non-repeating prompts,
6 depths 32K->258K, all 200 OK, and **zero allocator OOM/CUBLAS/illegal-memory in the engine
log** — the detector that caught dealignai's 155K near-miss. Run under real operator load, so
a stricter test than solo. Positive control passed (a ~265K prompt got a clean 400 naming the
limit). #54919 did not reproduce: 258K prefilled in 28.9 s, ~8,900 tok/s, near-linear.
⚠ The probe's MEMORY column was blind and must not be reused: `--kv-cache-memory` pins the
pool ("skipped memory profiling"), so GPU use is flat vs depth, and before/after `nvidia-smi`
bracketing cannot see a transient mid-prefill spike. Identical readings across an 8x depth
range were the tell. Peak-activation headroom remains UNMEASURED.
⭐ Calibration for re-runs: random hex words tokenize at **7.9 tok/word**.
- ⚠⚠ **NO LOCAL ROLLBACK.** dealignai weights DELETED 2026-09-14 on operator instruction
(125 GiB reclaimed). `.env.bak-preorca-20260914-023408` still names the old paths but they
no longer exist — it is a record, not a revert. Reverting = 126 GiB re-download.