test(flash-next-seat): depth-probe orca to 258K clean; record a blind instrument
Bisected context depth on the orcarouter checkpoint with non-repeating prompts (unique random hex per probe, so prefix caching cannot short-circuit the prefill). Six depths from 31,978 to 258,517 tokens, all served. The load-bearing evidence is the engine allocator log: zero OOM, CUBLAS, or illegal-memory entries across the run. That is the same detector that caught the dealignai near-miss at 155K on the previous checkpoint, where it did fire. The probe also ran under real concurrent operator load, making it a stricter test than a solo run rather than a weaker one. Positive control passed: a mis-sized first attempt produced a ~265K-token prompt and got a clean 400 naming the limit instead of killing the engine, so the probe could detect the failure mode it was looking for. Calibration for re-runs: random hex words tokenize at 7.9 tokens/word here. vLLM #54919 (long prefill starving decode for 3-7 minutes) did not reproduce: 258K prefilled in 28.9 s, roughly 8,900 tok/s, scaling near-linearly from 32K. Records that the probe's memory-headroom half was BLIND and must not be reused. It reported an identical 95,460 MiB used / 2,427 MiB free on every row across an 8x range of depths, which is the tell. Two causes: --kv-cache-memory pins the pool and the engine logs "skipped memory profiling", so GPU usage is flat with respect to depth; and the actual risk is a transient activation spike during prefill, which before/after nvidia-smi bracketing structurally cannot observe. Peak-activation headroom therefore remains unmeasured; the pass/fail result rests on the allocator log alone. Also qualifies the earlier 167.5 tok/s decode figure as a possibly-contended lower bound, and records the operator's independent 140 tok/s average measured in real use while this probe was loading the same card.
This commit is contained in:
@@ -92,7 +92,11 @@ both ways — as-shipped raises, with the declaration returns `Qwen4ExpPLEFp8Emb
|
|||||||
| on card | ~75 GiB; PLE 47.7 GiB pinned host RAM |
|
| on card | ~75 GiB; PLE 47.7 GiB pinned host RAM |
|
||||||
|
|
||||||
⚠ **Do NOT read 167.5 as a win over dealignai.** dealignai's ~121 tok/s in this file came
|
⚠ **Do NOT read 167.5 as a win over dealignai.** dealignai's ~121 tok/s in this file came
|
||||||
from a different harness/prompt; cross-harness comparison is invalid. What IS established is
|
from a different harness/prompt; cross-harness comparison is invalid. ⚠ The 167.5 may itself
|
||||||
|
have been contended — the operator was using the seat around that window — so treat it as a
|
||||||
|
LOWER BOUND, not a clean solo figure. Operator's own session reported **140 tok/s average**
|
||||||
|
in real use while the 258K depth probe was running against the same card: an independent,
|
||||||
|
contended floor that agrees with the picture. What IS established is
|
||||||
that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did **not**
|
that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did **not**
|
||||||
cost decode speed, which was the standing risk of giving up FP4 tensor-core compute.
|
cost decode speed, which was the standing risk of giving up FP4 tensor-core compute.
|
||||||
|
|
||||||
@@ -100,7 +104,16 @@ cost decode speed, which was the standing risk of giving up FP4 tensor-core comp
|
|||||||
|
|
||||||
- Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context
|
- Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context
|
||||||
degradation axis). Needs a controlled harness + noise floor.
|
degradation axis). Needs a controlled harness + noise floor.
|
||||||
- Deep-prefill probe at 262K against THIS checkpoint. Startup is not a depth test.
|
- ✅ **Deep-prefill probe DONE 2026-09-14: clean to 258,517 tokens.** Non-repeating prompts,
|
||||||
|
6 depths 32K->258K, all 200 OK, and **zero allocator OOM/CUBLAS/illegal-memory in the engine
|
||||||
|
log** — the detector that caught dealignai's 155K near-miss. Run under real operator load, so
|
||||||
|
a stricter test than solo. Positive control passed (a ~265K prompt got a clean 400 naming the
|
||||||
|
limit). #54919 did not reproduce: 258K prefilled in 28.9 s, ~8,900 tok/s, near-linear.
|
||||||
|
⚠ The probe's MEMORY column was blind and must not be reused: `--kv-cache-memory` pins the
|
||||||
|
pool ("skipped memory profiling"), so GPU use is flat vs depth, and before/after `nvidia-smi`
|
||||||
|
bracketing cannot see a transient mid-prefill spike. Identical readings across an 8x depth
|
||||||
|
range were the tell. Peak-activation headroom remains UNMEASURED.
|
||||||
|
⭐ Calibration for re-runs: random hex words tokenize at **7.9 tok/word**.
|
||||||
- ⚠⚠ **NO LOCAL ROLLBACK.** dealignai weights DELETED 2026-09-14 on operator instruction
|
- ⚠⚠ **NO LOCAL ROLLBACK.** dealignai weights DELETED 2026-09-14 on operator instruction
|
||||||
(125 GiB reclaimed). `.env.bak-preorca-20260914-023408` still names the old paths but they
|
(125 GiB reclaimed). `.env.bak-preorca-20260914-023408` still names the old paths but they
|
||||||
no longer exist — it is a record, not a revert. Reverting = 126 GiB re-download.
|
no longer exist — it is a record, not a revert. Reverting = 126 GiB re-download.
|
||||||
|
|||||||
@@ -164,17 +164,38 @@ Open issues worth knowing about on SM120, none of them blocking:
|
|||||||
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
||||||
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
||||||
|
|
||||||
## Raising context
|
## Context depth — PROBED 2026-09-14, clean to 258,517 tokens
|
||||||
|
|
||||||
128K is a starting value, not a measured one. Before raising it, bisect with a
|
262,144 is the configured ceiling and it has now been bisected with a **non-repeating**
|
||||||
**non-repeating** prompt — a repeated one hashes to cached blocks and never
|
prompt (unique random hex per probe, so prefix caching cannot short-circuit the prefill —
|
||||||
prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the
|
a repeated prompt hashes to cached blocks and never prefills deep).
|
||||||
hard way: three successive context cuts all sized the *KV pool* while the crashes
|
|
||||||
were governed by *processing depth*, which is a different number.
|
|
||||||
|
|
||||||
The point of a ceiling is the refusal. Below it the seat serves; above it vLLM
|
| prompt tokens | 31,978 | 64,154 | 128,191 | 196,172 | 240,290 | **258,517** |
|
||||||
returns a clean 400 naming the limit, instead of the engine dying and taking every
|
|---|---|---|---|---|---|---|
|
||||||
in-flight request with it.
|
| result | ok | ok | ok | ok | ok | **ok** |
|
||||||
|
|
||||||
|
**The load-bearing evidence is the engine's own allocator log: zero OOM / CUBLAS /
|
||||||
|
illegal-memory / traceback entries across the whole run** — the same detector that caught the
|
||||||
|
dealignai near-miss (`OOM on device 0 ... 466 MiB wanted, 403 MiB free`) at 155K on the
|
||||||
|
previous checkpoint. It fired then; it is silent here. The run also happened **under real
|
||||||
|
concurrent operator load**, which makes it a stricter test than a solo probe, not a weaker one.
|
||||||
|
|
||||||
|
**Positive control passed.** A mis-sized first attempt built a ~265K-token prompt and got a
|
||||||
|
clean `400` naming the limit rather than killing the engine — so the probe could detect the
|
||||||
|
failure it was looking for. (Calibration for anyone re-running it: random hex words tokenize
|
||||||
|
at **7.9 tokens/word** on this tokenizer.)
|
||||||
|
|
||||||
|
**#54919 did not reproduce.** That issue reports long prefill starving decode for 3-7 minutes;
|
||||||
|
258K prefilled in **28.9 s** (~8,900 tok/s), scaling near-linearly from 32K.
|
||||||
|
|
||||||
|
⚠⚠ **DO NOT reuse the memory-headroom half of that probe — the gauge was blind.** It sampled
|
||||||
|
`nvidia-smi` before and after each request and reported an identical 95,460 MiB / 2,427 MiB
|
||||||
|
free on *every* row. Two reasons: `--kv-cache-memory` pins the pool and the engine log says it
|
||||||
|
**"skipped memory profiling"**, so GPU usage is constant regardless of depth; and the risk is a
|
||||||
|
*transient* activation spike **during** prefill, which before/after bracketing structurally
|
||||||
|
cannot see. Identical readings across a 8x range of depths are the tell. Real peak-activation
|
||||||
|
headroom needs in-process sampling during the prefill. The pass/fail result stands on the
|
||||||
|
allocator log, not on that column.
|
||||||
|
|
||||||
## Not done yet
|
## Not done yet
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user