diff --git a/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md b/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md index 9b149f2..c87dc52 100644 --- a/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md +++ b/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md @@ -92,7 +92,11 @@ both ways — as-shipped raises, with the declaration returns `Qwen4ExpPLEFp8Emb | on card | ~75 GiB; PLE 47.7 GiB pinned host RAM | ⚠ **Do NOT read 167.5 as a win over dealignai.** dealignai's ~121 tok/s in this file came -from a different harness/prompt; cross-harness comparison is invalid. What IS established is +from a different harness/prompt; cross-harness comparison is invalid. ⚠ The 167.5 may itself +have been contended — the operator was using the seat around that window — so treat it as a +LOWER BOUND, not a clean solo figure. Operator's own session reported **140 tok/s average** +in real use while the 258K depth probe was running against the same card: an independent, +contended floor that agrees with the picture. What IS established is that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did **not** cost decode speed, which was the standing risk of giving up FP4 tensor-core compute. @@ -100,7 +104,16 @@ cost decode speed, which was the standing risk of giving up FP4 tensor-core comp - Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context degradation axis). Needs a controlled harness + noise floor. -- Deep-prefill probe at 262K against THIS checkpoint. Startup is not a depth test. +- ✅ **Deep-prefill probe DONE 2026-09-14: clean to 258,517 tokens.** Non-repeating prompts, + 6 depths 32K->258K, all 200 OK, and **zero allocator OOM/CUBLAS/illegal-memory in the engine + log** — the detector that caught dealignai's 155K near-miss. Run under real operator load, so + a stricter test than solo. Positive control passed (a ~265K prompt got a clean 400 naming the + limit). #54919 did not reproduce: 258K prefilled in 28.9 s, ~8,900 tok/s, near-linear. + ⚠ The probe's MEMORY column was blind and must not be reused: `--kv-cache-memory` pins the + pool ("skipped memory profiling"), so GPU use is flat vs depth, and before/after `nvidia-smi` + bracketing cannot see a transient mid-prefill spike. Identical readings across an 8x depth + range were the tell. Peak-activation headroom remains UNMEASURED. + ⭐ Calibration for re-runs: random hex words tokenize at **7.9 tok/word**. - ⚠⚠ **NO LOCAL ROLLBACK.** dealignai weights DELETED 2026-09-14 on operator instruction (125 GiB reclaimed). `.env.bak-preorca-20260914-023408` still names the old paths but they no longer exist — it is a record, not a revert. Reverting = 126 GiB re-download. diff --git a/stacks/flash-next-seat/README.md b/stacks/flash-next-seat/README.md index 3e35127..92a4c1e 100644 --- a/stacks/flash-next-seat/README.md +++ b/stacks/flash-next-seat/README.md @@ -164,17 +164,38 @@ Open issues worth knowing about on SM120, none of them blocking: | **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing | | **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here | -## Raising context +## Context depth — PROBED 2026-09-14, clean to 258,517 tokens -128K is a starting value, not a measured one. Before raising it, bisect with a -**non-repeating** prompt — a repeated one hashes to cached blocks and never -prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the -hard way: three successive context cuts all sized the *KV pool* while the crashes -were governed by *processing depth*, which is a different number. +262,144 is the configured ceiling and it has now been bisected with a **non-repeating** +prompt (unique random hex per probe, so prefix caching cannot short-circuit the prefill — +a repeated prompt hashes to cached blocks and never prefills deep). -The point of a ceiling is the refusal. Below it the seat serves; above it vLLM -returns a clean 400 naming the limit, instead of the engine dying and taking every -in-flight request with it. +| prompt tokens | 31,978 | 64,154 | 128,191 | 196,172 | 240,290 | **258,517** | +|---|---|---|---|---|---|---| +| result | ok | ok | ok | ok | ok | **ok** | + +**The load-bearing evidence is the engine's own allocator log: zero OOM / CUBLAS / +illegal-memory / traceback entries across the whole run** — the same detector that caught the +dealignai near-miss (`OOM on device 0 ... 466 MiB wanted, 403 MiB free`) at 155K on the +previous checkpoint. It fired then; it is silent here. The run also happened **under real +concurrent operator load**, which makes it a stricter test than a solo probe, not a weaker one. + +**Positive control passed.** A mis-sized first attempt built a ~265K-token prompt and got a +clean `400` naming the limit rather than killing the engine — so the probe could detect the +failure it was looking for. (Calibration for anyone re-running it: random hex words tokenize +at **7.9 tokens/word** on this tokenizer.) + +**#54919 did not reproduce.** That issue reports long prefill starving decode for 3-7 minutes; +258K prefilled in **28.9 s** (~8,900 tok/s), scaling near-linearly from 32K. + +⚠⚠ **DO NOT reuse the memory-headroom half of that probe — the gauge was blind.** It sampled +`nvidia-smi` before and after each request and reported an identical 95,460 MiB / 2,427 MiB +free on *every* row. Two reasons: `--kv-cache-memory` pins the pool and the engine log says it +**"skipped memory profiling"**, so GPU usage is constant regardless of depth; and the risk is a +*transient* activation spike **during** prefill, which before/after bracketing structurally +cannot see. Identical readings across a 8x range of depths are the tell. Real peak-activation +headroom needs in-process sampling during the prefill. The pass/fail result stands on the +allocator log, not on that column. ## Not done yet