vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: record prefill result (roughly doubled) on the gen-seat mixed requant

Browse Source
This commit is contained in:
vh
2026-08-15 02:34:04 -07:00
parent 4a5c3fcccf
commit 930197a56a
2 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md
+7
View File
@@ -89,6 +89,9 @@ names — 0 overlap, MLP union = layers 0-63 — before any GPU time was spent.
| metric | W4A16 | mixed | delta | | metric | W4A16 | mixed | delta |
|---|---|---|---| |---|---|---|---|
| decode tok/s | 80.12 | **94.53** | **+18.0%** | | decode tok/s | 80.12 | **94.53** | **+18.0%** |
| prefill tok/s (~6.7k prompt) | 3,206 | **6,334** | **+98%** |
| prefill tok/s (~27k prompt) | 2,862 | **5,085** | **+78%** |
| TTFT on a ~27k doc | 9.43 s | **5.31 s** | −44% |
| MTP acceptance | 47.8% | 47.7% | unchanged | | MTP acceptance | 47.8% | 47.7% | unchanged |
| perplexity (6 passages) | 6.941 | 7.059 | +1.7% worse | | perplexity (6 passages) | 6.941 | 7.059 | +1.7% worse |
| abliteration compliance | 4/4 | 4/4 | preserved | | abliteration compliance | 4/4 | 4/4 | preserved |
@@ -107,6 +110,10 @@ routing. Commit `74f596b`.
uninitialised → 0% acceptance). `post_quant.py` now **re-injects it after the uninitialised → 0% acceptance). `post_quant.py` now **re-injects it after the
graft and re-verifies**. That check *fired on this run* — it was not graft and re-verifies**. That check *fired on this run* — it was not
hypothetical. hypothetical.
- **Prefix caching silently fakes prefill numbers too.** The prefill harness originally used a
*seeded* nonce, so run 2 regenerated run 1's prompts verbatim and read **~41k tok/s of
cache-hit** instead of ~5k of real prefill. Same class of error as the decode bench. Use
`SystemRandom`; never seed a cache-busting nonce.
- **vLLM's `prompt_logprobs` are garbage while speculative decoding is on** — - **vLLM's `prompt_logprobs` are garbage while speculative decoding is on** —
~uniform over the vocab (median rank ~10⁵, logprob ≈ log(1/vocab); " Paris" ~uniform over the vocab (median rank ~10⁵, logprob ≈ log(1/vocab); " Paris"
after "The capital of France is" ranked 69698). Perplexity must be measured on after "The capital of France is" ranked 69698). Perplexity must be measured on
persistent-memory.md
+1 -1
View File
@@ -113,7 +113,7 @@ _As of 2026-08-15 — **GEN SEAT REQUANTED (mixed NVFP4+FP8, +18% decode) and th
- **✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`. - **✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`.
- **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s (+18.0%)** at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`. - **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s decode (+18.0%)** and — the bigger win — **prefill roughly DOUBLED** (3,206→6,334 tok/s at 6.7k prompt; 2,862→5,085 at 27k; TTFT on a 27k doc 9.43→5.31 s), at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Prefill > decode is the expected ordering (decode is bandwidth-bound and 4-bit either way; prefill is compute-bound = where native FP4 replaces Marlin) — the `summarizer` aliases feel this most. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`.
- **⚠️ GPU0 is at 94.4/97.9 GB** (gen 0.43 + meromero 0.52). `GEN_GPU_MEM_UTIL` was cut 0.45→0.43 because the smaller mixed weights let gen soak the slack as KV and starved meromero by 0.18 GiB → crash-loop. **Any future util bump on either GPU0 seat must be checked against the other.** - **⚠️ GPU0 is at 94.4/97.9 GB** (gen 0.43 + meromero 0.52). `GEN_GPU_MEM_UTIL` was cut 0.45→0.43 because the smaller mixed weights let gen soak the slack as KV and starved meromero by 0.18 GiB → crash-loop. **Any future util bump on either GPU0 seat must be checked against the other.**