From 930197a56afd6fd3291b7bde94c8369f5ab63d69 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 15 Aug 2026 02:34:04 -0700 Subject: [PATCH] memory: record prefill result (roughly doubled) on the gen-seat mixed requant --- persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md | 7 +++++++ persistent-memory.md | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md b/persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md index 9863c2d..5809247 100644 --- a/persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md +++ b/persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md @@ -89,6 +89,9 @@ names — 0 overlap, MLP union = layers 0-63 — before any GPU time was spent. | metric | W4A16 | mixed | delta | |---|---|---|---| | decode tok/s | 80.12 | **94.53** | **+18.0%** | +| prefill tok/s (~6.7k prompt) | 3,206 | **6,334** | **+98%** | +| prefill tok/s (~27k prompt) | 2,862 | **5,085** | **+78%** | +| TTFT on a ~27k doc | 9.43 s | **5.31 s** | −44% | | MTP acceptance | 47.8% | 47.7% | unchanged | | perplexity (6 passages) | 6.941 | 7.059 | +1.7% worse | | abliteration compliance | 4/4 | 4/4 | preserved | @@ -107,6 +110,10 @@ routing. Commit `74f596b`. uninitialised → 0% acceptance). `post_quant.py` now **re-injects it after the graft and re-verifies**. That check *fired on this run* — it was not hypothetical. +- **Prefix caching silently fakes prefill numbers too.** The prefill harness originally used a + *seeded* nonce, so run 2 regenerated run 1's prompts verbatim and read **~41k tok/s of + cache-hit** instead of ~5k of real prefill. Same class of error as the decode bench. Use + `SystemRandom`; never seed a cache-busting nonce. - **vLLM's `prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab (median rank ~10⁵, logprob ≈ log(1/vocab); " Paris" after "The capital of France is" ranked 69698). Perplexity must be measured on diff --git a/persistent-memory.md b/persistent-memory.md index 7081408..259b7ed 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -113,7 +113,7 @@ _As of 2026-08-15 — **GEN SEAT REQUANTED (mixed NVFP4+FP8, +18% decode) and th - **✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`. -- **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s (+18.0%)** at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`. +- **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s decode (+18.0%)** and — the bigger win — **prefill roughly DOUBLED** (3,206→6,334 tok/s at 6.7k prompt; 2,862→5,085 at 27k; TTFT on a 27k doc 9.43→5.31 s), at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Prefill > decode is the expected ordering (decode is bandwidth-bound and 4-bit either way; prefill is compute-bound = where native FP4 replaces Marlin) — the `summarizer` aliases feel this most. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`. - **⚠️ GPU0 is at 94.4/97.9 GB** (gen 0.43 + meromero 0.52). `GEN_GPU_MEM_UTIL` was cut 0.45→0.43 because the smaller mixed weights let gen soak the slack as KV and starved meromero by 0.18 GiB → crash-loop. **Any future util bump on either GPU0 seat must be checked against the other.**