vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: gen seat RESOLVED — mixed FP8-attn build on pinned nightly, coherent through 60k

Browse Source 
Supersedes the stale 'GEN SEAT = AEON' current-state line. Records the
final resolved config (JonathanColetti/Heretic mixed NVFP4+FP8 on pinned
vLLM nightly, MTP on), the two-real-causes root cause, the AEON purge, and
the pinned-nightly follow-up (move to stable once #51113 ships). A cold
session now reads the correct gen-seat state.
This commit is contained in:
vh
2026-08-17 02:13:42 -07:00
parent 2f2bbce73d
commit 2185964a6a
1 changed files with 3 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+3 -1
View File
@@ -119,7 +119,9 @@ _As of 2026-08-16 — **one loop open: awaiting brokkr-smithy-dev's refusal batt
- **⏳ AWAITING: operator's hands-on read of Fable-Fusion's prose.** The refusal question is settled (below); prose quality is the only open input, and it needs a human. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8-27B base) exists but is **GATED** — our HF token gets `403 awaiting review`; **operator ruled it not interesting, do not re-propose.**
- **✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`.
- **🟢 GEN SEAT — RESOLVED 2026-08-17 (multi-day degeneration saga closed).** Primary gen = the in-house **JonathanColetti/Heretic mixed NVFP4+FP8 build** (`/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, FP8 attention) on **vLLM nightly pinned** `nightly-311b3513…` (`v0.27.2rc1.dev150`, carries the #51113 mamba fix), **MTP ON, prefix-caching ON**. Operator-confirmed **coherent through 60k tokens** of real multi-turn. Root cause was TWO compounding real causes, NOT one: (1) the genuine vLLM `qwen3_5_mtp`×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed enough by nightly), and (2) **AEON's full W4A4** being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). **AEON PURGED** from /tank (operator ruled no-good; re-pullable `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`). Full lesson: `docs/pfi/model-quantization-playbook.md` §3.8. Primary **until the DavidAU Qwen3.8 lands.** ⚠ pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one. Commits `d28a371`,`2f2bbce`. **7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here.**
- **[HISTORICAL] ✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`.
- **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s decode (+18.0%)** and — the bigger win — **prefill roughly DOUBLED** (3,206→6,334 tok/s at 6.7k prompt; 2,862→5,085 at 27k; TTFT on a 27k doc 9.43→5.31 s), at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Prefill > decode is the expected ordering (decode is bandwidth-bound and 4-bit either way; prefill is compute-bound = where native FP4 replaces Marlin) — the `summarizer` aliases feel this most. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`.