vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — gen seat mixed NVFP4+FP8 requant (+18%) + char-rp tool parser; both queued items closed

Browse Source
This commit is contained in:
vh
2026-08-15 02:22:58 -07:00
parent 74f596b1d3
commit fa4f652a39
2 changed files with 150 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+10 -4
View File
@@ -109,24 +109,30 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-08-15 — **UNCENSORED GEN SEAT LANDED**: JonathanColetti/Qwen3.8-27B-Uncensored is live as the fleet `gen` seat (project `gen-seat` / container `vllm-gen`, ana-ml2 :8015), all 7 gateway aliases repointed. **NEXT (operator-queued): chase W4A8** for the ~20% decode gain — see the handoff._
_As of 2026-08-15 — **GEN SEAT REQUANTED (mixed NVFP4+FP8, +18% decode) and the char-rp tool-call parser FIXED.** Both were the two open items from the prior session; both are closed, verified, and committed. Nothing queued behind them._
- **✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`.
- **⟳ NEXT — chase W4A8 (operator-queued).** Current seat is W4A16/Marlin (~68 tok/s decode); native NVFP4+FP8 (W4A8) recovers ~20% (→~82 tok/s) + faster prefill (MTP-acceptance half of the gap is inherent, not recoverable). bf16 source KEPT at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16` (no re-pull). **Avoid modelopt 0.43** (dependency hell for qwen3_5); use llm-compressor W4A8 + calibration. Full recipe + mandatory post-steps (graft MTP + reconstruct preproc + `re:^mtp.*` ignore) in the handoff + `reference_abliteration_mtp_lessons`.
- **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s (+18.0%)** at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`.
- **⚠️ GPU0 is at 94.4/97.9 GB** (gen 0.43 + meromero 0.52). `GEN_GPU_MEM_UTIL` was cut 0.45→0.43 because the smaller mixed weights let gen soak the slack as KV and starved meromero by 0.18 GiB → crash-loop. **Any future util bump on either GPU0 seat must be checked against the other.**
- **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013 (R42 v13 gate passed). Passive watch: caps ~34 req/s; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm for rollback. Full arc `docs/pfi/reranker-selection-ledger.md`.
- **EVIDENCE HOLD (partial):** WT #394 index-row half lifted+swept; the **FILE half STILL STANDS** — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`). rex393-fiction-* + r42-gate-* also KEEP.
- **OPEN FOLLOW-UPS (parked):** char-rp/MeroMero tool-call-parser fix (gemma4 → NOT qwen3_coder, open since session start); chatterbox-fast flat-vs-package build-context divergence; CI-flip runner-auth research; #363 research-wing ingest (no deadline); zonos-gateway CI-wire.
- **✅ CHAR-RP TOOL-CALLING — FIXED (2026-08-15).** MeroMero (Gemma-4) had **no** tool parser at all, so every tools-bearing request 400'd. Fixed with `--tool-call-parser gemma4 --enable-auto-tool-choice --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": false}'` — the last flag is **mandatory**, not decorative (the parser defaults `enable_thinking` to True, which pre-inits the engine to REASONING and returns null `content` for all plain RP prose). Verified green: tool call streaming + non-streaming, tool round-trip, prose in `content`, vision. Committed `b8f0f4c`. Details in `stacks/meromero-charrp/README.md`.
- **OPEN FOLLOW-UPS (parked):** chatterbox-fast flat-vs-package build-context divergence; CI-flip runner-auth research; #363 research-wing ingest (no deadline); zonos-gateway CI-wire.
- **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), NEVER after a plain operator turn (bounces rc3); spawn `althing-cli monitor` / `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&`.
- **eshpfi push state:** `origin/main` behind — unpushed: this session's `680c30e` (gen-seat), plus the earlier eRP dual-seat arc (`f08b6cb`/`7bd7375`), wgtunnel mirror (`398b58a`), dots.tts + secrets-broker arcs. Dotfiles: `1d1970f` (roster) + `6425cc6` (statusline bell) unpushed. **Push = operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, superseded).
- **eshpfi push state:** `origin/main` behind — unpushed: `74f596b` (gen-seat mixed requant) + `b8f0f4c` (char-rp tool parser) from the overnight session, `680c30e` (gen-seat), plus the earlier eRP dual-seat arc (`f08b6cb`/`7bd7375`), wgtunnel mirror (`398b58a`), dots.tts + secrets-broker arcs. Dotfiles: `1d1970f` (roster) + `6425cc6` (statusline bell) unpushed. **Push = operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, superseded).
## Recent decisions
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
- `[2026-08-12]` **eRP dual-seat overhaul: MeroMero-v2 (`char-rp`) + Dark-Scarlett (`char-rp-reasoning`), both NVFP4A16 @ 256K on ana-ml2; granite retired.** Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an `AutoModelForCausalLM` save wrote a flat `Qwen3_5TextConfig` that **both vLLM AND SGLang reject**) was fixed by re-quanting via the `Qwen3_5ForConditionalGeneration` **wrapper class**; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting `preprocessor_config.json` from `processor_config.json`. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed `f08b6cb`. granite downed + LiteLLM `summarizer`/`classifier`→gen. Full arc, lessons, dead-ends → `persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md`