memory: snapshot — gen seat mixed NVFP4+FP8 requant (+18%) + char-rp tool parser; both queued items closed
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
# gen-seat mixed NVFP4+FP8 requant + char-rp tool-parser fix (2026-08-15, overnight)
|
||||
|
||||
Autonomous overnight session. Two operator-queued items, both closed.
|
||||
|
||||
## 1. char-rp / MeroMero tool-call parser (parked since the prior session)
|
||||
|
||||
**Symptom:** every tools-bearing request to `char-rp` (:8016) returned
|
||||
`400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser`.
|
||||
The seat had **no tool parser configured at all** — the migration from the
|
||||
Magidonia GGUF seat dropped it.
|
||||
|
||||
**Fix.** MeroMero-v2 is Gemma-4 and emits its own native
|
||||
`<|tool_call>call:name{...}<tool_call|>` syntax, not the qwen3_coder XML the
|
||||
Qwen-family seats use. vLLM 0.24 ships a `gemma4` tool parser whose
|
||||
TOOL_CALL_START/END + CHANNEL_START/END + escape constants match this
|
||||
tokenizer's `etc_token`/`eoc_token`/`escape_token` exactly (verified before
|
||||
deploying, not assumed).
|
||||
|
||||
Four flags, and they are a **set**:
|
||||
|
||||
```
|
||||
--tool-call-parser gemma4
|
||||
--enable-auto-tool-choice
|
||||
--reasoning-parser gemma4
|
||||
--default-chat-template-kwargs '{"enable_thinking": false}'
|
||||
```
|
||||
|
||||
- Without the **reasoning parser**, the post-tool-response turn leaks a literal
|
||||
`<|channel>thought\n<channel|>` prefix into `content` (upstream vllm #45834 —
|
||||
the chat template leaves the prompt inside an open channel block).
|
||||
- The **`enable_thinking: false`** is mandatory, not cosmetic. The parser reads
|
||||
it from `chat_template_kwargs` and **defaults it to `True`**
|
||||
(`vllm/parser/gemma4.py:439`). True → `is_reasoning_end()` returns False at a
|
||||
new turn → engine pre-initialises to REASONING → **all plain RP prose lands in
|
||||
`reasoning_content` and `content` comes back null**, breaking every char-rp
|
||||
consumer. Caught by reading the parser before deploying it.
|
||||
- **Zero behavioural risk, proven not asserted:** `chat_template.jinja:350`
|
||||
already defaults `enable_thinking` to false, so passing it explicitly renders a
|
||||
**byte-identical prompt** — diffed across plain / with-tools / post-tool-response
|
||||
/ system-prompt shapes before the flag went anywhere near the live seat.
|
||||
|
||||
Verified green: tool call (streaming + non-streaming), tool-result round-trip
|
||||
(leak gone), plain prose in `content` with `reasoning` null, vision. Commit
|
||||
`b8f0f4c`.
|
||||
|
||||
## 2. gen seat requant — the "W4A8" framing was wrong
|
||||
|
||||
**The queued task was not servable as specified.** vLLM 0.24's compressed-tensors
|
||||
dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with exactly
|
||||
two activation options — `None` (W4A16, which **forces the Marlin kernel**,
|
||||
`kernels/linear/__init__.py:881-883`) or NVFP4 (W4A4). Anything else, FP8
|
||||
included, raises `ValueError: For NVFP4 weights, input quantization must also be
|
||||
NVFP4 format`. `CompressedTensorsW4A8Fp8` exists but is **INT4** weights
|
||||
(`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`) gated on `_check_scheme_supported(90,
|
||||
match_exact=True)` — Hopper only, so on Blackwell it is closed twice over.
|
||||
|
||||
The ~20% intuition was right; the *scheme name* was wrong. FP8 has to enter
|
||||
**per-layer-group**, not as activations on NVFP4 weights.
|
||||
|
||||
**Two baseline corrections.** The handoff's "~68 tok/s, ~42% acceptance" did not
|
||||
reproduce. Cache-busted (unique prompt per run — with a fixed prompt, prefix
|
||||
caching returns byte-identical timings and you measure nothing), the incumbent
|
||||
W4A16 build already did **80.12 tok/s at 47.8% acceptance** — i.e. essentially
|
||||
*at* the handoff's stated W4A8 target of ~82. Had that not been re-measured the
|
||||
whole chase would have been declared a success for doing nothing.
|
||||
|
||||
**The shortcut that saved hours.** `unsloth/Qwen3.8-27B-NVFP4` was already on-box
|
||||
(pulled the previous day) — same architecture, same size, a published
|
||||
mixed-precision scheme. Serving it as a probe measured **+19.1% at identical MTP
|
||||
acceptance** — proving the gain was real and kernel-level *before* committing to
|
||||
a requant. Its config was then read out as the reference recipe.
|
||||
|
||||
**The recipe** (byte-for-byte unsloth's, applied to the abliterated weights):
|
||||
|
||||
| group | scheme | targets |
|
||||
|---|---|---|
|
||||
| `group_0` | FP8 W8A8, channel weights + per-token dynamic acts | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **layers 56-63** MLPs |
|
||||
| `group_1` | NVFP4 W4A4, tensor_group gsize16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` acts | **layers 0-55** MLP `{gate,up,down}_proj` |
|
||||
| kv | FP8 static tensor | — |
|
||||
| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | — |
|
||||
|
||||
Holding the **last 8 layers' MLPs at FP8** is the accuracy trick. Targets were
|
||||
made explicitly non-overlapping (group_1 enumerates 0-55) rather than trusting
|
||||
group precedence, and `validate_targets.py` proved coverage against real module
|
||||
names — 0 overlap, MLP union = layers 0-63 — before any GPU time was spent.
|
||||
|
||||
**Results (cache-busted, bs=1):**
|
||||
|
||||
| metric | W4A16 | mixed | delta |
|
||||
|---|---|---|---|
|
||||
| decode tok/s | 80.12 | **94.53** | **+18.0%** |
|
||||
| MTP acceptance | 47.8% | 47.7% | unchanged |
|
||||
| perplexity (6 passages) | 6.941 | 7.059 | +1.7% worse |
|
||||
| abliteration compliance | 4/4 | 4/4 | preserved |
|
||||
| weights on disk | 27.7 GB | 22.5 GB | −19% |
|
||||
|
||||
Surface test 6/6 on the live seat (plain chat, vision, tool calling, thinking
|
||||
split, 36K-token needle retrieval, streaming); all 7 LiteLLM aliases verified
|
||||
routing. Commit `74f596b`.
|
||||
|
||||
## Foot-guns banked
|
||||
|
||||
- **`llm-compressor` PRUNES `ignore` entries that matched no module at quant
|
||||
time.** The wrapper class never loads the MTP head, so `re:^mtp.*` matched
|
||||
nothing and was silently dropped from the saved config — the exact bug that
|
||||
cost two prior rounds (vLLM then loads the grafted BF16 MTP as quantized →
|
||||
uninitialised → 0% acceptance). `post_quant.py` now **re-injects it after the
|
||||
graft and re-verifies**. That check *fired on this run* — it was not
|
||||
hypothetical.
|
||||
- **vLLM's `prompt_logprobs` are garbage while speculative decoding is on** —
|
||||
~uniform over the vocab (median rank ~10⁵, logprob ≈ log(1/vocab); " Paris"
|
||||
after "The capital of France is" ranked 69698). Perplexity must be measured on
|
||||
a seat served **without** `--speculative-config`. The harness now raises rather
|
||||
than reporting the garbage.
|
||||
- **`gen-seat/.env` is mode 0600 / lkraven-owned** → *every* `docker compose`
|
||||
call needs `sudo`. Without it compose fails `permission denied` reading `.env`,
|
||||
**leaves the old container running**, and the change silently does not take —
|
||||
which produced one round of "benchmark results" that were just the unchanged
|
||||
baseline. Hard-verify against `docker inspect` argv after any such change.
|
||||
- **GPU0 co-residency is a zero-sum budget.** The smaller mixed weights meant gen
|
||||
at the old util 0.45 absorbed the slack as KV (17.0 GiB / 477K tokens) and left
|
||||
meromero **0.18 GiB** short of its 0.52 → crash-loop. Fixed at
|
||||
`GEN_GPU_MEM_UTIL=0.43` (15.1 GiB / 422K tokens, still 1.6× the 262K context).
|
||||
Both seats now 94.4/97.9 GB.
|
||||
|
||||
## Measured negatives — do not re-chase
|
||||
|
||||
- **`GEN_SPEC_TOKENS` is already optimal at 3.** Swept on the live seat:
|
||||
n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades
|
||||
acceptance for draft width and loses.
|
||||
- **W4A4-everywhere was never attempted** and should not be — the accuracy-safe
|
||||
shape is precisely the mixed one (FP8 on attention + late MLPs).
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Pipeline + acceptance harness + raw JSON: `services/gen-seat-mixed-quant/`
|
||||
- Stack docs: `stacks/gen-seat/README.md`, `stacks/meromero-charrp/README.md`
|
||||
- Rollback: `sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 …/.env`
|
||||
then `sudo docker compose up -d vllm-gen`; old build untouched at
|
||||
`/tank/aimodels/qwen38-27b-uncensored-nvfp4`.
|
||||
+10
-4
@@ -109,24 +109,30 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-08-15 — **UNCENSORED GEN SEAT LANDED**: JonathanColetti/Qwen3.8-27B-Uncensored is live as the fleet `gen` seat (project `gen-seat` / container `vllm-gen`, ana-ml2 :8015), all 7 gateway aliases repointed. **NEXT (operator-queued): chase W4A8** for the ~20% decode gain — see the handoff._
|
||||
_As of 2026-08-15 — **GEN SEAT REQUANTED (mixed NVFP4+FP8, +18% decode) and the char-rp tool-call parser FIXED.** Both were the two open items from the prior session; both are closed, verified, and committed. Nothing queued behind them._
|
||||
|
||||
- **✅ UNCENSORED GEN SEAT — DONE + LIVE (2026-08-15).** `gen-seat`/`vllm-gen` on ana-ml2 GPU0 `:8015` serves **`qwen3.8-27b-uncensored`** (JonathanColetti/Qwen3.8-27B-Uncensored, Heretic-abliterated, in-house NVFP4 **W4A16** compressed-tensors + grafted bf16 MTP, vision-intact, **262K** ctx, MTP n=3 ~42% accept / ~68 tok/s). Replaced the qwen3.6-35b-a3b-heretic MoE (which had briefly replaced granite/AEON). All 7 aliases repointed live + verified; compose renamed qwen36-27b-aeon→gen-seat, vllm-aeon-gen→vllm-gen, AEON_GEN_*→GEN_*, dead RP service dropped; repo mirrored + docs/memory refreshed; committed eshpfi `680c30e` + dotfiles `1d1970f`. Full arc + the definitive `re:^mtp.*`-ignore fix → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`.
|
||||
|
||||
- **⟳ NEXT — chase W4A8 (operator-queued).** Current seat is W4A16/Marlin (~68 tok/s decode); native NVFP4+FP8 (W4A8) recovers ~20% (→~82 tok/s) + faster prefill (MTP-acceptance half of the gap is inherent, not recoverable). bf16 source KEPT at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16` (no re-pull). **Avoid modelopt 0.43** (dependency hell for qwen3_5); use llm-compressor W4A8 + calibration. Full recipe + mandatory post-steps (graft MTP + reconstruct preproc + `re:^mtp.*` ignore) in the handoff + `reference_abliteration_mtp_lessons`.
|
||||
- **✅ GEN SEAT REQUANT — DONE + LIVE (2026-08-15, overnight).** The seat now runs **mixed-precision** `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed` (NVFP4 W4A4 layers 0-55 MLPs + FP8 W8A8 attn/`linear_attn`/`lm_head`/layers 56-63 MLPs + FP8 KV) — **80.12 → 94.53 tok/s (+18.0%)** at unchanged MTP acceptance (47.8→47.7%), +1.7% PPL, abliteration 4/4 preserved, weights 27.7→22.5 GB. Surface-verified live (chat/vision/tools/thinking/36K-needle/streaming 6/6) + all 7 aliases routing. **The queued "W4A8" framing was unservable** — vLLM 0.24 allows NVFP4 weights with ONLY A16 or A4, FP8 activations raise ValueError at load; FP8 has to enter per-layer-group. Committed `74f596b`. Pipeline + acceptance harness + raw numbers → `services/gen-seat-mixed-quant/`; full arc → Recent decisions `[2026-08-15]` + `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`. **Rollback = one `.env` line**, old build untouched at `…/qwen38-27b-uncensored-nvfp4`.
|
||||
|
||||
- **⚠️ GPU0 is at 94.4/97.9 GB** (gen 0.43 + meromero 0.52). `GEN_GPU_MEM_UTIL` was cut 0.45→0.43 because the smaller mixed weights let gen soak the slack as KV and starved meromero by 0.18 GiB → crash-loop. **Any future util bump on either GPU0 seat must be checked against the other.**
|
||||
|
||||
- **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013 (R42 v13 gate passed). Passive watch: caps ~34 req/s; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm for rollback. Full arc `docs/pfi/reranker-selection-ledger.md`.
|
||||
|
||||
- **EVIDENCE HOLD (partial):** WT #394 index-row half lifted+swept; the **FILE half STILL STANDS** — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`). rex393-fiction-* + r42-gate-* also KEEP.
|
||||
|
||||
- **OPEN FOLLOW-UPS (parked):** char-rp/MeroMero tool-call-parser fix (gemma4 → NOT qwen3_coder, open since session start); chatterbox-fast flat-vs-package build-context divergence; CI-flip runner-auth research; #363 research-wing ingest (no deadline); zonos-gateway CI-wire.
|
||||
- **✅ CHAR-RP TOOL-CALLING — FIXED (2026-08-15).** MeroMero (Gemma-4) had **no** tool parser at all, so every tools-bearing request 400'd. Fixed with `--tool-call-parser gemma4 --enable-auto-tool-choice --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": false}'` — the last flag is **mandatory**, not decorative (the parser defaults `enable_thinking` to True, which pre-inits the engine to REASONING and returns null `content` for all plain RP prose). Verified green: tool call streaming + non-streaming, tool round-trip, prose in `content`, vision. Committed `b8f0f4c`. Details in `stacks/meromero-charrp/README.md`.
|
||||
|
||||
- **OPEN FOLLOW-UPS (parked):** chatterbox-fast flat-vs-package build-context divergence; CI-flip runner-auth research; #363 research-wing ingest (no deadline); zonos-gateway CI-wire.
|
||||
|
||||
- **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), NEVER after a plain operator turn (bounces rc3); spawn `althing-cli monitor` / `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&`.
|
||||
|
||||
- **eshpfi push state:** `origin/main` behind — unpushed: this session's `680c30e` (gen-seat), plus the earlier eRP dual-seat arc (`f08b6cb`/`7bd7375`), wgtunnel mirror (`398b58a`), dots.tts + secrets-broker arcs. Dotfiles: `1d1970f` (roster) + `6425cc6` (statusline bell) unpushed. **Push = operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, superseded).
|
||||
- **eshpfi push state:** `origin/main` behind — unpushed: `74f596b` (gen-seat mixed requant) + `b8f0f4c` (char-rp tool parser) from the overnight session, `680c30e` (gen-seat), plus the earlier eRP dual-seat arc (`f08b6cb`/`7bd7375`), wgtunnel mirror (`398b58a`), dots.tts + secrets-broker arcs. Dotfiles: `1d1970f` (roster) + `6425cc6` (statusline bell) unpushed. **Push = operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, superseded).
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
|
||||
|
||||
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
|
||||
|
||||
- `[2026-08-12]` **eRP dual-seat overhaul: MeroMero-v2 (`char-rp`) + Dark-Scarlett (`char-rp-reasoning`), both NVFP4A16 @ 256K on ana-ml2; granite retired.** Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an `AutoModelForCausalLM` save wrote a flat `Qwen3_5TextConfig` that **both vLLM AND SGLang reject**) was fixed by re-quanting via the `Qwen3_5ForConditionalGeneration` **wrapper class**; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting `preprocessor_config.json` from `processor_config.json`. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed `f08b6cb`. granite downed + LiteLLM `summarizer`/`classifier`→gen. Full arc, lessons, dead-ends → `persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md`
|
||||
|
||||
Reference in New Issue
Block a user