memory: snapshot — gen seat mixed NVFP4+FP8 requant (+18%) + char-rp tool parser; both queued items closed

persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md

@@ -0,0 +1,140 @@
+# gen-seat mixed NVFP4+FP8 requant + char-rp tool-parser fix (2026-08-15, overnight)
+
+Autonomous overnight session. Two operator-queued items, both closed.
+
+## 1. char-rp / MeroMero tool-call parser (parked since the prior session)
+
+**Symptom:** every tools-bearing request to `char-rp` (:8016) returned
+`400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser`.
+The seat had **no tool parser configured at all** — the migration from the
+Magidonia GGUF seat dropped it.
+
+**Fix.** MeroMero-v2 is Gemma-4 and emits its own native
+`<|tool_call>call:name{...}<tool_call|>` syntax, not the qwen3_coder XML the
+Qwen-family seats use. vLLM 0.24 ships a `gemma4` tool parser whose
+TOOL_CALL_START/END + CHANNEL_START/END + escape constants match this
+tokenizer's `etc_token`/`eoc_token`/`escape_token` exactly (verified before
+deploying, not assumed).
+
+Four flags, and they are a **set**:
+
+```
+--tool-call-parser gemma4
+--enable-auto-tool-choice
+--reasoning-parser gemma4
+--default-chat-template-kwargs '{"enable_thinking": false}'
+```
+
+- Without the **reasoning parser**, the post-tool-response turn leaks a literal
+  `<|channel>thought\n<channel|>` prefix into `content` (upstream vllm #45834 —
+  the chat template leaves the prompt inside an open channel block).
+- The **`enable_thinking: false`** is mandatory, not cosmetic. The parser reads
+  it from `chat_template_kwargs` and **defaults it to `True`**
+  (`vllm/parser/gemma4.py:439`). True → `is_reasoning_end()` returns False at a
+  new turn → engine pre-initialises to REASONING → **all plain RP prose lands in
+  `reasoning_content` and `content` comes back null**, breaking every char-rp
+  consumer. Caught by reading the parser before deploying it.
+- **Zero behavioural risk, proven not asserted:** `chat_template.jinja:350`
+  already defaults `enable_thinking` to false, so passing it explicitly renders a
+  **byte-identical prompt** — diffed across plain / with-tools / post-tool-response
+  / system-prompt shapes before the flag went anywhere near the live seat.
+
+Verified green: tool call (streaming + non-streaming), tool-result round-trip
+(leak gone), plain prose in `content` with `reasoning` null, vision. Commit
+`b8f0f4c`.
+
+## 2. gen seat requant — the "W4A8" framing was wrong
+
+**The queued task was not servable as specified.** vLLM 0.24's compressed-tensors
+dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with exactly
+two activation options — `None` (W4A16, which **forces the Marlin kernel**,
+`kernels/linear/__init__.py:881-883`) or NVFP4 (W4A4). Anything else, FP8
+included, raises `ValueError: For NVFP4 weights, input quantization must also be
+NVFP4 format`. `CompressedTensorsW4A8Fp8` exists but is **INT4** weights
+(`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`) gated on `_check_scheme_supported(90,
+match_exact=True)` — Hopper only, so on Blackwell it is closed twice over.
+
+The ~20% intuition was right; the *scheme name* was wrong. FP8 has to enter
+**per-layer-group**, not as activations on NVFP4 weights.
+
+**Two baseline corrections.** The handoff's "~68 tok/s, ~42% acceptance" did not
+reproduce. Cache-busted (unique prompt per run — with a fixed prompt, prefix
+caching returns byte-identical timings and you measure nothing), the incumbent
+W4A16 build already did **80.12 tok/s at 47.8% acceptance** — i.e. essentially
+*at* the handoff's stated W4A8 target of ~82. Had that not been re-measured the
+whole chase would have been declared a success for doing nothing.
+
+**The shortcut that saved hours.** `unsloth/Qwen3.8-27B-NVFP4` was already on-box
+(pulled the previous day) — same architecture, same size, a published
+mixed-precision scheme. Serving it as a probe measured **+19.1% at identical MTP
+acceptance** — proving the gain was real and kernel-level *before* committing to
+a requant. Its config was then read out as the reference recipe.
+
+**The recipe** (byte-for-byte unsloth's, applied to the abliterated weights):
+
+| group | scheme | targets |
+|---|---|---|
+| `group_0` | FP8 W8A8, channel weights + per-token dynamic acts | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **layers 56-63** MLPs |
+| `group_1` | NVFP4 W4A4, tensor_group gsize16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` acts | **layers 0-55** MLP `{gate,up,down}_proj` |
+| kv | FP8 static tensor | — |
+| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | — |
+
+Holding the **last 8 layers' MLPs at FP8** is the accuracy trick. Targets were
+made explicitly non-overlapping (group_1 enumerates 0-55) rather than trusting
+group precedence, and `validate_targets.py` proved coverage against real module
+names — 0 overlap, MLP union = layers 0-63 — before any GPU time was spent.
+
+**Results (cache-busted, bs=1):**
+
+| metric | W4A16 | mixed | delta |
+|---|---|---|---|
+| decode tok/s | 80.12 | **94.53** | **+18.0%** |
+| MTP acceptance | 47.8% | 47.7% | unchanged |
+| perplexity (6 passages) | 6.941 | 7.059 | +1.7% worse |
+| abliteration compliance | 4/4 | 4/4 | preserved |
+| weights on disk | 27.7 GB | 22.5 GB | −19% |
+
+Surface test 6/6 on the live seat (plain chat, vision, tool calling, thinking
+split, 36K-token needle retrieval, streaming); all 7 LiteLLM aliases verified
+routing. Commit `74f596b`.
+
+## Foot-guns banked
+
+- **`llm-compressor` PRUNES `ignore` entries that matched no module at quant
+  time.** The wrapper class never loads the MTP head, so `re:^mtp.*` matched
+  nothing and was silently dropped from the saved config — the exact bug that
+  cost two prior rounds (vLLM then loads the grafted BF16 MTP as quantized →
+  uninitialised → 0% acceptance). `post_quant.py` now **re-injects it after the
+  graft and re-verifies**. That check *fired on this run* — it was not
+  hypothetical.
+- **vLLM's `prompt_logprobs` are garbage while speculative decoding is on** —
+  ~uniform over the vocab (median rank ~10⁵, logprob ≈ log(1/vocab); " Paris"
+  after "The capital of France is" ranked 69698). Perplexity must be measured on
+  a seat served **without** `--speculative-config`. The harness now raises rather
+  than reporting the garbage.
+- **`gen-seat/.env` is mode 0600 / lkraven-owned** → *every* `docker compose`
+  call needs `sudo`. Without it compose fails `permission denied` reading `.env`,
+  **leaves the old container running**, and the change silently does not take —
+  which produced one round of "benchmark results" that were just the unchanged
+  baseline. Hard-verify against `docker inspect` argv after any such change.
+- **GPU0 co-residency is a zero-sum budget.** The smaller mixed weights meant gen
+  at the old util 0.45 absorbed the slack as KV (17.0 GiB / 477K tokens) and left
+  meromero **0.18 GiB** short of its 0.52 → crash-loop. Fixed at
+  `GEN_GPU_MEM_UTIL=0.43` (15.1 GiB / 422K tokens, still 1.6× the 262K context).
+  Both seats now 94.4/97.9 GB.
+
+## Measured negatives — do not re-chase
+
+- **`GEN_SPEC_TOKENS` is already optimal at 3.** Swept on the live seat:
+  n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades
+  acceptance for draft width and loses.
+- **W4A4-everywhere was never attempted** and should not be — the accuracy-safe
+  shape is precisely the mixed one (FP8 on attention + late MLPs).
+
+## Artifacts
+
+- Pipeline + acceptance harness + raw JSON: `services/gen-seat-mixed-quant/`
+- Stack docs: `stacks/gen-seat/README.md`, `stacks/meromero-charrp/README.md`
+- Rollback: `sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 …/.env`
+  then `sudo docker compose up -d vllm-gen`; old build untouched at
+  `/tank/aimodels/qwen38-27b-uncensored-nvfp4`.