diff --git a/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md b/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md new file mode 100644 index 0000000..e8fb0df --- /dev/null +++ b/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md @@ -0,0 +1,162 @@ +# Runbook — swap the `gen` seat to `absolute-heresy` + +Candidate: **`MuXodious/Qwen3.8-27B-absolute-heresy`** (Heretic v1.4.0 + SOMPOA, +trial T377), pinned `c23745939b34ca8b29ef707108572418ff031701`. + +Why it's a candidate: it beats the incumbent on **both** axes at once — +refusals **2/101** vs JonathanColetti's 12/100, first-token KL **0.0759** vs +0.1191 — and it is a structurally clean full checkpoint (1199 tensors, **15 +`mtp.*` present**, 333 `visual.*`, `lm_head`), so the existing mixed recipe +applies with no graft-and-reconstruct. + +⚠ It is a self-described **RC1**, published 2026-08-15, ~348 downloads / 9 +likes. Nothing here has been validated by anyone but its author, and its own +card carries a visibly broken GGUF benchmark block (RC1 and RC2 report +byte-identical piqa 0.4951 / copa 0.55 / winogrande 0.4957 — all at chance, +identical across two different models = broken harness). Treat every number on +that card as a claim, not a measurement. + +--- + +## Stage 0 — pull (DONE / in flight) + +Detached container `hf-pull-heresy` on ana-ml2 → +`/tank/aimodels/qwen38-27b-heresy-bf16` (~55.6 GB, 12 shards). `/tank` had +4.4 TB free. Write the `.PROVENANCE.txt` on completion (template: the sibling +`qwen38-27b-uncensored-bf16.PROVENANCE.txt`). + +--- + +## Stage 1 — bf16 MTP acceptance gate (DO THIS BEFORE SPENDING A QUANT) + +**Presence is not acceptance.** The incumbent's own provenance still reads +`MTP acceptance UNVERIFIED` — the author grafted 15 `mtp.*` verbatim and we +shipped on the assumption. It happened to measure 47.7% later; that was luck, +not method. Playbook: *test MTP on bf16 FIRST, isolate before deleting.* + +Gate: **acceptance ≳40%**. Below that, the mixed quant is not worth the GPU +time — the whole +18% decode story rests on MTP working. + +bf16 needs ~56 GB, so this needs a GPU window on its own (see § GPU reality). + +--- + +## Stage 2 — quant + +```bash +# 0. free, no GPU — prove the regexes hit the real module names +python3 validate_targets.py /tank/aimodels/qwen38-27b-heresy-bf16 +# expect: OVERLAP 0, MLP layer union covers 0-63 True + +# 1. ~20 min on one Blackwell. llmcompressor, NOT modelopt 0.43. +python3 quant_mixed_nvfp4.py \ + --model /tank/aimodels/qwen38-27b-heresy-bf16 \ + --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \ + --out /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed \ + --num-samples 256 --seqlen 2048 + +# 2. MANDATORY — graft MTP, restore preprocessor, re-inject the mtp ignore +python3 post_quant.py /tank/aimodels/qwen38-27b-heresy-bf16 \ + /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed +``` + +⚠ `llm-compressor` **prunes `ignore` entries that matched nothing at quant +time.** The wrapper class never loads the MTP head, so `re:^mtp.*` matches +nothing and is silently dropped — vLLM then treats the grafted BF16 head as +quantized, brings it up uninitialised, and speculative decoding runs at **0% +acceptance**. `post_quant.py` re-injects and *verifies*; it is not optional and +it has already fired on a real run. + +--- + +## Stage 3 — validate + +Playbook § 4.4 is explicit: **test on a temp port, never on the live seat.** +`bench/serve_probe.sh ` serves a candidate on `:8017` under its own +served-name. + +The full gate before any cutover: + +| check | tool | bar | +|---|---|---| +| MTP acceptance + cache-busted bs=1 decode | `bench/quickbench.py` | ≳40% accept, ≥ incumbent 94.5 tok/s | +| prefill / TTFT on long prompts | `bench/prefill_bench.py` | ≥ incumbent (6,334 @6.7k, 5,085 @27k) | +| perplexity + abliteration survival | `bench/eval_quality.py` | ≤ ~+2% vs incumbent 7.059 | +| **surface**: chat, vision, tool calling, thinking split, 36K needle, streaming | `bench/surface_test.py` | **all six** | +| refusal rate on our own battery | `services/refusal-probe/` | ≤ incumbent; the card's 2/101 is unverified | + +Measurement traps that have burned us — all documented, do not re-discover: +- **Bust the cache.** A fixed prompt + prefix caching returns byte-identical + timings; you measure nothing. +- **Never seed the prefill nonce.** A seeded nonce reproduces the prior run's + prompts verbatim → prefix cache serves them → you read ~41k tok/s of cache + hit instead of ~5k of real prefill. `SystemRandom`, always. +- **PPL must be measured with `--speculative-config` OFF.** Under MTP, + `prompt_logprobs` come back ~uniform over the vocab. The harness raises + rather than reporting garbage. + +--- + +## Stage 4 — cutover + +`gen-seat/.env`: point `GEN_MODEL` at the new dir, then + +```bash +cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen +``` + +⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose` +call against it needs `sudo`, or compose fails reading `.env`, leaves the old +container running, and **the change silently does not take**. + +⚠ Keep the pinned nightly. `GEN_IMAGE=vllm/vllm-openai:nightly-311b3513…` +carries the #51113 `qwen3_5_mtp`×GDN fix. Same architecture → same requirement. + +⚠ Watch the GPU0 co-tenant. If the new weights come out smaller, the seat +absorbs the slack as extra KV and can starve `meromero-charrp` — that exact +failure crash-looped it once. `GEN_GPU_MEM_UTIL=0.43` is the current setting +that keeps both resident at 89.8 / 97.9 GB. Re-check after the swap. + +**7 LiteLLM aliases ride this seat**: `gen`, `gen-reasoning`, `summarizer`, +`summarizer-large`, `classifier`, `image-judge`, `qwen-image-bench`. A bad +cutover degrades all of them at once. + +### Rollback + +Incumbent stays untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`. + +```bash +sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy- \ + /opt/docker/compose/gen-seat/.env +cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen +``` + +Take the `.env` backup **before** editing. Do not delete the incumbent weights +until the new seat has held through real multi-turn use — per the 2026-08-14 +delete-too-early lesson, and the whole multi-day degeneration saga that +followed. + +--- + +## GPU reality — why "in situ" is tempting and what to do instead + +Both cards sit at **~96.8 / 97.9 GB**. GPU0 is gen (0.43) + meromero (0.52); +GPU1 is fablefusion + the utility cluster. **There is no room to run a probe +alongside the live seat** — which is why replacing in place looks like the only +option. + +It isn't, quite. Downing `vllm-gen` frees its budget either way, so the +*disruption* is identical whether the candidate comes up on `:8017` or on the +live `:8015`. The difference is exposure: in situ points all 7 aliases at an +unvalidated RC the moment it starts. **Recommended sequence — same downtime, +much lower risk:** + +1. `compose down vllm-gen` (aliases are down either way for the window) +2. bring the candidate up via `serve_probe.sh` on `:8017`, its own served-name +3. run the Stage-3 gate +4. **pass** → flip `GEN_MODEL`, `up -d vllm-gen`, aliases return on validated + weights. **fail** → `up -d vllm-gen` unchanged, incumbent returns, nothing + downstream ever saw the candidate. + +The only thing in-situ buys is skipping one container start. It costs the +ability to fail without consequence.