docs(gen-seat): runbook for the absolute-heresy swap
Candidate MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, trial T377), pinned c2374593. Beats the incumbent on both axes: refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. Structurally a clean full checkpoint (1199 tensors, 15 mtp.*, 333 visual.*, lm_head), so the existing mixed NVFP4+FP8 recipe applies with no graft-and-reconstruct. Runbook carries the bf16 MTP-acceptance gate before any quant spend, the llm-compressor ignore-pruning foot-gun, the three measurement traps (cache-busting, unseeded prefill nonce, PPL with spec off), the .env 0600 sudo trap, the GPU0 co-tenant starvation risk, and rollback. Flags that the candidate is a 2-day-old RC1 whose own card carries a broken GGUF benchmark block (RC1 and RC2 report identical at-chance scores across three benchmarks), so its numbers are claims rather than measurements.
This commit is contained in:
@@ -0,0 +1,162 @@
|
|||||||
|
# Runbook — swap the `gen` seat to `absolute-heresy`
|
||||||
|
|
||||||
|
Candidate: **`MuXodious/Qwen3.8-27B-absolute-heresy`** (Heretic v1.4.0 + SOMPOA,
|
||||||
|
trial T377), pinned `c23745939b34ca8b29ef707108572418ff031701`.
|
||||||
|
|
||||||
|
Why it's a candidate: it beats the incumbent on **both** axes at once —
|
||||||
|
refusals **2/101** vs JonathanColetti's 12/100, first-token KL **0.0759** vs
|
||||||
|
0.1191 — and it is a structurally clean full checkpoint (1199 tensors, **15
|
||||||
|
`mtp.*` present**, 333 `visual.*`, `lm_head`), so the existing mixed recipe
|
||||||
|
applies with no graft-and-reconstruct.
|
||||||
|
|
||||||
|
⚠ It is a self-described **RC1**, published 2026-08-15, ~348 downloads / 9
|
||||||
|
likes. Nothing here has been validated by anyone but its author, and its own
|
||||||
|
card carries a visibly broken GGUF benchmark block (RC1 and RC2 report
|
||||||
|
byte-identical piqa 0.4951 / copa 0.55 / winogrande 0.4957 — all at chance,
|
||||||
|
identical across two different models = broken harness). Treat every number on
|
||||||
|
that card as a claim, not a measurement.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 0 — pull (DONE / in flight)
|
||||||
|
|
||||||
|
Detached container `hf-pull-heresy` on ana-ml2 →
|
||||||
|
`/tank/aimodels/qwen38-27b-heresy-bf16` (~55.6 GB, 12 shards). `/tank` had
|
||||||
|
4.4 TB free. Write the `.PROVENANCE.txt` on completion (template: the sibling
|
||||||
|
`qwen38-27b-uncensored-bf16.PROVENANCE.txt`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 1 — bf16 MTP acceptance gate (DO THIS BEFORE SPENDING A QUANT)
|
||||||
|
|
||||||
|
**Presence is not acceptance.** The incumbent's own provenance still reads
|
||||||
|
`MTP acceptance UNVERIFIED` — the author grafted 15 `mtp.*` verbatim and we
|
||||||
|
shipped on the assumption. It happened to measure 47.7% later; that was luck,
|
||||||
|
not method. Playbook: *test MTP on bf16 FIRST, isolate before deleting.*
|
||||||
|
|
||||||
|
Gate: **acceptance ≳40%**. Below that, the mixed quant is not worth the GPU
|
||||||
|
time — the whole +18% decode story rests on MTP working.
|
||||||
|
|
||||||
|
bf16 needs ~56 GB, so this needs a GPU window on its own (see § GPU reality).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 2 — quant
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 0. free, no GPU — prove the regexes hit the real module names
|
||||||
|
python3 validate_targets.py /tank/aimodels/qwen38-27b-heresy-bf16
|
||||||
|
# expect: OVERLAP 0, MLP layer union covers 0-63 True
|
||||||
|
|
||||||
|
# 1. ~20 min on one Blackwell. llmcompressor, NOT modelopt 0.43.
|
||||||
|
python3 quant_mixed_nvfp4.py \
|
||||||
|
--model /tank/aimodels/qwen38-27b-heresy-bf16 \
|
||||||
|
--calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||||||
|
--out /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed \
|
||||||
|
--num-samples 256 --seqlen 2048
|
||||||
|
|
||||||
|
# 2. MANDATORY — graft MTP, restore preprocessor, re-inject the mtp ignore
|
||||||
|
python3 post_quant.py /tank/aimodels/qwen38-27b-heresy-bf16 \
|
||||||
|
/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠ `llm-compressor` **prunes `ignore` entries that matched nothing at quant
|
||||||
|
time.** The wrapper class never loads the MTP head, so `re:^mtp.*` matches
|
||||||
|
nothing and is silently dropped — vLLM then treats the grafted BF16 head as
|
||||||
|
quantized, brings it up uninitialised, and speculative decoding runs at **0%
|
||||||
|
acceptance**. `post_quant.py` re-injects and *verifies*; it is not optional and
|
||||||
|
it has already fired on a real run.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 3 — validate
|
||||||
|
|
||||||
|
Playbook § 4.4 is explicit: **test on a temp port, never on the live seat.**
|
||||||
|
`bench/serve_probe.sh <model-dir>` serves a candidate on `:8017` under its own
|
||||||
|
served-name.
|
||||||
|
|
||||||
|
The full gate before any cutover:
|
||||||
|
|
||||||
|
| check | tool | bar |
|
||||||
|
|---|---|---|
|
||||||
|
| MTP acceptance + cache-busted bs=1 decode | `bench/quickbench.py` | ≳40% accept, ≥ incumbent 94.5 tok/s |
|
||||||
|
| prefill / TTFT on long prompts | `bench/prefill_bench.py` | ≥ incumbent (6,334 @6.7k, 5,085 @27k) |
|
||||||
|
| perplexity + abliteration survival | `bench/eval_quality.py` | ≤ ~+2% vs incumbent 7.059 |
|
||||||
|
| **surface**: chat, vision, tool calling, thinking split, 36K needle, streaming | `bench/surface_test.py` | **all six** |
|
||||||
|
| refusal rate on our own battery | `services/refusal-probe/` | ≤ incumbent; the card's 2/101 is unverified |
|
||||||
|
|
||||||
|
Measurement traps that have burned us — all documented, do not re-discover:
|
||||||
|
- **Bust the cache.** A fixed prompt + prefix caching returns byte-identical
|
||||||
|
timings; you measure nothing.
|
||||||
|
- **Never seed the prefill nonce.** A seeded nonce reproduces the prior run's
|
||||||
|
prompts verbatim → prefix cache serves them → you read ~41k tok/s of cache
|
||||||
|
hit instead of ~5k of real prefill. `SystemRandom`, always.
|
||||||
|
- **PPL must be measured with `--speculative-config` OFF.** Under MTP,
|
||||||
|
`prompt_logprobs` come back ~uniform over the vocab. The harness raises
|
||||||
|
rather than reporting garbage.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 4 — cutover
|
||||||
|
|
||||||
|
`gen-seat/.env`: point `GEN_MODEL` at the new dir, then
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose`
|
||||||
|
call against it needs `sudo`, or compose fails reading `.env`, leaves the old
|
||||||
|
container running, and **the change silently does not take**.
|
||||||
|
|
||||||
|
⚠ Keep the pinned nightly. `GEN_IMAGE=vllm/vllm-openai:nightly-311b3513…`
|
||||||
|
carries the #51113 `qwen3_5_mtp`×GDN fix. Same architecture → same requirement.
|
||||||
|
|
||||||
|
⚠ Watch the GPU0 co-tenant. If the new weights come out smaller, the seat
|
||||||
|
absorbs the slack as extra KV and can starve `meromero-charrp` — that exact
|
||||||
|
failure crash-looped it once. `GEN_GPU_MEM_UTIL=0.43` is the current setting
|
||||||
|
that keeps both resident at 89.8 / 97.9 GB. Re-check after the swap.
|
||||||
|
|
||||||
|
**7 LiteLLM aliases ride this seat**: `gen`, `gen-reasoning`, `summarizer`,
|
||||||
|
`summarizer-large`, `classifier`, `image-judge`, `qwen-image-bench`. A bad
|
||||||
|
cutover degrades all of them at once.
|
||||||
|
|
||||||
|
### Rollback
|
||||||
|
|
||||||
|
Incumbent stays untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-<date> \
|
||||||
|
/opt/docker/compose/gen-seat/.env
|
||||||
|
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
|
||||||
|
```
|
||||||
|
|
||||||
|
Take the `.env` backup **before** editing. Do not delete the incumbent weights
|
||||||
|
until the new seat has held through real multi-turn use — per the 2026-08-14
|
||||||
|
delete-too-early lesson, and the whole multi-day degeneration saga that
|
||||||
|
followed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GPU reality — why "in situ" is tempting and what to do instead
|
||||||
|
|
||||||
|
Both cards sit at **~96.8 / 97.9 GB**. GPU0 is gen (0.43) + meromero (0.52);
|
||||||
|
GPU1 is fablefusion + the utility cluster. **There is no room to run a probe
|
||||||
|
alongside the live seat** — which is why replacing in place looks like the only
|
||||||
|
option.
|
||||||
|
|
||||||
|
It isn't, quite. Downing `vllm-gen` frees its budget either way, so the
|
||||||
|
*disruption* is identical whether the candidate comes up on `:8017` or on the
|
||||||
|
live `:8015`. The difference is exposure: in situ points all 7 aliases at an
|
||||||
|
unvalidated RC the moment it starts. **Recommended sequence — same downtime,
|
||||||
|
much lower risk:**
|
||||||
|
|
||||||
|
1. `compose down vllm-gen` (aliases are down either way for the window)
|
||||||
|
2. bring the candidate up via `serve_probe.sh` on `:8017`, its own served-name
|
||||||
|
3. run the Stage-3 gate
|
||||||
|
4. **pass** → flip `GEN_MODEL`, `up -d vllm-gen`, aliases return on validated
|
||||||
|
weights. **fail** → `up -d vllm-gen` unchanged, incumbent returns, nothing
|
||||||
|
downstream ever saw the candidate.
|
||||||
|
|
||||||
|
The only thing in-situ buys is skipping one container start. It costs the
|
||||||
|
ability to fail without consequence.
|
||||||
Reference in New Issue
Block a user