vh
254c588921
Operator ruled the probe port for validation; runbook updated to match. The bf16 MTP acceptance gate is ~56 GB resident, which on a full 97.9 GB card means downing meromero-charrp as well as gen -- freeing gen's 0.43 (~42 GB) alone is not enough. Two seats down to answer one question. compare_mtp_head.py answers the common case for free. The Qwen3_5ForConditionalGeneration wrapper never loads the MTP head, so PEFT merges, Heretic runs, and llm-compressor passes all leave mtp.* as it came from the base. It hashes a candidate's 15 mtp.* tensors against the incumbent's grafted-verbatim head -- the one measured at 47.7% acceptance in production through this exact pipeline. Identical means the acceptance question is already answered; different means the head was edited and the real gate is warranted; missing means it was dropped. CPU only, reads just the shard holding mtp.*. The runbook states the residual risk plainly: an identical head proves the head is intact, not that the abliterated body still drafts well with it -- which the Stage-3 acceptance measurement on the 22 GB quantized build catches anyway.
205 lines
9.2 KiB
Markdown
205 lines
9.2 KiB
Markdown
# Runbook — swap the `gen` seat to `absolute-heresy`
|
||
|
||
Candidate: **`MuXodious/Qwen3.8-27B-absolute-heresy`** (Heretic v1.4.0 + SOMPOA,
|
||
trial T377), pinned `c23745939b34ca8b29ef707108572418ff031701`.
|
||
|
||
Why it's a candidate: it beats the incumbent on **both** axes at once —
|
||
refusals **2/101** vs JonathanColetti's 12/100, first-token KL **0.0759** vs
|
||
0.1191 — and it is a structurally clean full checkpoint (1199 tensors, **15
|
||
`mtp.*` present**, 333 `visual.*`, `lm_head`), so the existing mixed recipe
|
||
applies with no graft-and-reconstruct.
|
||
|
||
⚠ It is a self-described **RC1**, published 2026-08-15, ~348 downloads / 9
|
||
likes. Nothing here has been validated by anyone but its author, and its own
|
||
card carries a visibly broken GGUF benchmark block (RC1 and RC2 report
|
||
byte-identical piqa 0.4951 / copa 0.55 / winogrande 0.4957 — all at chance,
|
||
identical across two different models = broken harness). Treat every number on
|
||
that card as a claim, not a measurement.
|
||
|
||
---
|
||
|
||
## Stage 0 — pull (DONE / in flight)
|
||
|
||
Detached container `hf-pull-heresy` on ana-ml2 →
|
||
`/tank/aimodels/qwen38-27b-heresy-bf16` (~55.6 GB, 12 shards). `/tank` had
|
||
4.4 TB free. Write the `.PROVENANCE.txt` on completion (template: the sibling
|
||
`qwen38-27b-uncensored-bf16.PROVENANCE.txt`).
|
||
|
||
---
|
||
|
||
## Stage 1 — the MTP gate, WITHOUT downing a second seat
|
||
|
||
**Presence is not acceptance.** The incumbent's own provenance still reads
|
||
`MTP acceptance UNVERIFIED` — the author grafted 15 `mtp.*` verbatim and we
|
||
shipped on the assumption. It measured 47.7% later; that was luck, not method.
|
||
|
||
The literal reading of the playbook rule (*test MTP on bf16 FIRST*) is
|
||
**expensive here**: bf16 is ~56 GB resident, and on a full 97.9 GB card that
|
||
means downing `meromero-charrp` as well as `gen` — freeing gen's 0.43 (~42 GB)
|
||
alone is not enough. Two seats down to answer one question.
|
||
|
||
### 1a. Free, CPU-only: is the head one we have already measured?
|
||
|
||
The wrapper class `Qwen3_5ForConditionalGeneration` **does not load the MTP
|
||
head**, so PEFT merges, Heretic runs, and llm-compressor passes all leave
|
||
`mtp.*` exactly as it came from the base. If the candidate's 15 `mtp.*` tensors
|
||
are numerically identical to a head we have measured in production, the
|
||
acceptance question is already answered.
|
||
|
||
```bash
|
||
# runs in the vllm image (has torch + safetensors); reads only the mtp shard
|
||
sudo docker run --rm -v /tank/aimodels:/models --entrypoint python3 \
|
||
vllm/vllm-openai:latest /models/compare_mtp_head.py \
|
||
/models/qwen38-27b-heresy-bf16 /models/qwen38-27b-uncensored-bf16
|
||
```
|
||
|
||
Reference is the incumbent's `model-mtp.safetensors` — grafted verbatim from
|
||
base per its PROVENANCE, and **measured at 47.7% acceptance** on the live seat
|
||
through this exact pipeline. Known-good, not assumed.
|
||
|
||
- **IDENTICAL** → skip the bf16 gate entirely. Go to Stage 2 and verify
|
||
acceptance on the **quantized** build (~22 GB) inside the freed `gen` budget.
|
||
No second seat down.
|
||
- **DIFFERENT** → the head was edited. Not automatically bad — a deliberate
|
||
`mtp.*` abliteration is legitimate (hotdogs/Qwen3.8-27B-abliterated edits 2
|
||
mtp tensors on purpose) — but it is no longer a head we have measured, so
|
||
1b becomes necessary.
|
||
- **MISSING** → dropped; needs a graft.
|
||
|
||
### 1b. Only if 1a says DIFFERENT: the real bf16 gate
|
||
|
||
Gate: **acceptance ≳40%**. Below that the mixed quant is not worth the GPU time
|
||
— the whole +18% decode case rests on MTP working. Costs `gen` **and**
|
||
`meromero-charrp` for the window.
|
||
|
||
**Residual risk of taking the 1a shortcut, stated plainly:** an identical head
|
||
proves the *head* is intact, not that the surrounding model still drafts well
|
||
with it. The MTP head reads hidden states from a body that abliteration *did*
|
||
change, so acceptance could in principle move even with a byte-identical head.
|
||
That is exactly what the Stage-3 acceptance measurement on the quantized build
|
||
catches — at 22 GB instead of 56, and after only ~20 min of quant spend rather
|
||
than a second seat's downtime. The shortcut trades a small, bounded risk for a
|
||
real saving; it does not skip the measurement.
|
||
|
||
---
|
||
|
||
## Stage 2 — quant
|
||
|
||
```bash
|
||
# 0. free, no GPU — prove the regexes hit the real module names
|
||
python3 validate_targets.py /tank/aimodels/qwen38-27b-heresy-bf16
|
||
# expect: OVERLAP 0, MLP layer union covers 0-63 True
|
||
|
||
# 1. ~20 min on one Blackwell. llmcompressor, NOT modelopt 0.43.
|
||
python3 quant_mixed_nvfp4.py \
|
||
--model /tank/aimodels/qwen38-27b-heresy-bf16 \
|
||
--calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||
--out /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed \
|
||
--num-samples 256 --seqlen 2048
|
||
|
||
# 2. MANDATORY — graft MTP, restore preprocessor, re-inject the mtp ignore
|
||
python3 post_quant.py /tank/aimodels/qwen38-27b-heresy-bf16 \
|
||
/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed
|
||
```
|
||
|
||
⚠ `llm-compressor` **prunes `ignore` entries that matched nothing at quant
|
||
time.** The wrapper class never loads the MTP head, so `re:^mtp.*` matches
|
||
nothing and is silently dropped — vLLM then treats the grafted BF16 head as
|
||
quantized, brings it up uninitialised, and speculative decoding runs at **0%
|
||
acceptance**. `post_quant.py` re-injects and *verifies*; it is not optional and
|
||
it has already fired on a real run.
|
||
|
||
---
|
||
|
||
## Stage 3 — validate
|
||
|
||
Playbook § 4.4 is explicit: **test on a temp port, never on the live seat.**
|
||
`bench/serve_probe.sh <model-dir>` serves a candidate on `:8017` under its own
|
||
served-name.
|
||
|
||
The full gate before any cutover:
|
||
|
||
| check | tool | bar |
|
||
|---|---|---|
|
||
| MTP acceptance + cache-busted bs=1 decode | `bench/quickbench.py` | ≳40% accept, ≥ incumbent 94.5 tok/s |
|
||
| prefill / TTFT on long prompts | `bench/prefill_bench.py` | ≥ incumbent (6,334 @6.7k, 5,085 @27k) |
|
||
| perplexity + abliteration survival | `bench/eval_quality.py` | ≤ ~+2% vs incumbent 7.059 |
|
||
| **surface**: chat, vision, tool calling, thinking split, 36K needle, streaming | `bench/surface_test.py` | **all six** |
|
||
| refusal rate on our own battery | `services/refusal-probe/` | ≤ incumbent; the card's 2/101 is unverified |
|
||
|
||
Measurement traps that have burned us — all documented, do not re-discover:
|
||
- **Bust the cache.** A fixed prompt + prefix caching returns byte-identical
|
||
timings; you measure nothing.
|
||
- **Never seed the prefill nonce.** A seeded nonce reproduces the prior run's
|
||
prompts verbatim → prefix cache serves them → you read ~41k tok/s of cache
|
||
hit instead of ~5k of real prefill. `SystemRandom`, always.
|
||
- **PPL must be measured with `--speculative-config` OFF.** Under MTP,
|
||
`prompt_logprobs` come back ~uniform over the vocab. The harness raises
|
||
rather than reporting garbage.
|
||
|
||
---
|
||
|
||
## Stage 4 — cutover
|
||
|
||
`gen-seat/.env`: point `GEN_MODEL` at the new dir, then
|
||
|
||
```bash
|
||
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
|
||
```
|
||
|
||
⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose`
|
||
call against it needs `sudo`, or compose fails reading `.env`, leaves the old
|
||
container running, and **the change silently does not take**.
|
||
|
||
⚠ Keep the pinned nightly. `GEN_IMAGE=vllm/vllm-openai:nightly-311b3513…`
|
||
carries the #51113 `qwen3_5_mtp`×GDN fix. Same architecture → same requirement.
|
||
|
||
⚠ Watch the GPU0 co-tenant. If the new weights come out smaller, the seat
|
||
absorbs the slack as extra KV and can starve `meromero-charrp` — that exact
|
||
failure crash-looped it once. `GEN_GPU_MEM_UTIL=0.43` is the current setting
|
||
that keeps both resident at 89.8 / 97.9 GB. Re-check after the swap.
|
||
|
||
**7 LiteLLM aliases ride this seat**: `gen`, `gen-reasoning`, `summarizer`,
|
||
`summarizer-large`, `classifier`, `image-judge`, `qwen-image-bench`. A bad
|
||
cutover degrades all of them at once.
|
||
|
||
### Rollback
|
||
|
||
Incumbent stays untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`.
|
||
|
||
```bash
|
||
sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-<date> \
|
||
/opt/docker/compose/gen-seat/.env
|
||
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
|
||
```
|
||
|
||
Take the `.env` backup **before** editing. Do not delete the incumbent weights
|
||
until the new seat has held through real multi-turn use — per the 2026-08-14
|
||
delete-too-early lesson, and the whole multi-day degeneration saga that
|
||
followed.
|
||
|
||
---
|
||
|
||
## GPU reality — why "in situ" is tempting and what to do instead
|
||
|
||
Both cards sit at **~96.8 / 97.9 GB**. GPU0 is gen (0.43) + meromero (0.52);
|
||
GPU1 is fablefusion + the utility cluster. **There is no room to run a probe
|
||
alongside the live seat** — which is why replacing in place looks like the only
|
||
option.
|
||
|
||
It isn't, quite. Downing `vllm-gen` frees its budget either way, so the
|
||
*disruption* is identical whether the candidate comes up on `:8017` or on the
|
||
live `:8015`. The difference is exposure: in situ points all 7 aliases at an
|
||
unvalidated RC the moment it starts. **Operator ruled 2026-08-17: use the probe
|
||
port.** Sequence — same downtime, much lower risk:
|
||
|
||
1. `compose down vllm-gen` (aliases are down either way for the window)
|
||
2. bring the candidate up via `serve_probe.sh` on `:8017`, its own served-name
|
||
3. run the Stage-3 gate
|
||
4. **pass** → flip `GEN_MODEL`, `up -d vllm-gen`, aliases return on validated
|
||
weights. **fail** → `up -d vllm-gen` unchanged, incumbent returns, nothing
|
||
downstream ever saw the candidate.
|
||
|
||
The only thing in-situ buys is skipping one container start. It costs the
|
||
ability to fail without consequence.
|