# Runbook — swap the `gen` seat to `absolute-heresy`

Candidate: **`MuXodious/Qwen3.8-27B-absolute-heresy`** (Heretic v1.4.0 + SOMPOA,
trial T377), pinned `c23745939b34ca8b29ef707108572418ff031701`.

Why it's a candidate: it beats the incumbent on **both** axes at once —
refusals **2/101** vs JonathanColetti's 12/100, first-token KL **0.0759** vs
0.1191 — and it is a structurally clean full checkpoint (1199 tensors, **15
`mtp.*` present**, 333 `visual.*`, `lm_head`), so the existing mixed recipe
applies with no graft-and-reconstruct.

⚠ It is a self-described **RC1**, published 2026-08-15, ~348 downloads / 9
likes. Nothing here has been validated by anyone but its author, and its own
card carries a visibly broken GGUF benchmark block (RC1 and RC2 report
byte-identical piqa 0.4951 / copa 0.55 / winogrande 0.4957 — all at chance,
identical across two different models = broken harness). Treat every number on
that card as a claim, not a measurement.

---

## Stage 0 — pull (DONE / in flight)

Detached container `hf-pull-heresy` on ana-ml2 →
`/tank/aimodels/qwen38-27b-heresy-bf16` (~55.6 GB, 12 shards). `/tank` had
4.4 TB free. Write the `.PROVENANCE.txt` on completion (template: the sibling
`qwen38-27b-uncensored-bf16.PROVENANCE.txt`).

---

## Stage 1 — the MTP gate, WITHOUT downing a second seat

**Presence is not acceptance.** The incumbent's own provenance still reads
`MTP acceptance UNVERIFIED` — the author grafted 15 `mtp.*` verbatim and we
shipped on the assumption. It measured 47.7% later; that was luck, not method.

The literal reading of the playbook rule (*test MTP on bf16 FIRST*) is
**expensive here**: bf16 is ~56 GB resident, and on a full 97.9 GB card that
means downing `meromero-charrp` as well as `gen` — freeing gen's 0.43 (~42 GB)
alone is not enough. Two seats down to answer one question.

### 1a. Free, CPU-only: is the head one we have already measured?

The wrapper class `Qwen3_5ForConditionalGeneration` **does not load the MTP
head**, so PEFT merges, Heretic runs, and llm-compressor passes all leave
`mtp.*` exactly as it came from the base. If the candidate's 15 `mtp.*` tensors
are numerically identical to a head we have measured in production, the
acceptance question is already answered.

```bash
# runs in the vllm image (has torch + safetensors); reads only the mtp shard
sudo docker run --rm -v /tank/aimodels:/models --entrypoint python3 \
  vllm/vllm-openai:latest /models/compare_mtp_head.py \
  /models/qwen38-27b-heresy-bf16 /models/qwen38-27b-uncensored-bf16
```

Reference is the incumbent's `model-mtp.safetensors` — grafted verbatim from
base per its PROVENANCE, and **measured at 47.7% acceptance** on the live seat
through this exact pipeline. Known-good, not assumed.

- **IDENTICAL** → skip the bf16 gate entirely. Go to Stage 2 and verify
  acceptance on the **quantized** build (~22 GB) inside the freed `gen` budget.
  No second seat down.
- **DIFFERENT** → the head was edited. Not automatically bad — a deliberate
  `mtp.*` abliteration is legitimate (hotdogs/Qwen3.8-27B-abliterated edits 2
  mtp tensors on purpose) — but it is no longer a head we have measured, so
  1b becomes necessary.
- **MISSING** → dropped; needs a graft.

### 1b. Only if 1a says DIFFERENT: the real bf16 gate

Gate: **acceptance ≳40%**. Below that the mixed quant is not worth the GPU time
— the whole +18% decode case rests on MTP working. Costs `gen` **and**
`meromero-charrp` for the window.

**Residual risk of taking the 1a shortcut, stated plainly:** an identical head
proves the *head* is intact, not that the surrounding model still drafts well
with it. The MTP head reads hidden states from a body that abliteration *did*
change, so acceptance could in principle move even with a byte-identical head.
That is exactly what the Stage-3 acceptance measurement on the quantized build
catches — at 22 GB instead of 56, and after only ~20 min of quant spend rather
than a second seat's downtime. The shortcut trades a small, bounded risk for a
real saving; it does not skip the measurement.

---

## Stage 2 — quant

```bash
# 0. free, no GPU — prove the regexes hit the real module names
python3 validate_targets.py /tank/aimodels/qwen38-27b-heresy-bf16
#    expect: OVERLAP 0, MLP layer union covers 0-63 True

# 1. ~20 min on one Blackwell. llmcompressor, NOT modelopt 0.43.
python3 quant_mixed_nvfp4.py \
  --model /tank/aimodels/qwen38-27b-heresy-bf16 \
  --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
  --out   /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed \
  --num-samples 256 --seqlen 2048

# 2. MANDATORY — graft MTP, restore preprocessor, re-inject the mtp ignore
python3 post_quant.py /tank/aimodels/qwen38-27b-heresy-bf16 \
                      /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed
```

⚠ `llm-compressor` **prunes `ignore` entries that matched nothing at quant
time.** The wrapper class never loads the MTP head, so `re:^mtp.*` matches
nothing and is silently dropped — vLLM then treats the grafted BF16 head as
quantized, brings it up uninitialised, and speculative decoding runs at **0%
acceptance**. `post_quant.py` re-injects and *verifies*; it is not optional and
it has already fired on a real run.

---

## Stage 3 — validate

Playbook § 4.4 is explicit: **test on a temp port, never on the live seat.**
`bench/serve_probe.sh <model-dir>` serves a candidate on `:8017` under its own
served-name.

The full gate before any cutover:

| check | tool | bar |
|---|---|---|
| MTP acceptance + cache-busted bs=1 decode | `bench/quickbench.py` | ≳40% accept, ≥ incumbent 94.5 tok/s |
| prefill / TTFT on long prompts | `bench/prefill_bench.py` | ≥ incumbent (6,334 @6.7k, 5,085 @27k) |
| perplexity + abliteration survival | `bench/eval_quality.py` | ≤ ~+2% vs incumbent 7.059 |
| **surface**: chat, vision, tool calling, thinking split, 36K needle, streaming | `bench/surface_test.py` | **all six** |
| refusal rate on our own battery | `services/refusal-probe/` | ≤ incumbent; the card's 2/101 is unverified |

Measurement traps that have burned us — all documented, do not re-discover:
- **Bust the cache.** A fixed prompt + prefix caching returns byte-identical
  timings; you measure nothing.
- **Never seed the prefill nonce.** A seeded nonce reproduces the prior run's
  prompts verbatim → prefix cache serves them → you read ~41k tok/s of cache
  hit instead of ~5k of real prefill. `SystemRandom`, always.
- **PPL must be measured with `--speculative-config` OFF.** Under MTP,
  `prompt_logprobs` come back ~uniform over the vocab. The harness raises
  rather than reporting garbage.

---

## Stage 4 — cutover

`gen-seat/.env`: point `GEN_MODEL` at the new dir, then

```bash
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
```

⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose`
call against it needs `sudo`, or compose fails reading `.env`, leaves the old
container running, and **the change silently does not take**.

⚠ Keep the pinned nightly. `GEN_IMAGE=vllm/vllm-openai:nightly-311b3513…`
carries the #51113 `qwen3_5_mtp`×GDN fix. Same architecture → same requirement.

⚠ Watch the GPU0 co-tenant. If the new weights come out smaller, the seat
absorbs the slack as extra KV and can starve `meromero-charrp` — that exact
failure crash-looped it once. `GEN_GPU_MEM_UTIL=0.43` is the current setting
that keeps both resident at 89.8 / 97.9 GB. Re-check after the swap.

**7 LiteLLM aliases ride this seat**: `gen`, `gen-reasoning`, `summarizer`,
`summarizer-large`, `classifier`, `image-judge`, `qwen-image-bench`. A bad
cutover degrades all of them at once.

### Rollback

Incumbent stays untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`.

```bash
sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-<date> \
        /opt/docker/compose/gen-seat/.env
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
```

Take the `.env` backup **before** editing. Do not delete the incumbent weights
until the new seat has held through real multi-turn use — per the 2026-08-14
delete-too-early lesson, and the whole multi-day degeneration saga that
followed.

---

## GPU reality — why "in situ" is tempting and what to do instead

Both cards sit at **~96.8 / 97.9 GB**. GPU0 is gen (0.43) + meromero (0.52);
GPU1 is fablefusion + the utility cluster. **There is no room to run a probe
alongside the live seat** — which is why replacing in place looks like the only
option.

It isn't, quite. Downing `vllm-gen` frees its budget either way, so the
*disruption* is identical whether the candidate comes up on `:8017` or on the
live `:8015`. The difference is exposure: in situ points all 7 aliases at an
unvalidated RC the moment it starts. **Operator ruled 2026-08-17: use the probe
port.** Sequence — same downtime, much lower risk:

1. `compose down vllm-gen` (aliases are down either way for the window)
2. bring the candidate up via `serve_probe.sh` on `:8017`, its own served-name
3. run the Stage-3 gate
4. **pass** → flip `GEN_MODEL`, `up -d vllm-gen`, aliases return on validated
   weights. **fail** → `up -d vllm-gen` unchanged, incumbent returns, nothing
   downstream ever saw the candidate.

The only thing in-situ buys is skipping one container start. It costs the
ability to fail without consequence.