esh-pfi-infrastructure/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md

Runbook — swap the gen seat to absolute-heresy

Candidate: MuXodious/Qwen3.8-27B-absolute-heresy (Heretic v1.4.0 + SOMPOA, trial T377), pinned c23745939b34ca8b29ef707108572418ff031701.

Why it's a candidate: it beats the incumbent on both axes at once — refusals 2/101 vs JonathanColetti's 12/100, first-token KL 0.0759 vs 0.1191 — and it is a structurally clean full checkpoint (1199 tensors, 15 mtp.* present, 333 visual.*, lm_head), so the existing mixed recipe applies with no graft-and-reconstruct.

⚠ It is a self-described RC1, published 2026-08-15, ~348 downloads / 9 likes. Nothing here has been validated by anyone but its author, and its own card carries a visibly broken GGUF benchmark block (RC1 and RC2 report byte-identical piqa 0.4951 / copa 0.55 / winogrande 0.4957 — all at chance, identical across two different models = broken harness). Treat every number on that card as a claim, not a measurement.

---

Stage 0 — pull (DONE / in flight)

Detached container hf-pull-heresy on ana-ml2 → /tank/aimodels/qwen38-27b-heresy-bf16 (~55.6 GB, 12 shards). /tank had 4.4 TB free. Write the .PROVENANCE.txt on completion (template: the sibling qwen38-27b-uncensored-bf16.PROVENANCE.txt).

---

Stage 1 — the MTP gate, WITHOUT downing a second seat

Presence is not acceptance. The incumbent's own provenance still reads MTP acceptance UNVERIFIED — the author grafted 15 mtp.* verbatim and we shipped on the assumption. It measured 47.7% later; that was luck, not method.

The literal reading of the playbook rule (test MTP on bf16 FIRST) is expensive here: bf16 is ~56 GB resident, and on a full 97.9 GB card that means downing meromero-charrp as well as gen — freeing gen's 0.43 (~42 GB) alone is not enough. Two seats down to answer one question.

1a. Free, CPU-only: is the head one we have already measured?

The wrapper class Qwen3_5ForConditionalGeneration does not load the MTP head, so PEFT merges, Heretic runs, and llm-compressor passes all leave mtp.* exactly as it came from the base. If the candidate's 15 mtp.* tensors are numerically identical to a head we have measured in production, the acceptance question is already answered.

# runs in the vllm image (has torch + safetensors); reads only the mtp shard
sudo docker run --rm -v /tank/aimodels:/models --entrypoint python3 \
  vllm/vllm-openai:latest /models/compare_mtp_head.py \
  /models/qwen38-27b-heresy-bf16 /models/qwen38-27b-uncensored-bf16

Reference is the incumbent's model-mtp.safetensors — grafted verbatim from base per its PROVENANCE, and measured at 47.7% acceptance on the live seat through this exact pipeline. Known-good, not assumed.

• IDENTICAL → skip the bf16 gate entirely. Go to Stage 2 and verify acceptance on the quantized build (~22 GB) inside the freed gen budget. No second seat down.
• DIFFERENT → the head was edited. Not automatically bad — a deliberate mtp.* abliteration is legitimate (hotdogs/Qwen3.8-27B-abliterated edits 2 mtp tensors on purpose) — but it is no longer a head we have measured, so 1b becomes necessary.
• MISSING → dropped; needs a graft.

1b. Only if 1a says DIFFERENT: the real bf16 gate

Gate: acceptance ≳40%. Below that the mixed quant is not worth the GPU time — the whole +18% decode case rests on MTP working. Costs gen and meromero-charrp for the window.

Residual risk of taking the 1a shortcut, stated plainly: an identical head proves the head is intact, not that the surrounding model still drafts well with it. The MTP head reads hidden states from a body that abliteration did change, so acceptance could in principle move even with a byte-identical head. That is exactly what the Stage-3 acceptance measurement on the quantized build catches — at 22 GB instead of 56, and after only ~20 min of quant spend rather than a second seat's downtime. The shortcut trades a small, bounded risk for a real saving; it does not skip the measurement.

---

Stage 2 — quant

# 0. free, no GPU — prove the regexes hit the real module names
python3 validate_targets.py /tank/aimodels/qwen38-27b-heresy-bf16
#    expect: OVERLAP 0, MLP layer union covers 0-63 True

# 1. ~20 min on one Blackwell. llmcompressor, NOT modelopt 0.43.
python3 quant_mixed_nvfp4.py \
  --model /tank/aimodels/qwen38-27b-heresy-bf16 \
  --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
  --out   /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed \
  --num-samples 256 --seqlen 2048

# 2. MANDATORY — graft MTP, restore preprocessor, re-inject the mtp ignore
python3 post_quant.py /tank/aimodels/qwen38-27b-heresy-bf16 \
                      /tank/aimodels/qwen38-27b-heresy-nvfp4-mixed

⚠ llm-compressor prunes ignore entries that matched nothing at quant time. The wrapper class never loads the MTP head, so re:^mtp.* matches nothing and is silently dropped — vLLM then treats the grafted BF16 head as quantized, brings it up uninitialised, and speculative decoding runs at 0% acceptance. post_quant.py re-injects and verifies; it is not optional and it has already fired on a real run.

---

Stage 3 — validate

Playbook § 4.4 is explicit: test on a temp port, never on the live seat. bench/serve_probe.sh <model-dir> serves a candidate on :8017 under its own served-name.

The full gate before any cutover:

check	tool	bar
MTP acceptance + cache-busted bs=1 decode	bench/quickbench.py	≳40% accept, ≥ incumbent 94.5 tok/s
prefill / TTFT on long prompts	bench/prefill_bench.py	≥ incumbent (6,334 @6.7k, 5,085 @27k)
perplexity + abliteration survival	bench/eval_quality.py	≤ ~+2% vs incumbent 7.059
surface: chat, vision, tool calling, thinking split, 36K needle, streaming	bench/surface_test.py	all six
refusal rate on our own battery	services/refusal-probe/	≤ incumbent; the card's 2/101 is unverified

Measurement traps that have burned us — all documented, do not re-discover:

• Bust the cache. A fixed prompt + prefix caching returns byte-identical timings; you measure nothing.
• Never seed the prefill nonce. A seeded nonce reproduces the prior run's prompts verbatim → prefix cache serves them → you read ~41k tok/s of cache hit instead of ~5k of real prefill. SystemRandom, always.
• PPL must be measured with --speculative-config OFF. Under MTP, prompt_logprobs come back ~uniform over the vocab. The harness raises rather than reporting garbage.

---

Stage 4 — cutover

gen-seat/.env: point GEN_MODEL at the new dir, then

cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen

⚠ gen-seat/.env is mode 0600 and lkraven-owned — every docker compose call against it needs sudo, or compose fails reading .env, leaves the old container running, and the change silently does not take.

⚠ Keep the pinned nightly. GEN_IMAGE=vllm/vllm-openai:nightly-311b3513… carries the #51113 qwen3_5_mtp×GDN fix. Same architecture → same requirement.

⚠ Watch the GPU0 co-tenant. If the new weights come out smaller, the seat absorbs the slack as extra KV and can starve meromero-charrp — that exact failure crash-looped it once. GEN_GPU_MEM_UTIL=0.43 is the current setting that keeps both resident at 89.8 / 97.9 GB. Re-check after the swap.

7 LiteLLM aliases ride this seat: gen, gen-reasoning, summarizer, summarizer-large, classifier, image-judge, qwen-image-bench. A bad cutover degrades all of them at once.

Rollback

Incumbent stays untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed.

sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-<date> \
        /opt/docker/compose/gen-seat/.env
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen

Take the .env backup before editing. Do not delete the incumbent weights until the new seat has held through real multi-turn use — per the 2026-08-14 delete-too-early lesson, and the whole multi-day degeneration saga that followed.

---

GPU reality — why "in situ" is tempting and what to do instead

Both cards sit at ~96.8 / 97.9 GB. GPU0 is gen (0.43) + meromero (0.52); GPU1 is fablefusion + the utility cluster. There is no room to run a probe alongside the live seat — which is why replacing in place looks like the only option.

It isn't, quite. Downing vllm-gen frees its budget either way, so the disruption is identical whether the candidate comes up on :8017 or on the live :8015. The difference is exposure: in situ points all 7 aliases at an unvalidated RC the moment it starts. Operator ruled 2026-08-17: use the probe port. Sequence — same downtime, much lower risk:

1. compose down vllm-gen (aliases are down either way for the window)
2. bring the candidate up via serve_probe.sh on :8017, its own served-name
3. run the Stage-3 gate
4. pass → flip GEN_MODEL, up -d vllm-gen, aliases return on validated weights. fail → up -d vllm-gen unchanged, incumbent returns, nothing downstream ever saw the candidate.

The only thing in-situ buys is skipping one container start. It costs the ability to fail without consequence.