diff --git a/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md b/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md index e8fb0df..eb72dc4 100644 --- a/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md +++ b/services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md @@ -27,17 +27,59 @@ Detached container `hf-pull-heresy` on ana-ml2 → --- -## Stage 1 — bf16 MTP acceptance gate (DO THIS BEFORE SPENDING A QUANT) +## Stage 1 — the MTP gate, WITHOUT downing a second seat **Presence is not acceptance.** The incumbent's own provenance still reads `MTP acceptance UNVERIFIED` — the author grafted 15 `mtp.*` verbatim and we -shipped on the assumption. It happened to measure 47.7% later; that was luck, -not method. Playbook: *test MTP on bf16 FIRST, isolate before deleting.* +shipped on the assumption. It measured 47.7% later; that was luck, not method. -Gate: **acceptance ≳40%**. Below that, the mixed quant is not worth the GPU -time — the whole +18% decode story rests on MTP working. +The literal reading of the playbook rule (*test MTP on bf16 FIRST*) is +**expensive here**: bf16 is ~56 GB resident, and on a full 97.9 GB card that +means downing `meromero-charrp` as well as `gen` — freeing gen's 0.43 (~42 GB) +alone is not enough. Two seats down to answer one question. -bf16 needs ~56 GB, so this needs a GPU window on its own (see § GPU reality). +### 1a. Free, CPU-only: is the head one we have already measured? + +The wrapper class `Qwen3_5ForConditionalGeneration` **does not load the MTP +head**, so PEFT merges, Heretic runs, and llm-compressor passes all leave +`mtp.*` exactly as it came from the base. If the candidate's 15 `mtp.*` tensors +are numerically identical to a head we have measured in production, the +acceptance question is already answered. + +```bash +# runs in the vllm image (has torch + safetensors); reads only the mtp shard +sudo docker run --rm -v /tank/aimodels:/models --entrypoint python3 \ + vllm/vllm-openai:latest /models/compare_mtp_head.py \ + /models/qwen38-27b-heresy-bf16 /models/qwen38-27b-uncensored-bf16 +``` + +Reference is the incumbent's `model-mtp.safetensors` — grafted verbatim from +base per its PROVENANCE, and **measured at 47.7% acceptance** on the live seat +through this exact pipeline. Known-good, not assumed. + +- **IDENTICAL** → skip the bf16 gate entirely. Go to Stage 2 and verify + acceptance on the **quantized** build (~22 GB) inside the freed `gen` budget. + No second seat down. +- **DIFFERENT** → the head was edited. Not automatically bad — a deliberate + `mtp.*` abliteration is legitimate (hotdogs/Qwen3.8-27B-abliterated edits 2 + mtp tensors on purpose) — but it is no longer a head we have measured, so + 1b becomes necessary. +- **MISSING** → dropped; needs a graft. + +### 1b. Only if 1a says DIFFERENT: the real bf16 gate + +Gate: **acceptance ≳40%**. Below that the mixed quant is not worth the GPU time +— the whole +18% decode case rests on MTP working. Costs `gen` **and** +`meromero-charrp` for the window. + +**Residual risk of taking the 1a shortcut, stated plainly:** an identical head +proves the *head* is intact, not that the surrounding model still drafts well +with it. The MTP head reads hidden states from a body that abliteration *did* +change, so acceptance could in principle move even with a byte-identical head. +That is exactly what the Stage-3 acceptance measurement on the quantized build +catches — at 22 GB instead of 56, and after only ~20 min of quant spend rather +than a second seat's downtime. The shortcut trades a small, bounded risk for a +real saving; it does not skip the measurement. --- @@ -148,8 +190,8 @@ option. It isn't, quite. Downing `vllm-gen` frees its budget either way, so the *disruption* is identical whether the candidate comes up on `:8017` or on the live `:8015`. The difference is exposure: in situ points all 7 aliases at an -unvalidated RC the moment it starts. **Recommended sequence — same downtime, -much lower risk:** +unvalidated RC the moment it starts. **Operator ruled 2026-08-17: use the probe +port.** Sequence — same downtime, much lower risk: 1. `compose down vllm-gen` (aliases are down either way for the window) 2. bring the candidate up via `serve_probe.sh` on `:8017`, its own served-name diff --git a/services/gen-seat-mixed-quant/compare_mtp_head.py b/services/gen-seat-mixed-quant/compare_mtp_head.py new file mode 100644 index 0000000..788e6a4 --- /dev/null +++ b/services/gen-seat-mixed-quant/compare_mtp_head.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +"""Compare a candidate checkpoint's MTP head against a known-good reference head. + +WHY THIS EXISTS +--------------- +The expensive gate before quantizing a new Qwen3.8-27B candidate is "does its MTP +head actually work?" — presence is not acceptance, and a dead head silently costs +the entire +18% decode case. Measuring acceptance for real needs the model resident +in VRAM; at bf16 that is ~56 GB, which on a full 97.9 GB card means downing a +SECOND seat, not just `gen`. + +This sidesteps that for the common case. Most community abliterations never touch +`mtp.*` at all: the `Qwen3_5ForConditionalGeneration` wrapper class does not load +the MTP head, so PEFT merges, Heretic runs, and llm-compressor passes all leave it +exactly as it came from the base. If a candidate's 15 `mtp.*` tensors are +numerically identical to a head we have already measured in production, its MTP is +that head — and the acceptance test is redundant. + +Reference to compare against: `/tank/aimodels/qwen38-27b-uncensored-bf16` +(`model-mtp.safetensors`). Per its PROVENANCE that head was grafted verbatim from +the base, and it measures **47.7% acceptance** on the live gen seat through our +exact mixed-quant pipeline. That makes it a known-good baseline, not a guess. + +READ THE RESULT HONESTLY +------------------------ +- **IDENTICAL** -> the candidate carries the pristine base head. The bf16 acceptance + gate buys nothing; go to quant and verify acceptance on the quantized build inside + the freed `gen` budget (~22 GB, no second seat). +- **DIFFERENT** -> something edited the head. That is NOT automatically bad (an + abliteration that deliberately includes `mtp.*` is a legitimate design — see + hotdogs/Qwen3.8-27B-abliterated, which edits 2 mtp tensors on purpose), but it + means the head is no longer one we have measured. Run the real acceptance gate. +- **MISSING** -> the head was dropped. Known failure mode; needs a graft. + +CPU only. Reads just the shard(s) holding `mtp.*` — no full model load. +""" + +import argparse +import hashlib +import json +import sys +from pathlib import Path + +from safetensors import safe_open + + +def load_mtp(model_dir: Path) -> dict: + """Return {tensor_name: tensor} for every mtp.* tensor, reading only the + shards that actually hold them.""" + index = model_dir / "model.safetensors.index.json" + if index.exists(): + weight_map = json.loads(index.read_text())["weight_map"] + names = [k for k in weight_map if k.startswith("mtp")] + shards = sorted({weight_map[n] for n in names}) + else: + # unsharded checkpoint + shards = ["model.safetensors"] + names = None + + out = {} + for shard in shards: + path = model_dir / shard + if not path.exists(): + raise SystemExit(f"missing shard: {path}") + with safe_open(path, framework="pt") as f: + for key in f.keys(): + if key.startswith("mtp"): + out[key] = f.get_tensor(key) + if names is not None and len(out) != len(names): + print(f" warning: index listed {len(names)} mtp tensors, read {len(out)}", + file=sys.stderr) + return out + + +def digest(tensor) -> str: + """SHA-256 over the raw tensor bytes. Dtype-sensitive by design — a head + stored at a different precision is not the same head for our purposes.""" + return hashlib.sha256(tensor.contiguous().view(-1).numpy().tobytes()).hexdigest() + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("candidate", type=Path, help="candidate model dir") + ap.add_argument("reference", type=Path, + help="known-good reference model dir (e.g. qwen38-27b-uncensored-bf16)") + args = ap.parse_args() + + cand = load_mtp(args.candidate) + ref = load_mtp(args.reference) + + print(f"candidate : {args.candidate} ({len(cand)} mtp tensors)") + print(f"reference : {args.reference} ({len(ref)} mtp tensors)") + print() + + if not cand: + print("VERDICT: MISSING — candidate has no mtp.* tensors. Needs a graft.") + return 2 + + only_cand = sorted(set(cand) - set(ref)) + only_ref = sorted(set(ref) - set(cand)) + if only_cand or only_ref: + print(" tensor-name mismatch:") + for n in only_cand: + print(f" only in candidate: {n}") + for n in only_ref: + print(f" only in reference: {n}") + print() + + differing = [] + for name in sorted(set(cand) & set(ref)): + c, r = cand[name], ref[name] + if c.dtype != r.dtype or c.shape != r.shape: + differing.append((name, f"dtype/shape {c.dtype}{tuple(c.shape)} vs " + f"{r.dtype}{tuple(r.shape)}")) + continue + if digest(c) != digest(r): + # quantify it — a tiny delta is a different story from a rewritten head + delta = (c.float() - r.float()).abs().max().item() + denom = r.float().abs().max().item() or 1.0 + differing.append((name, f"max|Δ| = {delta:.6g} (rel {delta / denom:.3%})")) + + for name, why in differing: + print(f" DIFFERS {name}: {why}") + + print() + if not differing and not only_cand and not only_ref: + print("VERDICT: IDENTICAL — candidate carries the reference MTP head verbatim.") + print(" The bf16 acceptance gate is redundant: this head is already measured") + print(" at 47.7% acceptance in production through our mixed-quant pipeline.") + print(" Proceed to quant; verify acceptance on the quantized build.") + return 0 + + print(f"VERDICT: DIFFERENT — {len(differing)} of {len(cand)} mtp tensors diverge.") + print(" The head is not one we have measured. Run the real bf16 acceptance gate") + print(" (~56 GB resident) before spending quant GPU time.") + return 1 + + +if __name__ == "__main__": + raise SystemExit(main())