vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(gen-seat): CPU-only MTP head check so the gate costs no second seat

Browse Source 
Operator ruled the probe port for validation; runbook updated to match.

The bf16 MTP acceptance gate is ~56 GB resident, which on a full 97.9 GB card
means downing meromero-charrp as well as gen -- freeing gen's 0.43 (~42 GB)
alone is not enough. Two seats down to answer one question.

compare_mtp_head.py answers the common case for free. The
Qwen3_5ForConditionalGeneration wrapper never loads the MTP head, so PEFT
merges, Heretic runs, and llm-compressor passes all leave mtp.* as it came
from the base. It hashes a candidate's 15 mtp.* tensors against the
incumbent's grafted-verbatim head -- the one measured at 47.7% acceptance in
production through this exact pipeline. Identical means the acceptance
question is already answered; different means the head was edited and the
real gate is warranted; missing means it was dropped.

CPU only, reads just the shard holding mtp.*. The runbook states the residual
risk plainly: an identical head proves the head is intact, not that the
abliterated body still drafts well with it -- which the Stage-3 acceptance
measurement on the 22 GB quantized build catches anyway.
This commit is contained in:
Vuong Hoang
2026-08-17 16:25:28 -07:00
parent 7997f111b0
commit 254c588921
2 changed files with 191 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md
+50 -8
View File
@@ -27,17 +27,59 @@ Detached container `hf-pull-heresy` on ana-ml2 →
---
## Stage 1 — bf16 MTP acceptance gate (DO THIS BEFORE SPENDING A QUANT)
## Stage 1 — the MTP gate, WITHOUT downing a second seat
**Presence is not acceptance.** The incumbent's own provenance still reads
`MTP acceptance UNVERIFIED` — the author grafted 15 `mtp.*` verbatim and we
shipped on the assumption. It happened to measure 47.7% later; that was luck,
not method. Playbook: *test MTP on bf16 FIRST, isolate before deleting.*
shipped on the assumption. It measured 47.7% later; that was luck, not method.
Gate: **acceptance ≳40%**. Below that, the mixed quant is not worth the GPU
time — the whole +18% decode story rests on MTP working.
The literal reading of the playbook rule (*test MTP on bf16 FIRST*) is
**expensive here**: bf16 is ~56 GB resident, and on a full 97.9 GB card that
means downing `meromero-charrp` as well as `gen` — freeing gen's 0.43 (~42 GB)
alone is not enough. Two seats down to answer one question.
bf16 needs ~56 GB, so this needs a GPU window on its own (see § GPU reality).
### 1a. Free, CPU-only: is the head one we have already measured?
The wrapper class `Qwen3_5ForConditionalGeneration` **does not load the MTP
head**, so PEFT merges, Heretic runs, and llm-compressor passes all leave
`mtp.*` exactly as it came from the base. If the candidate's 15 `mtp.*` tensors
are numerically identical to a head we have measured in production, the
acceptance question is already answered.
```bash
# runs in the vllm image (has torch + safetensors); reads only the mtp shard
sudo docker run --rm -v /tank/aimodels:/models --entrypoint python3 \
vllm/vllm-openai:latest /models/compare_mtp_head.py \
/models/qwen38-27b-heresy-bf16 /models/qwen38-27b-uncensored-bf16
```
Reference is the incumbent's `model-mtp.safetensors` — grafted verbatim from
base per its PROVENANCE, and **measured at 47.7% acceptance** on the live seat
through this exact pipeline. Known-good, not assumed.
- **IDENTICAL** → skip the bf16 gate entirely. Go to Stage 2 and verify
acceptance on the **quantized** build (~22 GB) inside the freed `gen` budget.
No second seat down.
- **DIFFERENT** → the head was edited. Not automatically bad — a deliberate
`mtp.*` abliteration is legitimate (hotdogs/Qwen3.8-27B-abliterated edits 2
mtp tensors on purpose) — but it is no longer a head we have measured, so
1b becomes necessary.
- **MISSING** → dropped; needs a graft.
### 1b. Only if 1a says DIFFERENT: the real bf16 gate
Gate: **acceptance ≳40%**. Below that the mixed quant is not worth the GPU time
— the whole +18% decode case rests on MTP working. Costs `gen` **and**
`meromero-charrp` for the window.
**Residual risk of taking the 1a shortcut, stated plainly:** an identical head
proves the *head* is intact, not that the surrounding model still drafts well
with it. The MTP head reads hidden states from a body that abliteration *did*
change, so acceptance could in principle move even with a byte-identical head.
That is exactly what the Stage-3 acceptance measurement on the quantized build
catches — at 22 GB instead of 56, and after only ~20 min of quant spend rather
than a second seat's downtime. The shortcut trades a small, bounded risk for a
real saving; it does not skip the measurement.
---
@@ -148,8 +190,8 @@ option.
It isn't, quite. Downing `vllm-gen` frees its budget either way, so the
*disruption* is identical whether the candidate comes up on `:8017` or on the
live `:8015`. The difference is exposure: in situ points all 7 aliases at an
unvalidated RC the moment it starts. **Recommended sequence — same downtime,
much lower risk:**
unvalidated RC the moment it starts. **Operator ruled 2026-08-17: use the probe
port.** Sequence — same downtime, much lower risk:
1. `compose down vllm-gen` (aliases are down either way for the window)
2. bring the candidate up via `serve_probe.sh` on `:8017`, its own served-name
services/gen-seat-mixed-quant/compare_mtp_head.py
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""Compare a candidate checkpoint's MTP head against a known-good reference head.
WHY THIS EXISTS
---------------
The expensive gate before quantizing a new Qwen3.8-27B candidate is "does its MTP
head actually work?" — presence is not acceptance, and a dead head silently costs
the entire +18% decode case. Measuring acceptance for real needs the model resident
in VRAM; at bf16 that is ~56 GB, which on a full 97.9 GB card means downing a
SECOND seat, not just `gen`.
This sidesteps that for the common case. Most community abliterations never touch
`mtp.*` at all: the `Qwen3_5ForConditionalGeneration` wrapper class does not load
the MTP head, so PEFT merges, Heretic runs, and llm-compressor passes all leave it
exactly as it came from the base. If a candidate's 15 `mtp.*` tensors are
numerically identical to a head we have already measured in production, its MTP is
that head — and the acceptance test is redundant.
Reference to compare against: `/tank/aimodels/qwen38-27b-uncensored-bf16`
(`model-mtp.safetensors`). Per its PROVENANCE that head was grafted verbatim from
the base, and it measures **47.7% acceptance** on the live gen seat through our
exact mixed-quant pipeline. That makes it a known-good baseline, not a guess.
READ THE RESULT HONESTLY
------------------------
- **IDENTICAL** -> the candidate carries the pristine base head. The bf16 acceptance
gate buys nothing; go to quant and verify acceptance on the quantized build inside
the freed `gen` budget (~22 GB, no second seat).
- **DIFFERENT** -> something edited the head. That is NOT automatically bad (an
abliteration that deliberately includes `mtp.*` is a legitimate design — see
hotdogs/Qwen3.8-27B-abliterated, which edits 2 mtp tensors on purpose), but it
means the head is no longer one we have measured. Run the real acceptance gate.
- **MISSING** -> the head was dropped. Known failure mode; needs a graft.
CPU only. Reads just the shard(s) holding `mtp.*` — no full model load.
"""
import argparse
import hashlib
import json
import sys
from pathlib import Path
from safetensors import safe_open
def load_mtp(model_dir: Path) -> dict:
"""Return {tensor_name: tensor} for every mtp.* tensor, reading only the
shards that actually hold them."""
index = model_dir / "model.safetensors.index.json"
if index.exists():
weight_map = json.loads(index.read_text())["weight_map"]
names = [k for k in weight_map if k.startswith("mtp")]
shards = sorted({weight_map[n] for n in names})
else:
# unsharded checkpoint
shards = ["model.safetensors"]
names = None
out = {}
for shard in shards:
path = model_dir / shard
if not path.exists():
raise SystemExit(f"missing shard: {path}")
with safe_open(path, framework="pt") as f:
for key in f.keys():
if key.startswith("mtp"):
out[key] = f.get_tensor(key)
if names is not None and len(out) != len(names):
print(f" warning: index listed {len(names)} mtp tensors, read {len(out)}",
file=sys.stderr)
return out
def digest(tensor) -> str:
"""SHA-256 over the raw tensor bytes. Dtype-sensitive by design — a head
stored at a different precision is not the same head for our purposes."""
return hashlib.sha256(tensor.contiguous().view(-1).numpy().tobytes()).hexdigest()
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("candidate", type=Path, help="candidate model dir")
ap.add_argument("reference", type=Path,
help="known-good reference model dir (e.g. qwen38-27b-uncensored-bf16)")
args = ap.parse_args()
cand = load_mtp(args.candidate)
ref = load_mtp(args.reference)
print(f"candidate : {args.candidate} ({len(cand)} mtp tensors)")
print(f"reference : {args.reference} ({len(ref)} mtp tensors)")
print()
if not cand:
print("VERDICT: MISSING — candidate has no mtp.* tensors. Needs a graft.")
return 2
only_cand = sorted(set(cand) - set(ref))
only_ref = sorted(set(ref) - set(cand))
if only_cand or only_ref:
print(" tensor-name mismatch:")
for n in only_cand:
print(f" only in candidate: {n}")
for n in only_ref:
print(f" only in reference: {n}")
print()
differing = []
for name in sorted(set(cand) & set(ref)):
c, r = cand[name], ref[name]
if c.dtype != r.dtype or c.shape != r.shape:
differing.append((name, f"dtype/shape {c.dtype}{tuple(c.shape)} vs "
f"{r.dtype}{tuple(r.shape)}"))
continue
if digest(c) != digest(r):
# quantify it — a tiny delta is a different story from a rewritten head
delta = (c.float() - r.float()).abs().max().item()
denom = r.float().abs().max().item() or 1.0
differing.append((name, f"max|Δ| = {delta:.6g} (rel {delta / denom:.3%})"))
for name, why in differing:
print(f" DIFFERS {name}: {why}")
print()
if not differing and not only_cand and not only_ref:
print("VERDICT: IDENTICAL — candidate carries the reference MTP head verbatim.")
print(" The bf16 acceptance gate is redundant: this head is already measured")
print(" at 47.7% acceptance in production through our mixed-quant pipeline.")
print(" Proceed to quant; verify acceptance on the quantized build.")
return 0
print(f"VERDICT: DIFFERENT — {len(differing)} of {len(cand)} mtp tensors diverge.")
print(" The head is not one we have measured. Run the real bf16 acceptance gate")
print(" (~56 GB resident) before spending quant GPU time.")
return 1
if __name__ == "__main__":
raise SystemExit(main())