vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
110 lines
4.4 KiB
Python
110 lines
4.4 KiB
Python
"""Playbook §4.3 post-steps for a Gemma-4 NVFP4 output tree.
|
|
|
|
Steps 1 and 3 (MTP graft, `re:^mtp.*` re-injection) are N/A on Gemma-4 -- it ships
|
|
no MTP head at all, verified as 0 mtp tensors in both bf16 sources. That leaves:
|
|
|
|
step 2 restore processor_config.json + preprocessor_config.json
|
|
step 4 confirm the saved tokenizer.json has truncation: null
|
|
|
|
Step 4 is not a formality here. The A4B output was quantized WITH the calibration
|
|
dataset, and build_calib calls the fast tokenizer with truncation=True,
|
|
max_length=8192 -- which mutates the Rust backend in place, and save_pretrained
|
|
then bakes the cap into the shipped tokenizer.json. It is latent on the
|
|
transformers that wrote it and fatal on a newer one. The fix edits the one
|
|
`truncation` key rather than copying the source file wholesale, so nothing else in
|
|
a 32 MB tokenizer can quietly change underneath it.
|
|
|
|
Derivation of preprocessor_config.json is `processor_config.json["image_processor"]`
|
|
verbatim; that reproduces the 2026-08-21 known-good output byte for byte.
|
|
|
|
Idempotent, and reports per step whether it CHANGED or was already correct.
|
|
Run with --check to verify without writing.
|
|
"""
|
|
import argparse
|
|
import json
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--src", required=True, help="bf16 source tree")
|
|
ap.add_argument("--out", required=True, help="quantized output tree")
|
|
ap.add_argument("--check", action="store_true", help="report only, write nothing")
|
|
a = ap.parse_args()
|
|
src, out = Path(a.src), Path(a.out)
|
|
mode = "CHECK" if a.check else "APPLY"
|
|
print(f"[{mode}] src={src}\n[{mode}] out={out}\n")
|
|
|
|
rc = 0
|
|
|
|
|
|
def step(n, desc):
|
|
print(f"-- step {n}: {desc}")
|
|
|
|
|
|
step(1, "MTP graft")
|
|
mtp = [k for k in json.loads((out / "config.json").read_text()).get(
|
|
"quantization_config", {}).get("ignore", []) if "mtp" in k.lower()]
|
|
idx = out / "model.safetensors.index.json"
|
|
tensors = json.loads(idx.read_text())["weight_map"] if idx.exists() else {}
|
|
n_mtp = sum(1 for k in tensors if k.startswith("mtp"))
|
|
print(f" N/A for Gemma-4 (no MTP head). mtp tensors in output index: {n_mtp}; "
|
|
f"mtp entries in ignore list: {len(mtp)}")
|
|
if n_mtp:
|
|
print(" *** unexpected mtp tensors -- step 3 would become live, investigate")
|
|
rc = 1
|
|
|
|
step(2, "restore processor_config.json + preprocessor_config.json")
|
|
spc = src / "processor_config.json"
|
|
if not spc.exists():
|
|
print(f" *** source has no processor_config.json -- cannot restore")
|
|
rc = 1
|
|
else:
|
|
opc = out / "processor_config.json"
|
|
if opc.exists() and opc.read_bytes() == spc.read_bytes():
|
|
print(" processor_config.json already present and identical to source")
|
|
elif a.check:
|
|
print(f" processor_config.json MISSING/differs -> would copy from source")
|
|
else:
|
|
shutil.copy2(spc, opc)
|
|
print(" processor_config.json CHANGED (copied from source)")
|
|
|
|
want = json.dumps(dict(json.loads(spc.read_text())["image_processor"]), indent=1)
|
|
opre = out / "preprocessor_config.json"
|
|
if opre.exists() and opre.read_text() == want:
|
|
print(" preprocessor_config.json already present and correct")
|
|
elif a.check:
|
|
print(" preprocessor_config.json MISSING/differs -> would derive from image_processor")
|
|
else:
|
|
opre.write_text(want)
|
|
print(" preprocessor_config.json CHANGED (derived from processor_config"
|
|
"['image_processor'])")
|
|
|
|
step(4, "confirm saved tokenizer.json has truncation: null")
|
|
tj = out / "tokenizer.json"
|
|
tok = json.loads(tj.read_text())
|
|
trunc = tok.get("truncation")
|
|
if trunc is None:
|
|
print(" truncation is null -- clean")
|
|
else:
|
|
print(f" *** truncation BAKED IN: {trunc}")
|
|
stok = json.loads((src / "tokenizer.json").read_text())
|
|
others = [k for k in set(tok) | set(stok)
|
|
if k != "truncation" and tok.get(k) != stok.get(k)]
|
|
print(f" other top-level keys differing from source: {others or 'none'}")
|
|
if a.check:
|
|
print(" would set truncation -> null")
|
|
rc = 1
|
|
else:
|
|
bak = tj.with_name("tokenizer.json.bak-pre-truncfix")
|
|
if not bak.exists():
|
|
shutil.copy2(tj, bak)
|
|
print(f" backed up -> {bak.name}")
|
|
tok["truncation"] = None
|
|
tmp = tj.with_suffix(".json.tmp")
|
|
tmp.write_text(json.dumps(tok, ensure_ascii=False, indent=2))
|
|
tmp.replace(tj)
|
|
print(" truncation CHANGED -> null")
|
|
|
|
print(f"\n[{mode}] done rc={rc}")
|
|
raise SystemExit(rc)
|