vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
93 lines
3.7 KiB
Python
93 lines
3.7 KiB
Python
"""Did the quant actually do what the recipe says, on the tensors it claims?
|
|
|
|
`rc=0` and a plausible file size prove neither. The two things the operator asked
|
|
for -- W4A16, vision towers intact -- are properties of the tensor table, so read
|
|
the tensor table. Parses safetensors headers directly (u64 length + JSON), so no
|
|
torch, no GPU, and no 20 GB load.
|
|
|
|
Checks, per module family:
|
|
* language-model Linears -> must be NVFP4-packed (uint8 blobs + *_scale companions)
|
|
* vision / audio towers -> must still be BF16, i.e. PRESERVED not dropped
|
|
* embeddings / lm_head / norms -> BF16 per the ignore list
|
|
|
|
Run it against a known-good tree as well. A checker that has only ever seen the
|
|
tree it was written for cannot tell "correct" from "blind".
|
|
"""
|
|
import argparse
|
|
import json
|
|
import struct
|
|
from collections import defaultdict
|
|
from pathlib import Path
|
|
|
|
|
|
def tensors(root: Path):
|
|
"""Yield (name, dtype, shape) for every tensor in a sharded or single-file tree."""
|
|
idx = root / "model.safetensors.index.json"
|
|
files = sorted({Path(v) for v in json.loads(idx.read_text())["weight_map"].values()}) \
|
|
if idx.exists() else [Path("model.safetensors")]
|
|
for f in files:
|
|
p = root / f
|
|
with p.open("rb") as fh:
|
|
n = struct.unpack("<Q", fh.read(8))[0]
|
|
head = json.loads(fh.read(n))
|
|
for name, meta in head.items():
|
|
if name == "__metadata__":
|
|
continue
|
|
yield name, meta["dtype"], meta["shape"]
|
|
|
|
|
|
def family(name: str) -> str:
|
|
if "vision_tower" in name or "embed_vision" in name:
|
|
return "vision_tower"
|
|
if "audio_tower" in name or "embed_audio" in name:
|
|
return "audio_tower"
|
|
if "multi_modal_projector" in name or "mm_projector" in name:
|
|
return "projector"
|
|
if "embed_tokens" in name:
|
|
return "embeddings"
|
|
if name.startswith("lm_head") or ".lm_head" in name:
|
|
return "lm_head"
|
|
if "norm" in name:
|
|
return "norms"
|
|
if "language_model" in name or ".layers." in name:
|
|
return "language_model"
|
|
return "other"
|
|
|
|
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("trees", nargs="+")
|
|
a = ap.parse_args()
|
|
|
|
for t in a.trees:
|
|
root = Path(t)
|
|
print(f"\n{'='*70}\n{root}")
|
|
cfg = json.loads((root / "config.json").read_text())
|
|
q = cfg.get("quantization_config", {})
|
|
groups = q.get("config_groups", {})
|
|
for gname, g in groups.items():
|
|
w = g.get("weights", {})
|
|
i = g.get("input_activations")
|
|
print(f" {gname}: weights num_bits={w.get('num_bits')} type={w.get('type')} "
|
|
f"strategy={w.get('strategy')} | input_activations="
|
|
f"{'None (WEIGHT-ONLY)' if i is None else i}")
|
|
print(f" format={q.get('format')} kv_cache_scheme={q.get('kv_cache_scheme')} "
|
|
f"status={q.get('quantization_status')}")
|
|
tc = cfg.get("text_config", {})
|
|
print(f" text_config: per_layer_config={'PRESENT' if 'per_layer_config' in tc else 'absent'}"
|
|
f" head_dim={tc.get('head_dim')} global_head_dim={tc.get('global_head_dim')}"
|
|
f" num_key_value_heads={tc.get('num_key_value_heads')}"
|
|
f" num_global_key_value_heads={tc.get('num_global_key_value_heads')}")
|
|
|
|
by = defaultdict(lambda: defaultdict(int))
|
|
packed = defaultdict(int)
|
|
for name, dt, shape in tensors(root):
|
|
f = family(name)
|
|
by[f][dt] += 1
|
|
if name.endswith("weight_packed") or name.endswith("weight_scale"):
|
|
packed[f] += 1
|
|
print(" tensor dtypes by family:")
|
|
for f in sorted(by):
|
|
dts = ", ".join(f"{d}x{c}" for d, c in sorted(by[f].items()))
|
|
note = f" [{packed[f]} packed/scale tensors]" if packed[f] else ""
|
|
print(f" {f:16} {dts}{note}")
|