vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
51 lines
3.0 KiB
Python
51 lines
3.0 KiB
Python
"""Do the CHECKPOINT's tensor shapes agree with num_global_key_value_heads=4?
|
|
|
|
Patching a config to satisfy a constructor is only safe if the weights already have
|
|
the shape the patched value implies. If the abliteration reshaped attention, the
|
|
patch would silence the error and produce a quietly wrong quant -- worse than the
|
|
crash, because it ships.
|
|
|
|
For a full-attention layer, k_proj/v_proj out-features == num_kv_heads * head_dim.
|
|
zerofata's canonical v2: num_global_key_value_heads=4, global_head_dim=512
|
|
=> expected out-features 4 * 512 = 2048 on the GLOBAL (full-attention) layers.
|
|
"""
|
|
import json, sys
|
|
from safetensors import safe_open
|
|
from pathlib import Path
|
|
|
|
def probe(label, root, n_global_kv, global_head_dim, n_kv, head_dim):
|
|
root = Path(root)
|
|
idx = json.load(open(root / "model.safetensors.index.json"))["weight_map"]
|
|
types = json.load(open(root / "config.json"))["text_config"]["layer_types"]
|
|
full = [i for i, t in enumerate(types) if t == "full_attention"][:2]
|
|
slide = [i for i, t in enumerate(types) if t == "sliding_attention"][:2]
|
|
print(f" -- {label}")
|
|
print(f" expected FULL k/v out-features = {n_global_kv} x {global_head_dim} = {n_global_kv*global_head_dim}"
|
|
if n_global_kv and global_head_dim else " expected FULL = (config lacks the fields)")
|
|
print(f" expected SLIDE k/v out-features = {n_kv} x {head_dim} = {n_kv*head_dim}")
|
|
for tag, idxs in (("full ", full), ("slide", slide)):
|
|
for li in idxs:
|
|
for proj in ("k_proj", "v_proj"):
|
|
key = f"model.language_model.layers.{li}.self_attn.{proj}.weight"
|
|
if key not in idx:
|
|
key = f"language_model.model.layers.{li}.self_attn.{proj}.weight"
|
|
if key not in idx:
|
|
cand = [k for k in idx if f"layers.{li}.self_attn.{proj}" in k]
|
|
key = cand[0] if cand else None
|
|
if not key:
|
|
print(f" {tag} L{li} {proj}: KEY NOT FOUND"); continue
|
|
with safe_open(root / idx[key], framework="pt") as f:
|
|
shape = f.get_slice(key).get_shape()
|
|
print(f" {tag} L{li} {proj}: shape {shape} out-features={shape[0]}")
|
|
|
|
cfg = json.load(open("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16/config.json"))["text_config"]
|
|
good = json.load(open("/tank/aimodels/meromero-v2-nvfp4-work/src/config.json"))["text_config"]
|
|
print(f" canonical (zerofata): num_global_key_value_heads={good.get('num_global_key_value_heads')} "
|
|
f"global_head_dim={good.get('global_head_dim')} num_key_value_heads={good.get('num_key_value_heads')} head_dim={good.get('head_dim')}")
|
|
probe("zerofata v2 (canonical)", "/tank/aimodels/meromero-v2-nvfp4-work/src",
|
|
good.get("num_global_key_value_heads"), good.get("global_head_dim"),
|
|
good.get("num_key_value_heads"), good.get("head_dim"))
|
|
probe("DogOnKeyboard v2 (to patch)", "/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16",
|
|
good.get("num_global_key_value_heads"), good.get("global_head_dim"),
|
|
cfg.get("num_key_value_heads"), cfg.get("head_dim"))
|