vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
75 lines
3.5 KiB
Python
75 lines
3.5 KiB
Python
"""Remove the redundant `per_layer_config` block that blocks attempt 4.
|
|
|
|
Why this and not `allow_global_per_layer_attribute_access=True`:
|
|
|
|
* `per_layer_config` here carries NO information. Its keys are exactly the ten
|
|
full_attention layer indices [5,11,...,59] and its only distinct value is
|
|
(head_dim 512, num_key_value_heads 4) -- which `global_head_dim: 512` and
|
|
`num_global_key_value_heads: 4`, already in this config, say verbatim.
|
|
Removing it is lossless, and it is the difference in what the two toolchain
|
|
versions can read: transformers 5.16.1 (which authored this file) emits the
|
|
per-layer form; llmcompressor 0.13.0 PINS transformers to 5.14.1, which has
|
|
the heterogeneity guard but not the gemma4 resolver, so it refuses the read.
|
|
* Forcing global access leaves the config heterogeneous and makes
|
|
`config.head_dim` answer 256 to every caller -- including the ones building
|
|
the 512-wide full-attention layers. Geometry survived that in my meta-device
|
|
check, but llmcompressor's own onloading code is a caller I have not audited,
|
|
and it is precisely what transformers' warning is about. No reason to take
|
|
that when the lossless option verifies identically.
|
|
|
|
Verified on transformers 5.14.1, the version the quant actually runs: this config
|
|
then reports head_dim/num_key_value_heads/global_head_dim/num_global_key_value_heads
|
|
identical to zerofata's canonical tree -- the config that quantized successfully on
|
|
2026-08-21 -- and builds k_proj (2048, 5376) on layer 5 and (4096, 5376) on layer 0,
|
|
matching the checkpoint.
|
|
"""
|
|
import json
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
CFG = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16/config.json")
|
|
BAK = CFG.with_name("config.json.bak-pre-perlayer-20260910")
|
|
|
|
NOTE = (
|
|
" | infra-ops 2026-09-10 (2nd patch): removed text_config.per_layer_config, "
|
|
"a transformers-5.16.1 serialization artifact that llmcompressor 0.13.0's "
|
|
"pinned transformers 5.14.1 cannot read (AmbiguousGlobalPerLayerAttributeError "
|
|
"on head_dim). It was exactly redundant with global_head_dim=512 / "
|
|
"num_global_key_value_heads=4 -- keys were the 10 full_attention layers, sole "
|
|
"value (512, 4). Config now matches zerofata's canonical shape. Original at "
|
|
"config.json.bak-pre-perlayer-20260910."
|
|
)
|
|
|
|
cfg = json.loads(CFG.read_text())
|
|
t = cfg["text_config"]
|
|
plc = t.get("per_layer_config")
|
|
|
|
if plc is None:
|
|
print("per_layer_config already absent -- nothing to do")
|
|
raise SystemExit(0)
|
|
|
|
# Re-prove the redundancy here rather than trusting the earlier session: a patch
|
|
# that silences an error on a config it did not actually verify is how a quietly
|
|
# wrong quant ships.
|
|
full = {i for i, x in enumerate(t["layer_types"]) if x == "full_attention"}
|
|
assert {int(k) for k in plc} == full, f"per_layer_config keys {sorted(plc)} != full-attn layers {sorted(full)}"
|
|
vals = {(v["head_dim"], v["num_key_value_heads"]) for v in plc.values()}
|
|
assert vals == {(t["global_head_dim"], t["num_global_key_value_heads"])}, \
|
|
f"per_layer_config carries {vals}, not the global (512, 4) -- NOT redundant, do not drop"
|
|
print(f"redundancy re-verified: {len(plc)} entries, all {vals.pop()}, "
|
|
f"== (global_head_dim, num_global_key_value_heads)")
|
|
|
|
if not BAK.exists():
|
|
shutil.copy2(CFG, BAK)
|
|
print(f"backed up -> {BAK.name}")
|
|
else:
|
|
print(f"backup {BAK.name} already exists, left alone")
|
|
|
|
t.pop("per_layer_config")
|
|
cfg["_patched_by"] = cfg.get("_patched_by", "") + NOTE
|
|
|
|
tmp = CFG.with_suffix(".json.tmp")
|
|
tmp.write_text(json.dumps(cfg, indent=2) + "\n")
|
|
tmp.replace(CFG)
|
|
print(f"patched {CFG}")
|