vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
68 lines
2.5 KiB
Python
68 lines
2.5 KiB
Python
"""Reproduce the ACTUAL failing call, not a paraphrase of it.
|
|
|
|
quant_a16_datafree.py dies inside `AutoTokenizer.from_pretrained(path,
|
|
trust_remote_code=True)`. A bare `AutoConfig.from_pretrained(dir)` does NOT
|
|
reproduce it -- I checked, and all four config variants sailed through. So the
|
|
trigger lives in the tokenizer path, and testing the config alone would have sent
|
|
me off patching a file that was never the problem.
|
|
|
|
POSITIVE CONTROL, and it is the whole point of this script: zerofata's canonical
|
|
v2 tree quantized cleanly on 2026-08-21. If it now fails on this same call, the
|
|
config is exonerated and the toolchain moved under us -- `vllm/vllm-openai:latest`
|
|
was re-pulled mid-campaign and carries transformers 5.16.1 where the successful
|
|
August run had 5.12.1.
|
|
"""
|
|
import json, os, tempfile, traceback
|
|
from pathlib import Path
|
|
|
|
import transformers
|
|
from transformers import AutoTokenizer
|
|
|
|
print(f"transformers {transformers.__version__}", flush=True)
|
|
|
|
HERETIC = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16")
|
|
CANON = Path("/tank/aimodels/meromero-v2-nvfp4-work/src")
|
|
|
|
# Tokenizer loading reads config.json, so a variant needs the whole tree. Symlink
|
|
# everything, then overwrite the one file under test.
|
|
def tree(name, src, mutate=None):
|
|
d = Path(tempfile.mkdtemp(prefix=f"tok-{name}-"))
|
|
for f in src.iterdir():
|
|
if f.is_file():
|
|
os.symlink(f, d / f.name)
|
|
if mutate is not None:
|
|
cfg = json.loads((src / "config.json").read_text())
|
|
mutate(cfg)
|
|
(d / "config.json").unlink()
|
|
(d / "config.json").write_text(json.dumps(cfg))
|
|
return d
|
|
|
|
|
|
def drop_plc(c):
|
|
c["text_config"].pop("per_layer_config", None)
|
|
|
|
|
|
def force_global(c):
|
|
c["text_config"]["allow_global_per_layer_attribute_access"] = True
|
|
|
|
|
|
cases = [
|
|
("A-canonical-POSITIVE-CONTROL", tree("canon", CANON)),
|
|
("B-heretic-asis", tree("heretic", HERETIC)),
|
|
("C-heretic-drop-per_layer_config", tree("drop", HERETIC, drop_plc)),
|
|
("D-heretic-force-global-access", tree("force", HERETIC, force_global)),
|
|
("E-canonical-force-global-access", tree("canonforce", CANON, force_global)),
|
|
]
|
|
|
|
for name, d in cases:
|
|
print(f"\n=== {name} ===", flush=True)
|
|
try:
|
|
tok = AutoTokenizer.from_pretrained(d, trust_remote_code=True)
|
|
except Exception as e:
|
|
tb = traceback.format_exc().strip().splitlines()
|
|
print(f" FAILED {type(e).__name__}")
|
|
print(" " + "\n ".join(tb[-4:]))
|
|
continue
|
|
trunc = getattr(tok, "truncation_side", None)
|
|
print(f" OK {type(tok).__name__} vocab={len(tok)} truncation_side={trunc}")
|