vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
49 lines
2.2 KiB
Python
49 lines
2.2 KiB
Python
"""NVFP4A16 without a calibration dataset.
|
|
|
|
Playbook §3.16, measured 2026-09-08 on this architecture: with scheme NVFP4A16
|
|
llm-compressor logs `Inferred DataFreePipeline` and NEVER touches the dataset.
|
|
Passing one is therefore pure liability, and it cost two failures here:
|
|
|
|
* llmcompressor demands a model PROCESSOR whenever a dataset is provided, which
|
|
is what killed the v2 pass (`DogOnKeyboard` ships no processor_config.json).
|
|
* building the calib set calls the fast tokenizer with truncation=True, which
|
|
mutates the Rust backend in place and `save_pretrained` then BAKES that cap
|
|
into the shipped tokenizer.json -- playbook §3.14, fatal on a newer
|
|
transformers for a vision model.
|
|
|
|
Dropping the dataset removes both for zero loss, because the quant is data-free.
|
|
Everything else -- targets, ignore list, save path -- matches the reference script.
|
|
"""
|
|
import argparse, json, sys, importlib.util
|
|
spec = importlib.util.spec_from_file_location(
|
|
"ref", "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py")
|
|
ref = importlib.util.module_from_spec(spec)
|
|
# The reference module runs argparse at IMPORT with required=True args, so an
|
|
# empty argv still exits 2. Feed placeholders; our own parse happens after.
|
|
_real_argv = sys.argv
|
|
sys.argv = ["ref", "--model", "/dev/null", "--calib", "/dev/null", "--out", "/dev/null"]
|
|
spec.loader.exec_module(ref)
|
|
|
|
sys.argv = _real_argv
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--model", required=True)
|
|
ap.add_argument("--out", required=True)
|
|
ap.add_argument("--scheme", default="NVFP4A16")
|
|
a = ap.parse_args()
|
|
|
|
assert a.scheme.endswith("A16"), f"{a.scheme} is not weight-only; it needs calibration data"
|
|
print(f"loading {a.model}", flush=True)
|
|
model, tok = ref.load_model(a.model)
|
|
|
|
from llmcompressor import oneshot
|
|
from llmcompressor.modifiers.quantization import QuantizationModifier
|
|
recipe = QuantizationModifier(targets="Linear", scheme=a.scheme, ignore=ref.IGNORE)
|
|
print(f"NVFP4 oneshot (DATA-FREE): scheme={a.scheme}, Linear-only, "
|
|
f"vision/audio/projector/embed/lm_head/norms kept BF16", flush=True)
|
|
oneshot(model=model, recipe=recipe)
|
|
|
|
print(f"saving -> {a.out}", flush=True)
|
|
model.save_pretrained(a.out, save_compressed=True)
|
|
tok.save_pretrained(a.out)
|
|
print("DONE", flush=True)
|