vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
40 lines
2.4 KiB
Bash
Executable File
40 lines
2.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# NVFP4A16 quant of the DogOnKeyboard v2-31B heretic (abliterated) Gemma-4.
|
|
#
|
|
# Attempt 5. Four things had to be fixed to get here and all four are load-bearing:
|
|
# 1. config.json was missing num_global_key_value_heads / global_head_dim while
|
|
# declaring attention_k_eq_v -- patched from zerofata's canonical values after
|
|
# shape-verifying the checkpoint (full-attn k_proj [2048,5376] = 4x512).
|
|
# 2. config.json carried a per_layer_config block that transformers 5.14.1 refuses
|
|
# to read globally. Removed; it was exactly redundant with (1). See
|
|
# patch_perlayer.py for the redundancy proof.
|
|
# 3. No processor_config.json in the upload -> llmcompressor demands a processor
|
|
# whenever a dataset is passed. Dropping the dataset removes the demand, and
|
|
# costs nothing: NVFP4A16 is weight-only and runs a DataFreePipeline.
|
|
# 4. The reference module argparses at import; quant_a16_datafree.py feeds it
|
|
# placeholder argv and restores the real one.
|
|
#
|
|
# IMAGE PINNED BY DIGEST, deliberately. `vllm/vllm-openai:latest` was re-pulled
|
|
# between attempt 3 and attempt 4 and moved transformers 5.12.1 -> 5.16.1, which is
|
|
# why attempt 4's error looked like a new config problem and was not. Note the
|
|
# effective transformers is 5.14.1 either way -- llmcompressor 0.13.0 pins it, and
|
|
# it is 5.14.1 the config had to be made readable by.
|
|
IMAGE=vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1
|
|
SRC=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
|
OUT=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
|
set -uo pipefail
|
|
exec >> /home/infra-ops/quant/v2.log 2>&1
|
|
echo "=== $(date -Is) START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned)"
|
|
docker rm -f meromero-quant >/dev/null 2>&1
|
|
docker run --rm --name meromero-quant --gpus '"device=1"' --ipc host \
|
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
|
-v /tank/aimodels:/tank/aimodels -v /home/infra-ops/quant:/q \
|
|
--entrypoint bash "$IMAGE" -c "
|
|
set -e
|
|
pip install -q llmcompressor==0.13.0 tiktoken sentencepiece 2>&1 | tail -1
|
|
python3 -c 'import transformers; print(\"transformers\", transformers.__version__, flush=True)'
|
|
python3 /q/quant_a16_datafree.py --model $SRC --out $OUT --scheme NVFP4A16
|
|
"
|
|
rc=$? # captured BEFORE any other command -- an earlier wrapper read $? after an echo and always said 0
|
|
echo "=== $(date -Is) END rc=$rc size=$(du -sh $OUT 2>/dev/null | cut -f1)"
|