vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
84 lines
4.3 KiB
Plaintext
84 lines
4.3 KiB
Plaintext
### verify_quant.py — new v2 heretic quant vs the 2026-08-21 known-good canonical quant
|
|
$ sudo -n python3 verify_quant.py <new> <august-known-good>
|
|
|
|
======================================================================
|
|
/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
|
group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY)
|
|
format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed
|
|
text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4
|
|
tensor dtypes by family:
|
|
embeddings BF16x1
|
|
language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors]
|
|
norms BF16x361
|
|
vision_tower BF16x356
|
|
|
|
======================================================================
|
|
/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16
|
|
group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY)
|
|
format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed
|
|
text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4
|
|
tensor dtypes by family:
|
|
embeddings BF16x1
|
|
language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors]
|
|
norms BF16x361
|
|
vision_tower BF16x356
|
|
|
|
### post_quant_gemma4.py --check — v2 heretic output
|
|
[CHECK] src=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
|
[CHECK] out=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
|
|
|
-- step 1: MTP graft
|
|
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
|
-- step 2: restore processor_config.json + preprocessor_config.json
|
|
processor_config.json already present and identical to source
|
|
preprocessor_config.json already present and correct
|
|
-- step 4: confirm saved tokenizer.json has truncation: null
|
|
truncation is null -- clean
|
|
|
|
[CHECK] done rc=0
|
|
|
|
### post_quant_gemma4.py --check — A4B output (after the truncation fix)
|
|
[CHECK] src=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16
|
|
[CHECK] out=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
|
|
|
-- step 1: MTP graft
|
|
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
|
-- step 2: restore processor_config.json + preprocessor_config.json
|
|
processor_config.json already present and identical to source
|
|
preprocessor_config.json already present and correct
|
|
-- step 4: confirm saved tokenizer.json has truncation: null
|
|
truncation is null -- clean
|
|
|
|
[CHECK] done rc=0
|
|
|
|
### post_quant_gemma4.py --check — 2026-08-21 known-good tree (POSITIVE CONTROL, must be clean)
|
|
[CHECK] src=/tank/aimodels/meromero-v2-nvfp4-work/src
|
|
[CHECK] out=/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16
|
|
|
|
-- step 1: MTP graft
|
|
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
|
-- step 2: restore processor_config.json + preprocessor_config.json
|
|
processor_config.json already present and identical to source
|
|
preprocessor_config.json already present and correct
|
|
-- step 4: confirm saved tokenizer.json has truncation: null
|
|
truncation is null -- clean
|
|
|
|
[CHECK] done rc=0
|
|
|
|
### output tree
|
|
total 19762161
|
|
drwxr-xr-x 2 infra-ops infra-ops 13 Sep 10 10:43 .
|
|
drwxrwxr-x 62 llmuser llm 99 Sep 10 08:59 ..
|
|
-rw-r--r-- 1 root root 16934 Sep 10 10:43 chat_template.jinja
|
|
-rw-r--r-- 1 root root 19419 Sep 10 10:43 config.json
|
|
-rw-r--r-- 1 root root 204 Sep 10 10:43 generation_config.json
|
|
-rw------- 1 root root 19994044576 Sep 10 10:43 model-00001-of-00002.safetensors
|
|
-rw------- 1 root root 452731960 Sep 10 10:43 model-00002-of-00002.safetensors
|
|
-rw-r--r-- 1 root root 209808 Sep 10 10:43 model.safetensors.index.json
|
|
-rw-r--r-- 1 root root 375 Sep 10 10:43 preprocessor_config.json
|
|
-rw-r--r-- 1 root root 1689 Aug 12 02:28 processor_config.json
|
|
-rw-r--r-- 1 root root 430 Sep 10 10:43 recipe.yaml
|
|
-rw-r--r-- 1 root root 2819 Sep 10 10:43 tokenizer_config.json
|
|
-rw-r--r-- 1 root root 32169780 Sep 10 10:43 tokenizer.json
|
|
19G /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|