vh
1a5bc2ddf1
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
99 lines
7.6 KiB
Plaintext
99 lines
7.6 KiB
Plaintext
=== 2026-09-10T08:13:44-07:00 START A4B-heretic -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
|
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
|
loading /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16
|
|
Loading weights: 100%|██████████| 1013/1013 [00:00<00:00, 4035.87it/s]
|
|
building calibration (<= 512 @ seq 8192)
|
|
512 calibration rows
|
|
NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
|
2026-09-10T15:14:09.8205 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
|
2026-09-10T15:14:12.3242 | reset | INFO - Compression lifecycle reset
|
|
2026-09-10T15:14:12.8109 | apply_recipe_modifiers | WARNING - Detected an MoE model which has not been linearized. First load model `with llmcompressor.modeling.moe.linearize.load_quantizable_moe` before passing to `oneshot`. Falling back to post-load linearization.
|
|
2026-09-10T15:14:13.2653 | linearize_moe | WARNING - MoE is being linearized after loading in order to support efficient calibration of experts. However, this may be inefficient if the model checkpoint is already linearized (2D -> 3D -> 2D). Consider registering a load converter for faster load times. See https://docs.vllm.ai/projects/llm-compressor/en/latest/developer-tutorials/add-moe-support
|
|
Linearizing experts: 100%|██████████| 30/30 [00:35<00:00, 1.17s/it]
|
|
2026-09-10T15:14:48.4792 | from_modifiers | INFO - Creating recipe from modifiers
|
|
Applying quantization config: 100%|██████████| 11755/11755 [00:01<00:00, 8851.89it/s]
|
|
2026-09-10T15:14:50.3917 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
|
|
2026-09-10T15:14:50.3920 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
|
|
2026-09-10T15:15:14.4903 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
|
|
saving -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
|
Compressing model: 100%|██████████| 11755/11755 [00:11<00:00, 980.90it/s]
|
|
Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.96s/it]
|
|
Dispatching model: 100%|██████████| 16828/16828 [00:00<00:00, 43870.85it/s]
|
|
DONE. serve --quantization compressed-tensors (multimodal: vision+audio kept BF16; NO --language-model-only). No spec-decode; Gemma-4 has no MTP.
|
|
=== 2026-09-10T08:15:52-07:00 END A4B-heretic rc=0 size=16G
|
|
=== 2026-09-10T08:15:52-07:00 START v2-31B-heretic -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
|
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
|
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
|
Traceback (most recent call last):
|
|
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 76, in load_model
|
|
model = M.from_pretrained(
|
|
^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained
|
|
model = cls(config, *model_args, **model_kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__
|
|
self.model = Gemma4Model(config)
|
|
^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__
|
|
language_model = AutoModel.from_config(config=config.text_config)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config
|
|
return model_class._from_config(config, **kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config
|
|
model = cls(config, **kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__
|
|
[Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__
|
|
self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__
|
|
self.num_key_value_groups = config.num_attention_heads // num_key_value_heads
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~
|
|
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in <module>
|
|
sys.exit(main())
|
|
^^^^^^
|
|
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 99, in main
|
|
model, tok = load_model(a.model)
|
|
^^^^^^^^^^^^^^^^^^^
|
|
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 82, in load_model
|
|
model = M.from_pretrained(
|
|
^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 406, in from_pretrained
|
|
return model_class.from_pretrained(
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained
|
|
model = cls(config, *model_args, **model_kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__
|
|
self.model = Gemma4Model(config)
|
|
^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__
|
|
language_model = AutoModel.from_config(config=config.text_config)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config
|
|
return model_class._from_config(config, **kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config
|
|
model = cls(config, **kwargs)
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__
|
|
[Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__
|
|
self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx)
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__
|
|
self.num_key_value_groups = config.num_attention_heads // num_key_value_heads
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~
|
|
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
|
|
=== 2026-09-10T08:16:18-07:00 END v2-31B-heretic rc=1 size=
|
|
=== 2026-09-10T08:16:18-07:00 BATCH DONE
|