Land the MeroMero v2-31B NVFP4A16 quant and record the pinned-transformers trap
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
=== 2026-09-10T08:13:44-07:00 START A4B-heretic -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
loading /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16
|
||||
Loading weights: 100%|██████████| 1013/1013 [00:00<00:00, 4035.87it/s]
|
||||
building calibration (<= 512 @ seq 8192)
|
||||
512 calibration rows
|
||||
NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
||||
2026-09-10T15:14:09.8205 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
||||
2026-09-10T15:14:12.3242 | reset | INFO - Compression lifecycle reset
|
||||
2026-09-10T15:14:12.8109 | apply_recipe_modifiers | WARNING - Detected an MoE model which has not been linearized. First load model `with llmcompressor.modeling.moe.linearize.load_quantizable_moe` before passing to `oneshot`. Falling back to post-load linearization.
|
||||
2026-09-10T15:14:13.2653 | linearize_moe | WARNING - MoE is being linearized after loading in order to support efficient calibration of experts. However, this may be inefficient if the model checkpoint is already linearized (2D -> 3D -> 2D). Consider registering a load converter for faster load times. See https://docs.vllm.ai/projects/llm-compressor/en/latest/developer-tutorials/add-moe-support
|
||||
Linearizing experts: 100%|██████████| 30/30 [00:35<00:00, 1.17s/it]
|
||||
2026-09-10T15:14:48.4792 | from_modifiers | INFO - Creating recipe from modifiers
|
||||
Applying quantization config: 100%|██████████| 11755/11755 [00:01<00:00, 8851.89it/s]
|
||||
2026-09-10T15:14:50.3917 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
|
||||
2026-09-10T15:14:50.3920 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
|
||||
2026-09-10T15:15:14.4903 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
|
||||
saving -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
Compressing model: 100%|██████████| 11755/11755 [00:11<00:00, 980.90it/s]
|
||||
Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.96s/it]
|
||||
Dispatching model: 100%|██████████| 16828/16828 [00:00<00:00, 43870.85it/s]
|
||||
DONE. serve --quantization compressed-tensors (multimodal: vision+audio kept BF16; NO --language-model-only). No spec-decode; Gemma-4 has no MTP.
|
||||
=== 2026-09-10T08:15:52-07:00 END A4B-heretic rc=0 size=16G
|
||||
=== 2026-09-10T08:15:52-07:00 START v2-31B-heretic -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Traceback (most recent call last):
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 76, in load_model
|
||||
model = M.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained
|
||||
model = cls(config, *model_args, **model_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__
|
||||
self.model = Gemma4Model(config)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__
|
||||
language_model = AutoModel.from_config(config=config.text_config)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config
|
||||
return model_class._from_config(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config
|
||||
model = cls(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__
|
||||
[Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__
|
||||
self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__
|
||||
self.num_key_value_groups = config.num_attention_heads // num_key_value_heads
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~
|
||||
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in <module>
|
||||
sys.exit(main())
|
||||
^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 99, in main
|
||||
model, tok = load_model(a.model)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 82, in load_model
|
||||
model = M.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 406, in from_pretrained
|
||||
return model_class.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained
|
||||
model = cls(config, *model_args, **model_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__
|
||||
self.model = Gemma4Model(config)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__
|
||||
language_model = AutoModel.from_config(config=config.text_config)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config
|
||||
return model_class._from_config(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config
|
||||
model = cls(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__
|
||||
[Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__
|
||||
self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__
|
||||
self.num_key_value_groups = config.num_attention_heads // num_key_value_heads
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~
|
||||
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
|
||||
=== 2026-09-10T08:16:18-07:00 END v2-31B-heretic rc=1 size=
|
||||
=== 2026-09-10T08:16:18-07:00 BATCH DONE
|
||||
@@ -0,0 +1,13 @@
|
||||
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
||||
tree: /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
tokenizer OK (1.6s) truncation_side=right
|
||||
Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 19976.82it/s]
|
||||
Compressing model: 100%|██████████| 410/410 [00:01<00:00, 293.98it/s]
|
||||
Loading weights: 100%|██████████| 2008/2008 [00:00<00:00, 4966.04it/s]
|
||||
model loaded on CPU (2.7s) Gemma4ForConditionalGeneration
|
||||
parameters: 18.46 B
|
||||
tensors still on meta device: 0 (none -- every weight was materialised)
|
||||
prompt tokens: 23
|
||||
Decompressing model: 100%|██████████| 410/410 [01:10<00:00, 5.85it/s]
|
||||
generated 24 tokens in 82.1s (3.4s/tok, CPU)
|
||||
OUTPUT: 'A lighthouse is a tower with a bright light used to guide ships at sea and warn them of dangerous coastlines.'
|
||||
@@ -0,0 +1,265 @@
|
||||
=== 2026-09-10T08:59:18-07:00 START v2-31B-heretic (post config patch)
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 2518.29it/s]
|
||||
building calibration (<= 512 @ seq 8192)
|
||||
512 calibration rows
|
||||
NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
||||
2026-09-10T15:59:48.3371 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 68, in pre_process
|
||||
model_args.processor = initialize_processor_from_path(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 184, in initialize_processor_from_path
|
||||
processor = AutoProcessor.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/processing_auto.py", line 327, in from_pretrained
|
||||
return processor_class.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1715, in from_pretrained
|
||||
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1844, in _get_arguments_from_pretrained
|
||||
sub_processor = auto_processor_class.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/feature_extraction_auto.py", line 300, in from_pretrained
|
||||
config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/feature_extraction_utils.py", line 529, in get_feature_extractor_dict
|
||||
raise OSError(
|
||||
OSError: Can't load feature extractor for '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16' is the correct path to a directory containing a preprocessor_config.json file
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in <module>
|
||||
sys.exit(main())
|
||||
^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 111, in main
|
||||
oneshot(
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 468, in oneshot
|
||||
one_shot = Oneshot(**local_args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 178, in __init__
|
||||
pre_process(model_args, dataset_args, output_dir)
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 73, in pre_process
|
||||
raise RuntimeError(
|
||||
RuntimeError: An error occurred when attempting to initialize model processor, which is required when a dataset is provided. To resolve, create and pass in a processor directly to `oneshot`/`train`.
|
||||
=== 2026-09-10T08:59:54-07:00 END rc=0 size=512
|
||||
=== 2026-09-10T09:21:40-07:00 START v2-31B-heretic (config patched + processor + data-free)
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
usage: ref [-h] --model MODEL --out OUT [--scheme SCHEME]
|
||||
ref: error: the following arguments are required: --model, --out
|
||||
=== 2026-09-10T09:21:56-07:00 END rc=2 size=512
|
||||
=== 2026-09-10T09:23:50-07:00 START v2-31B-heretic (config patched + processor + data-free)
|
||||
Unable to find image 'vllm/vllm-openai:latest' locally
|
||||
latest: Pulling from vllm/vllm-openai
|
||||
cf57d2112d89: Already exists
|
||||
c567a87f21d2: Already exists
|
||||
0b3b5bd92824: Already exists
|
||||
70f87b6ed43e: Already exists
|
||||
526d5438c009: Already exists
|
||||
79ed78d42ca9: Pulling fs layer
|
||||
5785dfb2d94d: Pulling fs layer
|
||||
8136a02ba8b6: Pulling fs layer
|
||||
1dae32d336bd: Pulling fs layer
|
||||
4f4fb700ef54: Pulling fs layer
|
||||
0e1d24786a23: Pulling fs layer
|
||||
c83c61a504db: Pulling fs layer
|
||||
81801e5f6a47: Pulling fs layer
|
||||
e832d0ac2449: Pulling fs layer
|
||||
f41db59aec9f: Pulling fs layer
|
||||
4a1facbdf857: Pulling fs layer
|
||||
b1cc0c6d03ea: Pulling fs layer
|
||||
334119c098d4: Pulling fs layer
|
||||
b14dc82c93d9: Pulling fs layer
|
||||
f599001d1dac: Pulling fs layer
|
||||
66a08f34da9c: Pulling fs layer
|
||||
2411167b6874: Pulling fs layer
|
||||
a9b8ef092e47: Pulling fs layer
|
||||
0f36e99efdcd: Pulling fs layer
|
||||
32d0568ab58d: Pulling fs layer
|
||||
c83c61a504db: Waiting
|
||||
81801e5f6a47: Waiting
|
||||
4693bfabf3cd: Pulling fs layer
|
||||
e832d0ac2449: Waiting
|
||||
7994811847da: Pulling fs layer
|
||||
d80f1ecbeb8c: Pulling fs layer
|
||||
1dae32d336bd: Waiting
|
||||
8294aa869476: Pulling fs layer
|
||||
f41db59aec9f: Waiting
|
||||
d510763bc7fa: Pulling fs layer
|
||||
4f4fb700ef54: Waiting
|
||||
2b5be6c4f7e6: Pulling fs layer
|
||||
0e1d24786a23: Waiting
|
||||
c8097f43c701: Pulling fs layer
|
||||
4a1facbdf857: Waiting
|
||||
3c84f5c1ca0b: Pulling fs layer
|
||||
b1cc0c6d03ea: Waiting
|
||||
b989116cb4ee: Pulling fs layer
|
||||
f599001d1dac: Waiting
|
||||
334119c098d4: Waiting
|
||||
cfe38ec4fb3a: Pulling fs layer
|
||||
2411167b6874: Waiting
|
||||
b14dc82c93d9: Waiting
|
||||
a9b8ef092e47: Waiting
|
||||
66a08f34da9c: Waiting
|
||||
8294aa869476: Waiting
|
||||
d510763bc7fa: Waiting
|
||||
3c84f5c1ca0b: Waiting
|
||||
2b5be6c4f7e6: Waiting
|
||||
4693bfabf3cd: Waiting
|
||||
0f36e99efdcd: Waiting
|
||||
c8097f43c701: Waiting
|
||||
b989116cb4ee: Waiting
|
||||
32d0568ab58d: Waiting
|
||||
cfe38ec4fb3a: Waiting
|
||||
7994811847da: Waiting
|
||||
d80f1ecbeb8c: Waiting
|
||||
79ed78d42ca9: Verifying Checksum
|
||||
79ed78d42ca9: Download complete
|
||||
5785dfb2d94d: Verifying Checksum
|
||||
5785dfb2d94d: Download complete
|
||||
79ed78d42ca9: Pull complete
|
||||
5785dfb2d94d: Pull complete
|
||||
4f4fb700ef54: Verifying Checksum
|
||||
4f4fb700ef54: Download complete
|
||||
0e1d24786a23: Verifying Checksum
|
||||
0e1d24786a23: Download complete
|
||||
c83c61a504db: Verifying Checksum
|
||||
c83c61a504db: Download complete
|
||||
81801e5f6a47: Verifying Checksum
|
||||
81801e5f6a47: Download complete
|
||||
e832d0ac2449: Verifying Checksum
|
||||
e832d0ac2449: Download complete
|
||||
f41db59aec9f: Download complete
|
||||
4a1facbdf857: Verifying Checksum
|
||||
4a1facbdf857: Download complete
|
||||
b1cc0c6d03ea: Verifying Checksum
|
||||
b1cc0c6d03ea: Download complete
|
||||
8136a02ba8b6: Download complete
|
||||
8136a02ba8b6: Pull complete
|
||||
b14dc82c93d9: Verifying Checksum
|
||||
b14dc82c93d9: Download complete
|
||||
f599001d1dac: Download complete
|
||||
66a08f34da9c: Verifying Checksum
|
||||
66a08f34da9c: Download complete
|
||||
2411167b6874: Verifying Checksum
|
||||
2411167b6874: Download complete
|
||||
a9b8ef092e47: Verifying Checksum
|
||||
a9b8ef092e47: Download complete
|
||||
0f36e99efdcd: Verifying Checksum
|
||||
0f36e99efdcd: Download complete
|
||||
32d0568ab58d: Verifying Checksum
|
||||
32d0568ab58d: Download complete
|
||||
4693bfabf3cd: Verifying Checksum
|
||||
4693bfabf3cd: Download complete
|
||||
7994811847da: Verifying Checksum
|
||||
7994811847da: Download complete
|
||||
d80f1ecbeb8c: Verifying Checksum
|
||||
d80f1ecbeb8c: Download complete
|
||||
8294aa869476: Verifying Checksum
|
||||
8294aa869476: Download complete
|
||||
1dae32d336bd: Verifying Checksum
|
||||
1dae32d336bd: Download complete
|
||||
d510763bc7fa: Verifying Checksum
|
||||
d510763bc7fa: Download complete
|
||||
c8097f43c701: Verifying Checksum
|
||||
c8097f43c701: Download complete
|
||||
3c84f5c1ca0b: Verifying Checksum
|
||||
3c84f5c1ca0b: Download complete
|
||||
b989116cb4ee: Verifying Checksum
|
||||
b989116cb4ee: Download complete
|
||||
cfe38ec4fb3a: Download complete
|
||||
2b5be6c4f7e6: Verifying Checksum
|
||||
2b5be6c4f7e6: Download complete
|
||||
1dae32d336bd: Pull complete
|
||||
4f4fb700ef54: Pull complete
|
||||
0e1d24786a23: Pull complete
|
||||
c83c61a504db: Pull complete
|
||||
81801e5f6a47: Pull complete
|
||||
e832d0ac2449: Pull complete
|
||||
f41db59aec9f: Pull complete
|
||||
4a1facbdf857: Pull complete
|
||||
b1cc0c6d03ea: Pull complete
|
||||
334119c098d4: Verifying Checksum
|
||||
334119c098d4: Download complete
|
||||
334119c098d4: Pull complete
|
||||
b14dc82c93d9: Pull complete
|
||||
f599001d1dac: Pull complete
|
||||
66a08f34da9c: Pull complete
|
||||
2411167b6874: Pull complete
|
||||
a9b8ef092e47: Pull complete
|
||||
0f36e99efdcd: Pull complete
|
||||
32d0568ab58d: Pull complete
|
||||
4693bfabf3cd: Pull complete
|
||||
7994811847da: Pull complete
|
||||
d80f1ecbeb8c: Pull complete
|
||||
8294aa869476: Pull complete
|
||||
d510763bc7fa: Pull complete
|
||||
2b5be6c4f7e6: Pull complete
|
||||
c8097f43c701: Pull complete
|
||||
3c84f5c1ca0b: Pull complete
|
||||
b989116cb4ee: Pull complete
|
||||
cfe38ec4fb3a: Pull complete
|
||||
Digest: sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1
|
||||
Status: Downloaded newer image for vllm/vllm-openai:latest
|
||||
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Traceback (most recent call last):
|
||||
File "/q/quant_a16_datafree.py", line 36, in <module>
|
||||
model, tok = ref.load_model(a.model)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 73, in load_model
|
||||
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 747, in from_pretrained
|
||||
config = AutoConfig.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/configuration_auto.py", line 419, in from_pretrained
|
||||
return config_class.from_dict(config_dict, **unused_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 878, in from_dict
|
||||
config = cls(**config_dict)
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 275, in init_with_validate
|
||||
initial_init(self, *args, **kwargs) # type: ignore [call-arg]
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 131, in __init__
|
||||
self.__post_init__(**additional_kwargs)
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/configuration_gemma4.py", line 348, in __post_init__
|
||||
self.text_config = Gemma4TextConfig(**self.text_config)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 276, in init_with_validate
|
||||
cls.validate(self) # type: ignore [attr-defined]
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 251, in validate
|
||||
validator(self)
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 476, in validate_architecture
|
||||
hasattr(self, "head_dim")
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 464, in __getattribute__
|
||||
return super().__getattribute__(key)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py", line 266, in __getattribute__
|
||||
raise AmbiguousGlobalPerLayerAttributeError(
|
||||
transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].head_dim). To read the global config value from config.head_dim anyway, set `allow_global_per_layer_attribute_access` to `True` on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.
|
||||
=== 2026-09-10T09:28:02-07:00 END rc=1 size=512
|
||||
=== 2026-09-10T10:40:09-07:00 START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned)
|
||||
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
||||
transformers 5.14.1
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 5430.71it/s]
|
||||
NVFP4 oneshot (DATA-FREE): scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
||||
2026-09-10T17:40:48.5554 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
||||
2026-09-10T17:40:50.1494 | reset | INFO - Compression lifecycle reset
|
||||
2026-09-10T17:40:50.1782 | from_modifiers | INFO - Creating recipe from modifiers
|
||||
Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 2863.72it/s]
|
||||
2026-09-10T17:40:50.3684 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
|
||||
2026-09-10T17:40:50.3685 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
|
||||
2026-09-10T17:42:45.7622 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
|
||||
saving -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
Compressing model: 100%|██████████| 410/410 [00:11<00:00, 36.91it/s]
|
||||
Writing model shards: 100%|██████████| 2/2 [00:08<00:00, 4.01s/it]
|
||||
Dispatching model: 100%|██████████| 1763/1763 [00:00<00:00, 58662.76it/s]
|
||||
DONE
|
||||
=== 2026-09-10T10:43:16-07:00 END rc=0 size=19G
|
||||
@@ -0,0 +1,83 @@
|
||||
### verify_quant.py — new v2 heretic quant vs the 2026-08-21 known-good canonical quant
|
||||
$ sudo -n python3 verify_quant.py <new> <august-known-good>
|
||||
|
||||
======================================================================
|
||||
/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY)
|
||||
format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed
|
||||
text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4
|
||||
tensor dtypes by family:
|
||||
embeddings BF16x1
|
||||
language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors]
|
||||
norms BF16x361
|
||||
vision_tower BF16x356
|
||||
|
||||
======================================================================
|
||||
/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16
|
||||
group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY)
|
||||
format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed
|
||||
text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4
|
||||
tensor dtypes by family:
|
||||
embeddings BF16x1
|
||||
language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors]
|
||||
norms BF16x361
|
||||
vision_tower BF16x356
|
||||
|
||||
### post_quant_gemma4.py --check — v2 heretic output
|
||||
[CHECK] src=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
[CHECK] out=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
|
||||
-- step 1: MTP graft
|
||||
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
||||
-- step 2: restore processor_config.json + preprocessor_config.json
|
||||
processor_config.json already present and identical to source
|
||||
preprocessor_config.json already present and correct
|
||||
-- step 4: confirm saved tokenizer.json has truncation: null
|
||||
truncation is null -- clean
|
||||
|
||||
[CHECK] done rc=0
|
||||
|
||||
### post_quant_gemma4.py --check — A4B output (after the truncation fix)
|
||||
[CHECK] src=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16
|
||||
[CHECK] out=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
|
||||
-- step 1: MTP graft
|
||||
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
||||
-- step 2: restore processor_config.json + preprocessor_config.json
|
||||
processor_config.json already present and identical to source
|
||||
preprocessor_config.json already present and correct
|
||||
-- step 4: confirm saved tokenizer.json has truncation: null
|
||||
truncation is null -- clean
|
||||
|
||||
[CHECK] done rc=0
|
||||
|
||||
### post_quant_gemma4.py --check — 2026-08-21 known-good tree (POSITIVE CONTROL, must be clean)
|
||||
[CHECK] src=/tank/aimodels/meromero-v2-nvfp4-work/src
|
||||
[CHECK] out=/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16
|
||||
|
||||
-- step 1: MTP graft
|
||||
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
||||
-- step 2: restore processor_config.json + preprocessor_config.json
|
||||
processor_config.json already present and identical to source
|
||||
preprocessor_config.json already present and correct
|
||||
-- step 4: confirm saved tokenizer.json has truncation: null
|
||||
truncation is null -- clean
|
||||
|
||||
[CHECK] done rc=0
|
||||
|
||||
### output tree
|
||||
total 19762161
|
||||
drwxr-xr-x 2 infra-ops infra-ops 13 Sep 10 10:43 .
|
||||
drwxrwxr-x 62 llmuser llm 99 Sep 10 08:59 ..
|
||||
-rw-r--r-- 1 root root 16934 Sep 10 10:43 chat_template.jinja
|
||||
-rw-r--r-- 1 root root 19419 Sep 10 10:43 config.json
|
||||
-rw-r--r-- 1 root root 204 Sep 10 10:43 generation_config.json
|
||||
-rw------- 1 root root 19994044576 Sep 10 10:43 model-00001-of-00002.safetensors
|
||||
-rw------- 1 root root 452731960 Sep 10 10:43 model-00002-of-00002.safetensors
|
||||
-rw-r--r-- 1 root root 209808 Sep 10 10:43 model.safetensors.index.json
|
||||
-rw-r--r-- 1 root root 375 Sep 10 10:43 preprocessor_config.json
|
||||
-rw-r--r-- 1 root root 1689 Aug 12 02:28 processor_config.json
|
||||
-rw-r--r-- 1 root root 430 Sep 10 10:43 recipe.yaml
|
||||
-rw-r--r-- 1 root root 2819 Sep 10 10:43 tokenizer_config.json
|
||||
-rw-r--r-- 1 root root 32169780 Sep 10 10:43 tokenizer.json
|
||||
19G /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
Reference in New Issue
Block a user