vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1a5bc2ddf1511c64e514bff2f8720337421725d1
esh-pfi-infrastructure/services/meromero-quant/raw/v2-quant-run-2026-09-10.txt
T
vh 1a5bc2ddf1 Land the MeroMero v2-31B NVFP4A16 quant and record the pinned-transformers trap 
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G.

The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError`
on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a
`per_layer_config` key zerofata's canonical one lacks -- and the standing fix was
to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong.

`pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The
config was serialized by 5.16.1, which materializes `per_layer_config` from
`global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not
the gemma4 resolver. Under the image's own transformers the same config loads
fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the
toolchain moved mid-diagnosis. Two things separated "malformed upload" from
"moved toolchain": reproducing the real failing call (a bare AutoConfig load does
not reproduce it; the trigger is reached through AutoTokenizer) and keeping
zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control.

The fix drops `per_layer_config` rather than forcing global access. It is exactly
redundant -- keys are precisely the ten full_attention layer indices, sole value
(512, 4), verbatim the global fields -- and forcing instead would make
`config.head_dim` answer 256 to the callers building the 512-wide layers.
patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever
stops holding.

Verified on the tensor table rather than the exit code: the output is identical
family-for-family and count-for-count to the August canonical quant, with 356
BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load
leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test
has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a
live seat displaced.

Also fixes the A4B output, which had a truncation cap baked into its tokenizer
(max_length 8192) from being quantized with the calibration corpus.

Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16
to say drop the dataset outright for any A16 scheme.
2026-09-10 10:56:16 -07:00

266 lines
14 KiB
Plaintext
Raw Blame History

=== 2026-09-10T08:59:18-07:00 START v2-31B-heretic (post config patch)
[notice] To update, run: python3.12 -m pip install --upgrade pip
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 2518.29it/s]
building calibration (<= 512 @ seq 8192)
512 calibration rows
NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
2026-09-10T15:59:48.3371 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 68, in pre_process
model_args.processor = initialize_processor_from_path(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 184, in initialize_processor_from_path
processor = AutoProcessor.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/processing_auto.py", line 327, in from_pretrained
return processor_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1715, in from_pretrained
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1844, in _get_arguments_from_pretrained
sub_processor = auto_processor_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/feature_extraction_auto.py", line 300, in from_pretrained
config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/feature_extraction_utils.py", line 529, in get_feature_extractor_dict
raise OSError(
OSError: Can't load feature extractor for '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16' is the correct path to a directory containing a preprocessor_config.json file
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in <module>
sys.exit(main())
^^^^^^
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 111, in main
oneshot(
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 468, in oneshot
one_shot = Oneshot(**local_args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 178, in __init__
pre_process(model_args, dataset_args, output_dir)
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 73, in pre_process
raise RuntimeError(
RuntimeError: An error occurred when attempting to initialize model processor, which is required when a dataset is provided. To resolve, create and pass in a processor directly to `oneshot`/`train`.
=== 2026-09-10T08:59:54-07:00 END rc=0 size=512
=== 2026-09-10T09:21:40-07:00 START v2-31B-heretic (config patched + processor + data-free)
[notice] To update, run: python3.12 -m pip install --upgrade pip
usage: ref [-h] --model MODEL --out OUT [--scheme SCHEME]
ref: error: the following arguments are required: --model, --out
=== 2026-09-10T09:21:56-07:00 END rc=2 size=512
=== 2026-09-10T09:23:50-07:00 START v2-31B-heretic (config patched + processor + data-free)
Unable to find image 'vllm/vllm-openai:latest' locally
latest: Pulling from vllm/vllm-openai
cf57d2112d89: Already exists
c567a87f21d2: Already exists
0b3b5bd92824: Already exists
70f87b6ed43e: Already exists
526d5438c009: Already exists
79ed78d42ca9: Pulling fs layer
5785dfb2d94d: Pulling fs layer
8136a02ba8b6: Pulling fs layer
1dae32d336bd: Pulling fs layer
4f4fb700ef54: Pulling fs layer
0e1d24786a23: Pulling fs layer
c83c61a504db: Pulling fs layer
81801e5f6a47: Pulling fs layer
e832d0ac2449: Pulling fs layer
f41db59aec9f: Pulling fs layer
4a1facbdf857: Pulling fs layer
b1cc0c6d03ea: Pulling fs layer
334119c098d4: Pulling fs layer
b14dc82c93d9: Pulling fs layer
f599001d1dac: Pulling fs layer
66a08f34da9c: Pulling fs layer
2411167b6874: Pulling fs layer
a9b8ef092e47: Pulling fs layer
0f36e99efdcd: Pulling fs layer
32d0568ab58d: Pulling fs layer
c83c61a504db: Waiting
81801e5f6a47: Waiting
4693bfabf3cd: Pulling fs layer
e832d0ac2449: Waiting
7994811847da: Pulling fs layer
d80f1ecbeb8c: Pulling fs layer
1dae32d336bd: Waiting
8294aa869476: Pulling fs layer
f41db59aec9f: Waiting
d510763bc7fa: Pulling fs layer
4f4fb700ef54: Waiting
2b5be6c4f7e6: Pulling fs layer
0e1d24786a23: Waiting
c8097f43c701: Pulling fs layer
4a1facbdf857: Waiting
3c84f5c1ca0b: Pulling fs layer
b1cc0c6d03ea: Waiting
b989116cb4ee: Pulling fs layer
f599001d1dac: Waiting
334119c098d4: Waiting
cfe38ec4fb3a: Pulling fs layer
2411167b6874: Waiting
b14dc82c93d9: Waiting
a9b8ef092e47: Waiting
66a08f34da9c: Waiting
8294aa869476: Waiting
d510763bc7fa: Waiting
3c84f5c1ca0b: Waiting
2b5be6c4f7e6: Waiting
4693bfabf3cd: Waiting
0f36e99efdcd: Waiting
c8097f43c701: Waiting
b989116cb4ee: Waiting
32d0568ab58d: Waiting
cfe38ec4fb3a: Waiting
7994811847da: Waiting
d80f1ecbeb8c: Waiting
79ed78d42ca9: Verifying Checksum
79ed78d42ca9: Download complete
5785dfb2d94d: Verifying Checksum
5785dfb2d94d: Download complete
79ed78d42ca9: Pull complete
5785dfb2d94d: Pull complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
0e1d24786a23: Verifying Checksum
0e1d24786a23: Download complete
c83c61a504db: Verifying Checksum
c83c61a504db: Download complete
81801e5f6a47: Verifying Checksum
81801e5f6a47: Download complete
e832d0ac2449: Verifying Checksum
e832d0ac2449: Download complete
f41db59aec9f: Download complete
4a1facbdf857: Verifying Checksum
4a1facbdf857: Download complete
b1cc0c6d03ea: Verifying Checksum
b1cc0c6d03ea: Download complete
8136a02ba8b6: Download complete
8136a02ba8b6: Pull complete
b14dc82c93d9: Verifying Checksum
b14dc82c93d9: Download complete
f599001d1dac: Download complete
66a08f34da9c: Verifying Checksum
66a08f34da9c: Download complete
2411167b6874: Verifying Checksum
2411167b6874: Download complete
a9b8ef092e47: Verifying Checksum
a9b8ef092e47: Download complete
0f36e99efdcd: Verifying Checksum
0f36e99efdcd: Download complete
32d0568ab58d: Verifying Checksum
32d0568ab58d: Download complete
4693bfabf3cd: Verifying Checksum
4693bfabf3cd: Download complete
7994811847da: Verifying Checksum
7994811847da: Download complete
d80f1ecbeb8c: Verifying Checksum
d80f1ecbeb8c: Download complete
8294aa869476: Verifying Checksum
8294aa869476: Download complete
1dae32d336bd: Verifying Checksum
1dae32d336bd: Download complete
d510763bc7fa: Verifying Checksum
d510763bc7fa: Download complete
c8097f43c701: Verifying Checksum
c8097f43c701: Download complete
3c84f5c1ca0b: Verifying Checksum
3c84f5c1ca0b: Download complete
b989116cb4ee: Verifying Checksum
b989116cb4ee: Download complete
cfe38ec4fb3a: Download complete
2b5be6c4f7e6: Verifying Checksum
2b5be6c4f7e6: Download complete
1dae32d336bd: Pull complete
4f4fb700ef54: Pull complete
0e1d24786a23: Pull complete
c83c61a504db: Pull complete
81801e5f6a47: Pull complete
e832d0ac2449: Pull complete
f41db59aec9f: Pull complete
4a1facbdf857: Pull complete
b1cc0c6d03ea: Pull complete
334119c098d4: Verifying Checksum
334119c098d4: Download complete
334119c098d4: Pull complete
b14dc82c93d9: Pull complete
f599001d1dac: Pull complete
66a08f34da9c: Pull complete
2411167b6874: Pull complete
a9b8ef092e47: Pull complete
0f36e99efdcd: Pull complete
32d0568ab58d: Pull complete
4693bfabf3cd: Pull complete
7994811847da: Pull complete
d80f1ecbeb8c: Pull complete
8294aa869476: Pull complete
d510763bc7fa: Pull complete
2b5be6c4f7e6: Pull complete
c8097f43c701: Pull complete
3c84f5c1ca0b: Pull complete
b989116cb4ee: Pull complete
cfe38ec4fb3a: Pull complete
Digest: sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1
Status: Downloaded newer image for vllm/vllm-openai:latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
Traceback (most recent call last):
File "/q/quant_a16_datafree.py", line 36, in <module>
model, tok = ref.load_model(a.model)
^^^^^^^^^^^^^^^^^^^^^^^
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 73, in load_model
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 747, in from_pretrained
config = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/configuration_auto.py", line 419, in from_pretrained
return config_class.from_dict(config_dict, **unused_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 878, in from_dict
config = cls(**config_dict)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 275, in init_with_validate
initial_init(self, *args, **kwargs) # type: ignore [call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 131, in __init__
self.__post_init__(**additional_kwargs)
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/configuration_gemma4.py", line 348, in __post_init__
self.text_config = Gemma4TextConfig(**self.text_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 276, in init_with_validate
cls.validate(self) # type: ignore [attr-defined]
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 251, in validate
validator(self)
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 476, in validate_architecture
hasattr(self, "head_dim")
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 464, in __getattribute__
return super().__getattribute__(key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py", line 266, in __getattribute__
raise AmbiguousGlobalPerLayerAttributeError(
transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].head_dim). To read the global config value from config.head_dim anyway, set `allow_global_per_layer_attribute_access` to `True` on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.
=== 2026-09-10T09:28:02-07:00 END rc=1 size=512
=== 2026-09-10T10:40:09-07:00 START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
transformers 5.14.1
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 5430.71it/s]
NVFP4 oneshot (DATA-FREE): scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
2026-09-10T17:40:48.5554 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
2026-09-10T17:40:50.1494 | reset | INFO - Compression lifecycle reset
2026-09-10T17:40:50.1782 | from_modifiers | INFO - Creating recipe from modifiers
Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 2863.72it/s]
2026-09-10T17:40:50.3684 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
2026-09-10T17:40:50.3685 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
2026-09-10T17:42:45.7622 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
saving -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
Compressing model: 100%|██████████| 410/410 [00:11<00:00, 36.91it/s]
Writing model shards: 100%|██████████| 2/2 [00:08<00:00, 4.01s/it]
Dispatching model: 100%|██████████| 1763/1763 [00:00<00:00, 58662.76it/s]
DONE
=== 2026-09-10T10:43:16-07:00 END rc=0 size=19G
Reference in New Issue View Git Blame Copy Permalink