Land the MeroMero v2-31B NVFP4A16 quant and record the pinned-transformers trap
The v2 dense quant had failed four times. Attempt 5 lands it at 19 G. The blocker was not what it looked like. `AmbiguousGlobalPerLayerAttributeError` on `head_dim` read as a malformed upload -- DogOnKeyboard's config carries a `per_layer_config` key zerofata's canonical one lacks -- and the standing fix was to force `allow_global_per_layer_attribute_access=True`. Both halves were wrong. `pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 -> 5.14.1. The config was serialized by 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver. Under the image's own transformers the same config loads fine. `:latest` was also re-pulled during attempt 4 and no earlier run, so the toolchain moved mid-diagnosis. Two things separated "malformed upload" from "moved toolchain": reproducing the real failing call (a bare AutoConfig load does not reproduce it; the trigger is reached through AutoTokenizer) and keeping zerofata's canonical tree, quantized cleanly on 2026-08-21, as a positive control. The fix drops `per_layer_config` rather than forcing global access. It is exactly redundant -- keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields -- and forcing instead would make `config.head_dim` answer 256 to the callers building the 512-wide layers. patch_perlayer.py re-proves that redundancy at apply time and refuses if it ever stops holding. Verified on the tensor table rather than the exit code: the output is identical family-for-family and count-for-count to the August canonical quant, with 356 BF16 vision-tower tensors preserved and input_activations=None. A GPU-free load leaves 0 tensors on meta and generates coherent prose. The section 4.4 serve test has NOT run -- GPU1 has 19.9 GB free against 19.5 GB of weights, so it needs a live seat displaced. Also fixes the A4B output, which had a truncation cap baked into its tokenizer (max_length 8192) from being quantized with the calibration corpus. Playbook gains section 3.17 for the pinned-transformers class and sharpens 3.16 to say drop the dataset outright for any A16 scheme.
This commit is contained in:
@@ -471,6 +471,47 @@ Reference: `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (linearize_moe
|
||||
11,520 expert Linears + post-steps; the published `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`
|
||||
recipe replicated, 222→252 ignore entries with audio/norm/router regexes added).
|
||||
|
||||
**So for any `*A16` scheme, do not pass a dataset at all** — not a shorter one, none. It removes
|
||||
the §3.14 tokenizer bake-in *and* llm-compressor's "initialize model processor ... required when a
|
||||
dataset is provided" demand, which is fatal on any upload that ships no `processor_config.json`.
|
||||
Both of those cost an attempt on MeroMero v2 (2026-09-10); dropping the corpus costs nothing,
|
||||
because a `DataFreePipeline` was never going to read it. Driver:
|
||||
`services/meromero-quant/quant_a16_datafree.py`. **Confirmed twice more the same day**: the A4B
|
||||
heretic quant, run *with* the corpus, shipped `max_length: 8192` in its `tokenizer.json`; the v2
|
||||
dense, run without it, came out `truncation: null`.
|
||||
|
||||
### 3.17 ⭐⭐ The transformers you measured is not the transformers that ran — llm-compressor pins it
|
||||
|
||||
**Measured 2026-09-10, MeroMero v2-31B, and it cost a full misdiagnosis.** The quant died in
|
||||
`AutoTokenizer.from_pretrained` with
|
||||
`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`. The obvious reading
|
||||
was that the source config carried a `per_layer_config` key the canonical one lacked, so that key
|
||||
was the defect. It was not.
|
||||
|
||||
`pip install llmcompressor==0.13.0` **downgrades transformers underneath you** — 5.16.1 → 5.14.1 in
|
||||
the `vllm/vllm-openai` image. The config had been serialized *by* 5.16.1, which materializes
|
||||
`per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard
|
||||
but not the gemma4 resolver, so it refuses the global read. Under the image's own 5.16.1 the very
|
||||
same config loads fine, which is exactly what makes this class expensive: **the version you print
|
||||
at the top of the script is not the version the quant runs on.**
|
||||
|
||||
- **Print the version AFTER the install**, in the same container, and put it in the log.
|
||||
`python3 -c 'import transformers; print(transformers.__version__)'` as a pipeline step, not a
|
||||
thing you check by hand once.
|
||||
- **Pin the image by digest for the length of a campaign.** `:latest` was re-pulled between
|
||||
attempts 3 and 4 of this run and moved the toolchain mid-diagnosis, so the same command produced
|
||||
a different error for reasons that had nothing to do with the change under test.
|
||||
- **Reproduce the actual failing call, not a paraphrase.** A bare `AutoConfig.from_pretrained` did
|
||||
not reproduce this at all — the trigger was reached through `AutoTokenizer`. Testing the config
|
||||
in isolation would have "cleared" it.
|
||||
- **Keep a known-good tree as the positive control.** zerofata's canonical v2, quantized cleanly
|
||||
three weeks earlier, is what separated "this config is malformed" from "this toolchain moved".
|
||||
Without it, four green variants and one red one are just noise.
|
||||
|
||||
Related but distinct from §3.4, which is about version deadlocks you can *see*. This one is a
|
||||
silent downgrade inside a line you already trusted. Instruments: `services/meromero-quant/`
|
||||
(`tok_repro.py`, `perlayer_test.py`).
|
||||
|
||||
### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer
|
||||
|
||||
**Symptom (on a newer transformers, at startup, on a vision model):**
|
||||
@@ -579,6 +620,10 @@ Never optional, always in this order, and the last one **verifies rather than as
|
||||
|
||||
Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.
|
||||
|
||||
On Gemma-4 steps 1 and 3 are N/A — the family ships no MTP head at all — which leaves 2 and 4, and
|
||||
4 is the one that fires. `services/meromero-quant/post_quant_gemma4.py` runs them idempotently with
|
||||
a `--check` mode; point it at a tree you already trust before you trust its verdict on a new one.
|
||||
|
||||
### 4.4 Test on a temp port, never on the live seat
|
||||
|
||||
Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
|
||||
@@ -698,6 +743,7 @@ above, and where the two disagree, **this file wins**.
|
||||
|---|---|
|
||||
| `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. |
|
||||
| `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) |
|
||||
| `services/meromero-quant/` | NVFP4A16 on Gemma-4 (MeroMero A4B + v2-31B ablits): the five-attempt failure chain, the pinned-transformers trap (§3.17), and the GPU-free verification instruments. |
|
||||
| `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) |
|
||||
| `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it |
|
||||
| `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 |
|
||||
|
||||
+58
-8
@@ -1,7 +1,8 @@
|
||||
# `[2026-09-10]` MeroMero acquisition — the A4B quant landed, the v2 dense has failed FOUR times
|
||||
# `[2026-09-10]` MeroMero — both quants landed; the v2 dense took five attempts
|
||||
|
||||
Operator wanted a MeroMero seat. Getting there cost four quant attempts and corrected two of my own
|
||||
wrong hypotheses, so the failure chain is the durable part.
|
||||
Operator wanted a MeroMero seat. Getting there cost five quant attempts and corrected three wrong
|
||||
hypotheses (two of the previous session's, one of mine), so the failure chain is the durable part.
|
||||
**Both outputs now exist and are verified against a known-good tree. Neither is serving yet.**
|
||||
|
||||
## The family, because I got it wrong first
|
||||
|
||||
@@ -60,13 +61,62 @@ playbook §3.16: weight-only A16 runs a `DataFreePipeline` and never touches the
|
||||
truncation cap into the shipped tokenizer. Removing it kills both for zero loss.
|
||||
3. **My own bug**: the reference module runs argparse with `required=True` at IMPORT, so blanking
|
||||
`sys.argv` still exited 2. Placeholder args, real argv restored after.
|
||||
4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — OPEN. The
|
||||
DogOnKeyboard config carries a `per_layer_config` key that zerofata's lacks (it was the one key in
|
||||
the diff I noted and did not chase). transformers refuses global access to `head_dim` on a
|
||||
heterogeneous config; the fix is likely `allow_global_per_layer_attribute_access=True`, with the
|
||||
warning that a caller assuming homogeneity may then read the wrong value.
|
||||
4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — RESOLVED,
|
||||
and **not what it looked like**. See "The attempt-4 trap" below.
|
||||
5. Clean. `rc=0`, 19 G, 3m07s.
|
||||
|
||||
⚠ **My wrapper reported `rc=0` on a failed run** because it read `$?` after an `echo`. A wrapper that
|
||||
reports success on failure is the false-reassurance class; fixed to capture `$?` immediately.
|
||||
|
||||
|
||||
## The attempt-4 trap — the toolchain moved, the config was fine
|
||||
|
||||
The standing hypothesis was that DogOnKeyboard's `per_layer_config` key was the defect and
|
||||
`allow_global_per_layer_attribute_access=True` was the fix. Both halves were wrong, and the second
|
||||
half would have shipped a risk for no reason.
|
||||
|
||||
**`pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 → 5.14.1.** The config was
|
||||
serialized *by* 5.16.1, which materializes `per_layer_config` out of `global_head_dim` +
|
||||
`layer_types`; 5.14.1 has the heterogeneity guard but not the gemma4 resolver, so it refuses the
|
||||
global read. Under the image's own 5.16.1 the identical config loads fine. On top of that,
|
||||
`vllm/vllm-openai:latest` was re-pulled *during attempt 4 and in no earlier run* — the pull line is
|
||||
in that block alone — so the error changed for reasons unrelated to anything under test.
|
||||
|
||||
Two things made this findable, and neither was inspection:
|
||||
|
||||
- **Reproducing the real call.** A bare `AutoConfig.from_pretrained` does not reproduce it; the
|
||||
trigger is reached through `AutoTokenizer`. Testing the config alone would have cleared it.
|
||||
- **A known-good positive control.** zerofata's canonical v2, quantized cleanly on 2026-08-21, is
|
||||
what separated "this upload is malformed" from "this toolchain moved". Four green variants and one
|
||||
red one are noise without it.
|
||||
|
||||
**Fix: drop `per_layer_config`, don't force global access.** It is exactly redundant — keys are
|
||||
precisely the ten `full_attention` layer indices, sole value `(512, 4)`, verbatim the global fields.
|
||||
Forcing instead leaves the config heterogeneous and makes `config.head_dim` answer 256 to every
|
||||
caller including the ones building the 512-wide layers; geometry survived it in a meta-device check,
|
||||
but llmcompressor's onloading is an unaudited caller and that is what the warning is about. The
|
||||
patch re-proves the redundancy at apply time and refuses if it ever stops holding.
|
||||
|
||||
## What landed, and what is verified
|
||||
|
||||
- `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G**, and its tensor table is **identical family for
|
||||
family and count for count to the 2026-08-21 canonical quant**: 410 U8 packed + 410 F8_E4M3 +
|
||||
410 F32 scales on the LM Linears, **356 BF16 vision-tower tensors preserved**,
|
||||
`input_activations=None` (genuinely A16). Shard sizes match that tree byte for byte.
|
||||
- CPU load-and-generate: 0 tensors left on meta, decompresses, emits coherent prose. n=1, greedy,
|
||||
24 tokens — an "is it wired up" check and nothing more.
|
||||
- ⚠ **The A4B output had the §3.14 truncation cap baked in** (`max_length: 8192`), because it was
|
||||
quantized *with* the corpus. Caught and fixed; backup at `tokenizer.json.bak-pre-truncfix`. The v2,
|
||||
run data-free, came out `truncation: null`.
|
||||
|
||||
## Still owed
|
||||
|
||||
**The §4.4 serve test has NOT run.** GPU1 has 19.9 GB free against 19.5 GB of weights, so it cannot
|
||||
happen without displacing a live seat — operator's call. Until it does, *"vllm servable"* is
|
||||
unverified for this tree, and the dense 31B's 60-layer / kv-16 geometry still does not fit 262k on
|
||||
GPU1 beside the current tenants regardless.
|
||||
|
||||
Instruments and the full write-up: `services/meromero-quant/`. General lessons:
|
||||
`docs/pfi/model-quantization-playbook.md` §3.16, **§3.17 (new)**, §4.3.
|
||||
|
||||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|
||||
@@ -116,4 +116,4 @@ resolved at run start AND end.
|
||||
|
||||
Handoff bundle for adjudication at `/mnt/smithy/handoff/r49/`.
|
||||
|
||||
Related: [[2026-09-10-meromero-acquisition-and-four-quant-failures]]
|
||||
Related: [[2026-09-10-meromero-quants-and-the-pinned-transformers-trap]]
|
||||
|
||||
+15
-8
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-10 10:25 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; MeroMero A4B quantized, v2 dense blocked at attempt 4; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_
|
||||
_Last updated: 2026-09-10 11:00 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; **MeroMero BOTH quants landed** — v2 dense on attempt 5, serve test still owed; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -141,12 +141,19 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
### MeroMero seats
|
||||
|
||||
- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, quantized in-house, W4A16.
|
||||
⚠ Missing `preprocessor_config.json`; §4.3 post-steps owed before it serves.
|
||||
- ⛔ **v2-31B dense quant has failed FOUR times**, currently on
|
||||
`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute` — DogOnKeyboard's
|
||||
config carries a `per_layer_config` key zerofata's lacks. Likely fix
|
||||
`allow_global_per_layer_attribute_access=True`. → `persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md`
|
||||
- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, W4A16, §4.3 post-steps DONE.
|
||||
⚠ It had the §3.14 **truncation cap baked into `tokenizer.json`** (`max_length: 8192`, because it
|
||||
was quantized *with* the corpus). Fixed; backup `tokenizer.json.bak-pre-truncfix`.
|
||||
- ✅ `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G, landed on attempt 5.** Tensor table identical
|
||||
family-for-family to the 2026-08-21 canonical quant; **356 BF16 vision tensors preserved**;
|
||||
`input_activations=None` (genuinely A16). CPU load+generate coherent, 0 tensors on meta.
|
||||
⚠ Attempt 4's `AmbiguousGlobalPerLayerAttributeError` was **NOT a config defect** — llmcompressor
|
||||
0.13.0 downgrades transformers 5.16.1→5.14.1, and `:latest` moved mid-campaign. Fix was to DROP
|
||||
the redundant `per_layer_config`, not to force global access.
|
||||
→ `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md`,
|
||||
`services/meromero-quant/`, playbook **§3.17 (new)**
|
||||
- ⛔ **NEITHER quant has had its §4.4 serve test** — GPU1 has 19.9 GB free against 19.5 GB of v2
|
||||
weights, so it needs a live seat displaced. Operator's call; *"vllm servable"* unverified until then.
|
||||
- ⚠ **The A4B is 30 layers / kv 8 — Pfish-6's geometry**, so it fits 262k in the existing KV budget.
|
||||
The dense 31B is 60/16, ~4x KV per token, and does NOT fit 262k on GPU1 beside the other seats.
|
||||
- Pfish-6 remains the standing seat on ana-ml2 `:8021`. Nothing repointed, no `rp-fast` alias exists.
|
||||
@@ -174,7 +181,7 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
- `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md`
|
||||
- `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`.
|
||||
- `[2026-09-10]` **MeroMero: A4B MoE quantized in-house at W4A16; the v2 dense has failed four times and is OPEN.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* → `persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md`
|
||||
- `[2026-09-10]` **MeroMero: BOTH quants landed in-house at W4A16 — A4B first try, v2 dense on attempt 5.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* The durable lesson is **§3.17**: `pip install llmcompressor` silently pins transformers down a version, so attempt 4's error was a moved toolchain, not the malformed upload it looked like — a known-good positive control is what told them apart. Serve test still owed. → `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md`
|
||||
- `[2026-09-10]` **althing 3.6.2 deployed — post office + both heralds — and the fleet has TWO herald nodes, not seven.** Ask the post office's `nodes` table, not the box inventory. Cost a self-inflicted ~12 min bus outage. → `persistent-memory.d/2026-09-10-althing-362-rollout.md`
|
||||
- `[2026-09-10]` **A grep over a log that records your greps counts itself.** I reported forseti's drop defect as reproducing here with 3 drops in 21 s; the session had **zero**. Searching transcripts writes the search term into them. Filter by `"type":"system"` provenance, never content. Generalises to any instrument that can see itself. Auto-memory `feedback_grep_over_a_log_that_records_your_greps`.
|
||||
- `[2026-09-10]` **Operator-directed purges: 466 GB (qwopus + huihui 122B bf16) and 107.8 GB Docker on ana-ml2.** Serving/rollback artifacts and qwopus's MTP head verified intact after. ⚠ `/tank` is OUTSIDE restic, so both were final.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
# MeroMero NVFP4A16 quants (Gemma-4) — instruments and the failure chain
|
||||
|
||||
Two in-house W4A16 quants of the abliterated MeroMero models, 2026-09-10. The
|
||||
operator's brief was three clauses long: *"w4a16 vllm servable, vision towers
|
||||
intact, mtp if applicable."* Every published quant of these models is W4A4 (our own
|
||||
measured long-context collapse) or, for v2, does not exist at all — 0 of 27 v2 repos.
|
||||
|
||||
| output | source | result |
|
||||
|---|---|---|
|
||||
| `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `DogOnKeyboard` A4B ablit | **16 G**, first try, 2m08s |
|
||||
| `G4-MeroMero-v2-31B-heretic-NVFP4A16` | `DogOnKeyboard` v2-31B ablit | **19 G**, attempt **5** |
|
||||
|
||||
Both live on ana-ml2 under `/tank/aimodels/`. Neither is serving yet — see *Owed*.
|
||||
|
||||
## What the v2 dense cost, and why each layer mattered
|
||||
|
||||
Five attempts, five different causes. The order matters because each one masked the
|
||||
next.
|
||||
|
||||
1. **`num_key_value_heads` is None** at `Gemma4TextAttention.__init__`. The uploader
|
||||
set `attention_k_eq_v: true` but omitted `num_global_key_value_heads` and
|
||||
`global_head_dim` — a malformed upload, not a toolchain problem. Patched from
|
||||
zerofata's canonical values (4 / 512) after shape-verifying the checkpoint
|
||||
(`shape_verify.py`): full-attn `k_proj [2048,5376]` = 4×512, sliding
|
||||
`[4096,5376]` = 16×256, identical to canonical.
|
||||
⚠ **A 2-layer truncation test PASSED and hid this.** The failing branch is chosen
|
||||
per layer type and only `full_attention` layers take it. Testing each layer type
|
||||
individually found it in seconds.
|
||||
2. **`initialize model processor ... required when a dataset is provided`.** This
|
||||
upload ships no `processor_config.json`. Rather than supply one, the dataset was
|
||||
dropped entirely — NVFP4A16 is weight-only and runs a `DataFreePipeline`, so the
|
||||
corpus was never read anyway (playbook §3.16), and passing one also bakes a
|
||||
truncation cap into the shipped tokenizer (§3.14). Removing it kills both for
|
||||
zero loss. `quant_a16_datafree.py`.
|
||||
3. **Our own bug**: the reference module argparses at import with `required=True`,
|
||||
so blanking `sys.argv` still exited 2. Placeholder argv, real one restored after.
|
||||
4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`.**
|
||||
See below — this one was not what it looked like.
|
||||
5. Clean. `rc=0`, 19 G, 3m07s.
|
||||
|
||||
## The attempt-4 trap: the transformers you measured is not the one that ran
|
||||
|
||||
The obvious reading was "DogOnKeyboard's config carries a `per_layer_config` key
|
||||
zerofata's lacks, so that key is the defect." Two measurements said otherwise.
|
||||
|
||||
`tok_repro.py` reproduces the **actual failing call** rather than a paraphrase of it
|
||||
— a bare `AutoConfig.from_pretrained` does not reproduce it, and testing that
|
||||
instead would have sent us patching a file that was never the problem — with
|
||||
zerofata's canonical tree, which quantized cleanly on 2026-08-21, as the positive
|
||||
control. Against the container's shipped transformers **5.16.1**, every variant
|
||||
passes, the unmodified heretic config included. Run the same script *after*
|
||||
`pip install llmcompressor==0.13.0` and transformers is **5.14.1**: canonical
|
||||
passes, heretic fails. **llmcompressor pins transformers and silently downgrades
|
||||
it**, so the version printed before the install is not the version that runs.
|
||||
|
||||
Compounding it, `vllm/vllm-openai:latest` was re-pulled during attempt 4 and not
|
||||
before it — the pull line appears in that run's log block and in no earlier one — so
|
||||
the toolchain moved mid-campaign (the previous session recorded 5.12.1 in-container
|
||||
while diagnosing attempt 1). That is why attempt 4's error read as a *new config
|
||||
problem* and was not one. **`run_v2_quant.sh` now pins the image by digest.**
|
||||
|
||||
`per_layer_config` was in fact a 5.16.1 serialization artifact, and an exactly
|
||||
redundant one: its keys are precisely the ten `full_attention` layer indices
|
||||
[5,11,…,59] and its sole distinct value is `(head_dim 512, num_key_value_heads 4)` —
|
||||
verbatim what `global_head_dim: 512` / `num_global_key_value_heads: 4` already say.
|
||||
`patch_perlayer.py` re-proves that redundancy at patch time and refuses to drop the
|
||||
key if it ever stops holding.
|
||||
|
||||
**Why drop it rather than set `allow_global_per_layer_attribute_access=True`.** The
|
||||
forcing flag leaves the config heterogeneous and makes `config.head_dim` answer 256
|
||||
to every caller — including the ones building the 512-wide full-attention layers.
|
||||
`perlayer_test.py` builds all four variants on the meta device and reads the k_proj
|
||||
widths back: geometry survived the flag, so it was not wrong, but llmcompressor's
|
||||
own onloading is a caller nobody here has audited and the flag's warning names
|
||||
exactly that hazard. The lossless option verified identically, so there was no
|
||||
reason to take the risk.
|
||||
|
||||
## Verification — the tensor table, not the exit code
|
||||
|
||||
`rc=0` and a plausible file size prove neither of the operator's two checkable
|
||||
requirements. `verify_quant.py` parses the safetensors headers directly (no torch,
|
||||
no GPU, no 20 GB load) and reports dtypes by module family. The new quant is
|
||||
**identical, family for family and count for count, to the 2026-08-21 known-good
|
||||
canonical quant** — 410 U8 packed + 410 F8_E4M3 scales + 410 F32 global scales on the
|
||||
LM Linears, **356 BF16 vision-tower tensors preserved**, `input_activations=None`
|
||||
(genuinely A16, not A4). Shard sizes match that tree byte for byte. Full transcript
|
||||
in `raw/verification.txt`.
|
||||
|
||||
MTP is N/A and was checked on the sources, not assumed: Gemma-4 ships no MTP head at
|
||||
all — 0 mtp tensors and no mtp config keys in either bf16 source or in any published
|
||||
quant. The "mtp if applicable" clause is a no-op for this family.
|
||||
|
||||
`cpu_smoke.py` then loads the finished tree with no GPU at all, confirms **0 tensors
|
||||
left on the meta device** (a hole `from_pretrained` will not always raise on),
|
||||
decompresses, and generates:
|
||||
|
||||
> *"A lighthouse is a tower with a bright light used to guide ships at sea and warn
|
||||
> them of dangerous coastlines."*
|
||||
|
||||
24 greedy tokens, 3.4 s/tok on CPU. That is an "is it wired up" check and is offered
|
||||
as nothing more — n=1 says nothing about quality, and it says nothing about whether
|
||||
vLLM's sm_120 NVFP4 kernels serve the thing.
|
||||
|
||||
## Owed
|
||||
|
||||
- **§4.4 serve test on a temp port.** Not run. GPU1 has 19.9 GB free against 19.5 GB
|
||||
of weights, so it cannot happen without displacing a live seat, which is the
|
||||
operator's call. Until it runs, *"vllm servable"* is unverified for this tree.
|
||||
- The A4B output **had the §3.14 truncation cap baked in** (`max_length: 8192`) — it
|
||||
was quantized *with* the calibration corpus, before the data-free path existed.
|
||||
Fixed 2026-09-10 by `post_quant_gemma4.py`; backup at
|
||||
`tokenizer.json.bak-pre-truncfix`. Both trees now pass `--check` clean, as does the
|
||||
August tree the checker is calibrated against.
|
||||
|
||||
## Files
|
||||
|
||||
| file | what it does |
|
||||
|---|---|
|
||||
| `run_v2_quant.sh` | attempt-5 runner, image pinned by digest |
|
||||
| `run_quant_batch.sh` | the original two-model batch (A4B succeeded here, v2 did not) |
|
||||
| `quant_a16_datafree.py` | NVFP4A16 driver with no dataset, wrapping the reference recipe |
|
||||
| `patch_perlayer.py` | drops `per_layer_config`, re-proving its redundancy first |
|
||||
| `shape_verify.py` | do the checkpoint's k/v shapes agree with the patched config? |
|
||||
| `perlayer_test.py` | meta-device geometry across all four config variants |
|
||||
| `tok_repro.py` | reproduces the real failing call; canonical tree as positive control |
|
||||
| `post_quant_gemma4.py` | playbook §4.3 post-steps, idempotent, `--check` mode |
|
||||
| `verify_quant.py` | dtypes by module family, straight from safetensors headers |
|
||||
| `cpu_smoke.py` | GPU-free load + generate |
|
||||
| `raw/` | run logs and the verification transcript, so the claims can be re-derived |
|
||||
|
||||
`raw/` holds `batch-quant-run-2026-09-10.txt` (attempt 1, and the A4B success),
|
||||
`v2-quant-run-2026-09-10.txt` (attempts 2-5), `cpu-smoke-2026-09-10.txt`, and
|
||||
`verification.txt`. Progress-bar redraws are collapsed to one line per bar, final
|
||||
state; nothing else is edited. (`.txt` rather than `.log` because the repo ignores
|
||||
`*.log` — same convention as `scripts/training-probes/`.)
|
||||
|
||||
General lessons live in `docs/pfi/model-quantization-playbook.md` (§3.4, §3.14,
|
||||
§3.16, §3.17, §4.3) — read that first, and where it disagrees with this file, it wins.
|
||||
@@ -0,0 +1,57 @@
|
||||
"""GPU-free load-and-generate smoke test for the quantized tree.
|
||||
|
||||
§4.4 says test on a temp port, never on the live seat -- but GPU1 has 19.9 GB free
|
||||
against 19.5 GB of weights, so a vLLM serve test cannot happen without displacing a
|
||||
live seat, and that is not my call. This is what CAN be established without one:
|
||||
that the checkpoint's tensor names map cleanly onto the architecture (no missing or
|
||||
unexpected keys), that compressed-tensors can decompress it, and that it emits
|
||||
plausible tokens rather than garbage.
|
||||
|
||||
What it does NOT establish: that vLLM's sm_120 NVFP4 kernels serve it, or anything
|
||||
about long-context quality. Those still need the GPU. Saying so is part of the
|
||||
result -- a smoke test whose limits go unstated gets read as more than it is.
|
||||
|
||||
Deliberately greedy and short. This is an "is it wired up" check, not an eval; n=1
|
||||
proves nothing about quality and is not offered as if it did.
|
||||
"""
|
||||
import sys
|
||||
import time
|
||||
|
||||
import torch
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
path = sys.argv[1]
|
||||
print(f"tree: {path}", flush=True)
|
||||
|
||||
t0 = time.time()
|
||||
tok = AutoTokenizer.from_pretrained(path)
|
||||
print(f"tokenizer OK ({time.time()-t0:.1f}s) truncation_side={tok.truncation_side}", flush=True)
|
||||
|
||||
t0 = time.time()
|
||||
from transformers import AutoModelForImageTextToText as M
|
||||
model = M.from_pretrained(path, dtype=torch.bfloat16, device_map=None)
|
||||
print(f"model loaded on CPU ({time.time()-t0:.1f}s) {type(model).__name__}", flush=True)
|
||||
|
||||
n = sum(p.numel() for p in model.parameters())
|
||||
print(f"parameters: {n/1e9:.2f} B", flush=True)
|
||||
|
||||
# Any tensor still sitting on meta means a weight the checkpoint never supplied --
|
||||
# from_pretrained does not always raise on that, it just leaves the hole.
|
||||
meta = [k for k, v in model.state_dict().items() if v.is_meta]
|
||||
print(f"tensors still on meta device: {len(meta)}"
|
||||
+ (f" *** {meta[:5]}" if meta else " (none -- every weight was materialised)"), flush=True)
|
||||
|
||||
msgs = [{"role": "user", "content": "In one sentence, what is a lighthouse for?"}]
|
||||
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
|
||||
# transformers 5.x hands back a BatchEncoding here, not a bare tensor.
|
||||
ids = enc["input_ids"] if hasattr(enc, "keys") else enc
|
||||
print(f"prompt tokens: {ids.shape[-1]}", flush=True)
|
||||
|
||||
t0 = time.time()
|
||||
with torch.inference_mode():
|
||||
out = model.generate(ids, max_new_tokens=24, do_sample=False)
|
||||
dt = time.time() - t0
|
||||
text = tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)
|
||||
print(f"generated {out.shape[-1]-ids.shape[-1]} tokens in {dt:.1f}s "
|
||||
f"({dt/max(1,out.shape[-1]-ids.shape[-1]):.1f}s/tok, CPU)", flush=True)
|
||||
print(f"OUTPUT: {text!r}", flush=True)
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Remove the redundant `per_layer_config` block that blocks attempt 4.
|
||||
|
||||
Why this and not `allow_global_per_layer_attribute_access=True`:
|
||||
|
||||
* `per_layer_config` here carries NO information. Its keys are exactly the ten
|
||||
full_attention layer indices [5,11,...,59] and its only distinct value is
|
||||
(head_dim 512, num_key_value_heads 4) -- which `global_head_dim: 512` and
|
||||
`num_global_key_value_heads: 4`, already in this config, say verbatim.
|
||||
Removing it is lossless, and it is the difference in what the two toolchain
|
||||
versions can read: transformers 5.16.1 (which authored this file) emits the
|
||||
per-layer form; llmcompressor 0.13.0 PINS transformers to 5.14.1, which has
|
||||
the heterogeneity guard but not the gemma4 resolver, so it refuses the read.
|
||||
* Forcing global access leaves the config heterogeneous and makes
|
||||
`config.head_dim` answer 256 to every caller -- including the ones building
|
||||
the 512-wide full-attention layers. Geometry survived that in my meta-device
|
||||
check, but llmcompressor's own onloading code is a caller I have not audited,
|
||||
and it is precisely what transformers' warning is about. No reason to take
|
||||
that when the lossless option verifies identically.
|
||||
|
||||
Verified on transformers 5.14.1, the version the quant actually runs: this config
|
||||
then reports head_dim/num_key_value_heads/global_head_dim/num_global_key_value_heads
|
||||
identical to zerofata's canonical tree -- the config that quantized successfully on
|
||||
2026-08-21 -- and builds k_proj (2048, 5376) on layer 5 and (4096, 5376) on layer 0,
|
||||
matching the checkpoint.
|
||||
"""
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
CFG = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16/config.json")
|
||||
BAK = CFG.with_name("config.json.bak-pre-perlayer-20260910")
|
||||
|
||||
NOTE = (
|
||||
" | infra-ops 2026-09-10 (2nd patch): removed text_config.per_layer_config, "
|
||||
"a transformers-5.16.1 serialization artifact that llmcompressor 0.13.0's "
|
||||
"pinned transformers 5.14.1 cannot read (AmbiguousGlobalPerLayerAttributeError "
|
||||
"on head_dim). It was exactly redundant with global_head_dim=512 / "
|
||||
"num_global_key_value_heads=4 -- keys were the 10 full_attention layers, sole "
|
||||
"value (512, 4). Config now matches zerofata's canonical shape. Original at "
|
||||
"config.json.bak-pre-perlayer-20260910."
|
||||
)
|
||||
|
||||
cfg = json.loads(CFG.read_text())
|
||||
t = cfg["text_config"]
|
||||
plc = t.get("per_layer_config")
|
||||
|
||||
if plc is None:
|
||||
print("per_layer_config already absent -- nothing to do")
|
||||
raise SystemExit(0)
|
||||
|
||||
# Re-prove the redundancy here rather than trusting the earlier session: a patch
|
||||
# that silences an error on a config it did not actually verify is how a quietly
|
||||
# wrong quant ships.
|
||||
full = {i for i, x in enumerate(t["layer_types"]) if x == "full_attention"}
|
||||
assert {int(k) for k in plc} == full, f"per_layer_config keys {sorted(plc)} != full-attn layers {sorted(full)}"
|
||||
vals = {(v["head_dim"], v["num_key_value_heads"]) for v in plc.values()}
|
||||
assert vals == {(t["global_head_dim"], t["num_global_key_value_heads"])}, \
|
||||
f"per_layer_config carries {vals}, not the global (512, 4) -- NOT redundant, do not drop"
|
||||
print(f"redundancy re-verified: {len(plc)} entries, all {vals.pop()}, "
|
||||
f"== (global_head_dim, num_global_key_value_heads)")
|
||||
|
||||
if not BAK.exists():
|
||||
shutil.copy2(CFG, BAK)
|
||||
print(f"backed up -> {BAK.name}")
|
||||
else:
|
||||
print(f"backup {BAK.name} already exists, left alone")
|
||||
|
||||
t.pop("per_layer_config")
|
||||
cfg["_patched_by"] = cfg.get("_patched_by", "") + NOTE
|
||||
|
||||
tmp = CFG.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(cfg, indent=2) + "\n")
|
||||
tmp.replace(CFG)
|
||||
print(f"patched {CFG}")
|
||||
@@ -0,0 +1,100 @@
|
||||
"""Attempt-4 blocker: is dropping `per_layer_config` the correct fix, or must we
|
||||
force `allow_global_per_layer_attribute_access`?
|
||||
|
||||
The two candidates are NOT equivalent:
|
||||
* DROP -> config becomes homogeneous in transformers' eyes and the global
|
||||
`global_head_dim` / `num_global_key_value_heads` fields describe the
|
||||
full-attention layers, exactly as zerofata's canonical config does.
|
||||
* FORCE -> config stays heterogeneous; `config.head_dim` starts answering 256 to
|
||||
every caller, including the ones building the 512-wide full-attention
|
||||
layers. That is the hazard transformers' own warning names.
|
||||
|
||||
So this is not a "did the traceback go away" test. It builds the model on the meta
|
||||
device from each candidate and reads the ACTUAL k_proj widths back, against the
|
||||
checkpoint's measured 2048 (full) / 4096 (sliding). A candidate that constructs but
|
||||
mis-shapes a layer is a worse outcome than the crash, because it would ship.
|
||||
|
||||
Positive control: zerofata's canonical config, which we already quantized
|
||||
successfully on 2026-08-21, MUST pass every check here. If it doesn't, the
|
||||
instrument is broken and none of the negatives mean anything.
|
||||
"""
|
||||
import json, shutil, tempfile, traceback
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import transformers
|
||||
from transformers import AutoConfig
|
||||
|
||||
print(f"transformers {transformers.__version__} torch {torch.__version__}", flush=True)
|
||||
|
||||
HERETIC = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16")
|
||||
CANON = Path("/tank/aimodels/meromero-v2-nvfp4-work/src")
|
||||
|
||||
# Measured off the checkpoints by shape_verify.py; both trees agree.
|
||||
EXPECT = {"full_attention": 2048, "sliding_attention": 4096}
|
||||
|
||||
|
||||
def variant(name, cfg_dict):
|
||||
d = Path(tempfile.mkdtemp(prefix=f"cfg-{name}-"))
|
||||
(d / "config.json").write_text(json.dumps(cfg_dict))
|
||||
return name, d
|
||||
|
||||
|
||||
heretic = json.loads((HERETIC / "config.json").read_text())
|
||||
canon = json.loads((CANON / "config.json").read_text())
|
||||
|
||||
dropped = json.loads(json.dumps(heretic))
|
||||
dropped["text_config"].pop("per_layer_config")
|
||||
|
||||
forced = json.loads(json.dumps(heretic))
|
||||
forced["text_config"]["allow_global_per_layer_attribute_access"] = True
|
||||
|
||||
variants = [
|
||||
variant("A-canonical-POSITIVE-CONTROL", canon),
|
||||
variant("B-heretic-asis", heretic),
|
||||
variant("C-heretic-drop-per_layer_config", dropped),
|
||||
variant("D-heretic-force-global-access", forced),
|
||||
]
|
||||
|
||||
for name, d in variants:
|
||||
print(f"\n=== {name} ===", flush=True)
|
||||
try:
|
||||
cfg = AutoConfig.from_pretrained(d)
|
||||
except Exception as e:
|
||||
print(f" CONFIG FAILED: {type(e).__name__}: {str(e)[:160]}")
|
||||
continue
|
||||
t = cfg.text_config
|
||||
fields = {}
|
||||
for k in ("head_dim", "num_key_value_heads", "global_head_dim",
|
||||
"num_global_key_value_heads"):
|
||||
try:
|
||||
fields[k] = getattr(t, k, "<absent>")
|
||||
except Exception as e:
|
||||
fields[k] = f"<{type(e).__name__}>"
|
||||
print(f" config OK: {fields}")
|
||||
|
||||
try:
|
||||
from transformers import Gemma4ForConditionalGeneration as M
|
||||
with torch.device("meta"):
|
||||
model = M(cfg)
|
||||
except Exception:
|
||||
print(" MODEL BUILD FAILED:")
|
||||
print(" " + traceback.format_exc().strip().replace("\n", "\n ")[-1200:])
|
||||
continue
|
||||
|
||||
layer_types = t.layer_types
|
||||
probes = [next(i for i, x in enumerate(layer_types) if x == "full_attention"),
|
||||
next(i for i, x in enumerate(layer_types) if x == "sliding_attention")]
|
||||
layers = model.model.language_model.layers
|
||||
verdict = []
|
||||
for li in probes:
|
||||
got = tuple(layers[li].self_attn.k_proj.weight.shape)
|
||||
want = EXPECT[layer_types[li]]
|
||||
ok = got[0] == want
|
||||
verdict.append(ok)
|
||||
print(f" L{li:>2} {layer_types[li]:<18} k_proj {got} "
|
||||
f"want out={want} {'OK' if ok else '*** MISMATCH ***'}")
|
||||
print(f" => {'GEOMETRY MATCHES CHECKPOINT' if all(verdict) else 'GEOMETRY WRONG'}")
|
||||
|
||||
for _, d in variants:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Playbook §4.3 post-steps for a Gemma-4 NVFP4 output tree.
|
||||
|
||||
Steps 1 and 3 (MTP graft, `re:^mtp.*` re-injection) are N/A on Gemma-4 -- it ships
|
||||
no MTP head at all, verified as 0 mtp tensors in both bf16 sources. That leaves:
|
||||
|
||||
step 2 restore processor_config.json + preprocessor_config.json
|
||||
step 4 confirm the saved tokenizer.json has truncation: null
|
||||
|
||||
Step 4 is not a formality here. The A4B output was quantized WITH the calibration
|
||||
dataset, and build_calib calls the fast tokenizer with truncation=True,
|
||||
max_length=8192 -- which mutates the Rust backend in place, and save_pretrained
|
||||
then bakes the cap into the shipped tokenizer.json. It is latent on the
|
||||
transformers that wrote it and fatal on a newer one. The fix edits the one
|
||||
`truncation` key rather than copying the source file wholesale, so nothing else in
|
||||
a 32 MB tokenizer can quietly change underneath it.
|
||||
|
||||
Derivation of preprocessor_config.json is `processor_config.json["image_processor"]`
|
||||
verbatim; that reproduces the 2026-08-21 known-good output byte for byte.
|
||||
|
||||
Idempotent, and reports per step whether it CHANGED or was already correct.
|
||||
Run with --check to verify without writing.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--src", required=True, help="bf16 source tree")
|
||||
ap.add_argument("--out", required=True, help="quantized output tree")
|
||||
ap.add_argument("--check", action="store_true", help="report only, write nothing")
|
||||
a = ap.parse_args()
|
||||
src, out = Path(a.src), Path(a.out)
|
||||
mode = "CHECK" if a.check else "APPLY"
|
||||
print(f"[{mode}] src={src}\n[{mode}] out={out}\n")
|
||||
|
||||
rc = 0
|
||||
|
||||
|
||||
def step(n, desc):
|
||||
print(f"-- step {n}: {desc}")
|
||||
|
||||
|
||||
step(1, "MTP graft")
|
||||
mtp = [k for k in json.loads((out / "config.json").read_text()).get(
|
||||
"quantization_config", {}).get("ignore", []) if "mtp" in k.lower()]
|
||||
idx = out / "model.safetensors.index.json"
|
||||
tensors = json.loads(idx.read_text())["weight_map"] if idx.exists() else {}
|
||||
n_mtp = sum(1 for k in tensors if k.startswith("mtp"))
|
||||
print(f" N/A for Gemma-4 (no MTP head). mtp tensors in output index: {n_mtp}; "
|
||||
f"mtp entries in ignore list: {len(mtp)}")
|
||||
if n_mtp:
|
||||
print(" *** unexpected mtp tensors -- step 3 would become live, investigate")
|
||||
rc = 1
|
||||
|
||||
step(2, "restore processor_config.json + preprocessor_config.json")
|
||||
spc = src / "processor_config.json"
|
||||
if not spc.exists():
|
||||
print(f" *** source has no processor_config.json -- cannot restore")
|
||||
rc = 1
|
||||
else:
|
||||
opc = out / "processor_config.json"
|
||||
if opc.exists() and opc.read_bytes() == spc.read_bytes():
|
||||
print(" processor_config.json already present and identical to source")
|
||||
elif a.check:
|
||||
print(f" processor_config.json MISSING/differs -> would copy from source")
|
||||
else:
|
||||
shutil.copy2(spc, opc)
|
||||
print(" processor_config.json CHANGED (copied from source)")
|
||||
|
||||
want = json.dumps(dict(json.loads(spc.read_text())["image_processor"]), indent=1)
|
||||
opre = out / "preprocessor_config.json"
|
||||
if opre.exists() and opre.read_text() == want:
|
||||
print(" preprocessor_config.json already present and correct")
|
||||
elif a.check:
|
||||
print(" preprocessor_config.json MISSING/differs -> would derive from image_processor")
|
||||
else:
|
||||
opre.write_text(want)
|
||||
print(" preprocessor_config.json CHANGED (derived from processor_config"
|
||||
"['image_processor'])")
|
||||
|
||||
step(4, "confirm saved tokenizer.json has truncation: null")
|
||||
tj = out / "tokenizer.json"
|
||||
tok = json.loads(tj.read_text())
|
||||
trunc = tok.get("truncation")
|
||||
if trunc is None:
|
||||
print(" truncation is null -- clean")
|
||||
else:
|
||||
print(f" *** truncation BAKED IN: {trunc}")
|
||||
stok = json.loads((src / "tokenizer.json").read_text())
|
||||
others = [k for k in set(tok) | set(stok)
|
||||
if k != "truncation" and tok.get(k) != stok.get(k)]
|
||||
print(f" other top-level keys differing from source: {others or 'none'}")
|
||||
if a.check:
|
||||
print(" would set truncation -> null")
|
||||
rc = 1
|
||||
else:
|
||||
bak = tj.with_name("tokenizer.json.bak-pre-truncfix")
|
||||
if not bak.exists():
|
||||
shutil.copy2(tj, bak)
|
||||
print(f" backed up -> {bak.name}")
|
||||
tok["truncation"] = None
|
||||
tmp = tj.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(tok, ensure_ascii=False, indent=2))
|
||||
tmp.replace(tj)
|
||||
print(" truncation CHANGED -> null")
|
||||
|
||||
print(f"\n[{mode}] done rc={rc}")
|
||||
raise SystemExit(rc)
|
||||
@@ -0,0 +1,48 @@
|
||||
"""NVFP4A16 without a calibration dataset.
|
||||
|
||||
Playbook §3.16, measured 2026-09-08 on this architecture: with scheme NVFP4A16
|
||||
llm-compressor logs `Inferred DataFreePipeline` and NEVER touches the dataset.
|
||||
Passing one is therefore pure liability, and it cost two failures here:
|
||||
|
||||
* llmcompressor demands a model PROCESSOR whenever a dataset is provided, which
|
||||
is what killed the v2 pass (`DogOnKeyboard` ships no processor_config.json).
|
||||
* building the calib set calls the fast tokenizer with truncation=True, which
|
||||
mutates the Rust backend in place and `save_pretrained` then BAKES that cap
|
||||
into the shipped tokenizer.json -- playbook §3.14, fatal on a newer
|
||||
transformers for a vision model.
|
||||
|
||||
Dropping the dataset removes both for zero loss, because the quant is data-free.
|
||||
Everything else -- targets, ignore list, save path -- matches the reference script.
|
||||
"""
|
||||
import argparse, json, sys, importlib.util
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"ref", "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py")
|
||||
ref = importlib.util.module_from_spec(spec)
|
||||
# The reference module runs argparse at IMPORT with required=True args, so an
|
||||
# empty argv still exits 2. Feed placeholders; our own parse happens after.
|
||||
_real_argv = sys.argv
|
||||
sys.argv = ["ref", "--model", "/dev/null", "--calib", "/dev/null", "--out", "/dev/null"]
|
||||
spec.loader.exec_module(ref)
|
||||
|
||||
sys.argv = _real_argv
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", required=True)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--scheme", default="NVFP4A16")
|
||||
a = ap.parse_args()
|
||||
|
||||
assert a.scheme.endswith("A16"), f"{a.scheme} is not weight-only; it needs calibration data"
|
||||
print(f"loading {a.model}", flush=True)
|
||||
model, tok = ref.load_model(a.model)
|
||||
|
||||
from llmcompressor import oneshot
|
||||
from llmcompressor.modifiers.quantization import QuantizationModifier
|
||||
recipe = QuantizationModifier(targets="Linear", scheme=a.scheme, ignore=ref.IGNORE)
|
||||
print(f"NVFP4 oneshot (DATA-FREE): scheme={a.scheme}, Linear-only, "
|
||||
f"vision/audio/projector/embed/lm_head/norms kept BF16", flush=True)
|
||||
oneshot(model=model, recipe=recipe)
|
||||
|
||||
print(f"saving -> {a.out}", flush=True)
|
||||
model.save_pretrained(a.out, save_compressed=True)
|
||||
tok.save_pretrained(a.out)
|
||||
print("DONE", flush=True)
|
||||
@@ -0,0 +1,98 @@
|
||||
=== 2026-09-10T08:13:44-07:00 START A4B-heretic -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
loading /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16
|
||||
Loading weights: 100%|██████████| 1013/1013 [00:00<00:00, 4035.87it/s]
|
||||
building calibration (<= 512 @ seq 8192)
|
||||
512 calibration rows
|
||||
NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
||||
2026-09-10T15:14:09.8205 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
||||
2026-09-10T15:14:12.3242 | reset | INFO - Compression lifecycle reset
|
||||
2026-09-10T15:14:12.8109 | apply_recipe_modifiers | WARNING - Detected an MoE model which has not been linearized. First load model `with llmcompressor.modeling.moe.linearize.load_quantizable_moe` before passing to `oneshot`. Falling back to post-load linearization.
|
||||
2026-09-10T15:14:13.2653 | linearize_moe | WARNING - MoE is being linearized after loading in order to support efficient calibration of experts. However, this may be inefficient if the model checkpoint is already linearized (2D -> 3D -> 2D). Consider registering a load converter for faster load times. See https://docs.vllm.ai/projects/llm-compressor/en/latest/developer-tutorials/add-moe-support
|
||||
Linearizing experts: 100%|██████████| 30/30 [00:35<00:00, 1.17s/it]
|
||||
2026-09-10T15:14:48.4792 | from_modifiers | INFO - Creating recipe from modifiers
|
||||
Applying quantization config: 100%|██████████| 11755/11755 [00:01<00:00, 8851.89it/s]
|
||||
2026-09-10T15:14:50.3917 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
|
||||
2026-09-10T15:14:50.3920 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
|
||||
2026-09-10T15:15:14.4903 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
|
||||
saving -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
Compressing model: 100%|██████████| 11755/11755 [00:11<00:00, 980.90it/s]
|
||||
Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.96s/it]
|
||||
Dispatching model: 100%|██████████| 16828/16828 [00:00<00:00, 43870.85it/s]
|
||||
DONE. serve --quantization compressed-tensors (multimodal: vision+audio kept BF16; NO --language-model-only). No spec-decode; Gemma-4 has no MTP.
|
||||
=== 2026-09-10T08:15:52-07:00 END A4B-heretic rc=0 size=16G
|
||||
=== 2026-09-10T08:15:52-07:00 START v2-31B-heretic -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Traceback (most recent call last):
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 76, in load_model
|
||||
model = M.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained
|
||||
model = cls(config, *model_args, **model_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__
|
||||
self.model = Gemma4Model(config)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__
|
||||
language_model = AutoModel.from_config(config=config.text_config)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config
|
||||
return model_class._from_config(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config
|
||||
model = cls(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__
|
||||
[Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__
|
||||
self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__
|
||||
self.num_key_value_groups = config.num_attention_heads // num_key_value_heads
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~
|
||||
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in <module>
|
||||
sys.exit(main())
|
||||
^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 99, in main
|
||||
model, tok = load_model(a.model)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 82, in load_model
|
||||
model = M.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 406, in from_pretrained
|
||||
return model_class.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained
|
||||
model = cls(config, *model_args, **model_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__
|
||||
self.model = Gemma4Model(config)
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__
|
||||
language_model = AutoModel.from_config(config=config.text_config)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config
|
||||
return model_class._from_config(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config
|
||||
model = cls(config, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__
|
||||
[Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__
|
||||
self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__
|
||||
self.num_key_value_groups = config.num_attention_heads // num_key_value_heads
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~
|
||||
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
|
||||
=== 2026-09-10T08:16:18-07:00 END v2-31B-heretic rc=1 size=
|
||||
=== 2026-09-10T08:16:18-07:00 BATCH DONE
|
||||
@@ -0,0 +1,13 @@
|
||||
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
||||
tree: /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
tokenizer OK (1.6s) truncation_side=right
|
||||
Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 19976.82it/s]
|
||||
Compressing model: 100%|██████████| 410/410 [00:01<00:00, 293.98it/s]
|
||||
Loading weights: 100%|██████████| 2008/2008 [00:00<00:00, 4966.04it/s]
|
||||
model loaded on CPU (2.7s) Gemma4ForConditionalGeneration
|
||||
parameters: 18.46 B
|
||||
tensors still on meta device: 0 (none -- every weight was materialised)
|
||||
prompt tokens: 23
|
||||
Decompressing model: 100%|██████████| 410/410 [01:10<00:00, 5.85it/s]
|
||||
generated 24 tokens in 82.1s (3.4s/tok, CPU)
|
||||
OUTPUT: 'A lighthouse is a tower with a bright light used to guide ships at sea and warn them of dangerous coastlines.'
|
||||
@@ -0,0 +1,265 @@
|
||||
=== 2026-09-10T08:59:18-07:00 START v2-31B-heretic (post config patch)
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 2518.29it/s]
|
||||
building calibration (<= 512 @ seq 8192)
|
||||
512 calibration rows
|
||||
NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
||||
2026-09-10T15:59:48.3371 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 68, in pre_process
|
||||
model_args.processor = initialize_processor_from_path(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 184, in initialize_processor_from_path
|
||||
processor = AutoProcessor.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/processing_auto.py", line 327, in from_pretrained
|
||||
return processor_class.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1715, in from_pretrained
|
||||
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1844, in _get_arguments_from_pretrained
|
||||
sub_processor = auto_processor_class.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/feature_extraction_auto.py", line 300, in from_pretrained
|
||||
config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/feature_extraction_utils.py", line 529, in get_feature_extractor_dict
|
||||
raise OSError(
|
||||
OSError: Can't load feature extractor for '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16' is the correct path to a directory containing a preprocessor_config.json file
|
||||
|
||||
The above exception was the direct cause of the following exception:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in <module>
|
||||
sys.exit(main())
|
||||
^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 111, in main
|
||||
oneshot(
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 468, in oneshot
|
||||
one_shot = Oneshot(**local_args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 178, in __init__
|
||||
pre_process(model_args, dataset_args, output_dir)
|
||||
File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 73, in pre_process
|
||||
raise RuntimeError(
|
||||
RuntimeError: An error occurred when attempting to initialize model processor, which is required when a dataset is provided. To resolve, create and pass in a processor directly to `oneshot`/`train`.
|
||||
=== 2026-09-10T08:59:54-07:00 END rc=0 size=512
|
||||
=== 2026-09-10T09:21:40-07:00 START v2-31B-heretic (config patched + processor + data-free)
|
||||
[notice] To update, run: python3.12 -m pip install --upgrade pip
|
||||
usage: ref [-h] --model MODEL --out OUT [--scheme SCHEME]
|
||||
ref: error: the following arguments are required: --model, --out
|
||||
=== 2026-09-10T09:21:56-07:00 END rc=2 size=512
|
||||
=== 2026-09-10T09:23:50-07:00 START v2-31B-heretic (config patched + processor + data-free)
|
||||
Unable to find image 'vllm/vllm-openai:latest' locally
|
||||
latest: Pulling from vllm/vllm-openai
|
||||
cf57d2112d89: Already exists
|
||||
c567a87f21d2: Already exists
|
||||
0b3b5bd92824: Already exists
|
||||
70f87b6ed43e: Already exists
|
||||
526d5438c009: Already exists
|
||||
79ed78d42ca9: Pulling fs layer
|
||||
5785dfb2d94d: Pulling fs layer
|
||||
8136a02ba8b6: Pulling fs layer
|
||||
1dae32d336bd: Pulling fs layer
|
||||
4f4fb700ef54: Pulling fs layer
|
||||
0e1d24786a23: Pulling fs layer
|
||||
c83c61a504db: Pulling fs layer
|
||||
81801e5f6a47: Pulling fs layer
|
||||
e832d0ac2449: Pulling fs layer
|
||||
f41db59aec9f: Pulling fs layer
|
||||
4a1facbdf857: Pulling fs layer
|
||||
b1cc0c6d03ea: Pulling fs layer
|
||||
334119c098d4: Pulling fs layer
|
||||
b14dc82c93d9: Pulling fs layer
|
||||
f599001d1dac: Pulling fs layer
|
||||
66a08f34da9c: Pulling fs layer
|
||||
2411167b6874: Pulling fs layer
|
||||
a9b8ef092e47: Pulling fs layer
|
||||
0f36e99efdcd: Pulling fs layer
|
||||
32d0568ab58d: Pulling fs layer
|
||||
c83c61a504db: Waiting
|
||||
81801e5f6a47: Waiting
|
||||
4693bfabf3cd: Pulling fs layer
|
||||
e832d0ac2449: Waiting
|
||||
7994811847da: Pulling fs layer
|
||||
d80f1ecbeb8c: Pulling fs layer
|
||||
1dae32d336bd: Waiting
|
||||
8294aa869476: Pulling fs layer
|
||||
f41db59aec9f: Waiting
|
||||
d510763bc7fa: Pulling fs layer
|
||||
4f4fb700ef54: Waiting
|
||||
2b5be6c4f7e6: Pulling fs layer
|
||||
0e1d24786a23: Waiting
|
||||
c8097f43c701: Pulling fs layer
|
||||
4a1facbdf857: Waiting
|
||||
3c84f5c1ca0b: Pulling fs layer
|
||||
b1cc0c6d03ea: Waiting
|
||||
b989116cb4ee: Pulling fs layer
|
||||
f599001d1dac: Waiting
|
||||
334119c098d4: Waiting
|
||||
cfe38ec4fb3a: Pulling fs layer
|
||||
2411167b6874: Waiting
|
||||
b14dc82c93d9: Waiting
|
||||
a9b8ef092e47: Waiting
|
||||
66a08f34da9c: Waiting
|
||||
8294aa869476: Waiting
|
||||
d510763bc7fa: Waiting
|
||||
3c84f5c1ca0b: Waiting
|
||||
2b5be6c4f7e6: Waiting
|
||||
4693bfabf3cd: Waiting
|
||||
0f36e99efdcd: Waiting
|
||||
c8097f43c701: Waiting
|
||||
b989116cb4ee: Waiting
|
||||
32d0568ab58d: Waiting
|
||||
cfe38ec4fb3a: Waiting
|
||||
7994811847da: Waiting
|
||||
d80f1ecbeb8c: Waiting
|
||||
79ed78d42ca9: Verifying Checksum
|
||||
79ed78d42ca9: Download complete
|
||||
5785dfb2d94d: Verifying Checksum
|
||||
5785dfb2d94d: Download complete
|
||||
79ed78d42ca9: Pull complete
|
||||
5785dfb2d94d: Pull complete
|
||||
4f4fb700ef54: Verifying Checksum
|
||||
4f4fb700ef54: Download complete
|
||||
0e1d24786a23: Verifying Checksum
|
||||
0e1d24786a23: Download complete
|
||||
c83c61a504db: Verifying Checksum
|
||||
c83c61a504db: Download complete
|
||||
81801e5f6a47: Verifying Checksum
|
||||
81801e5f6a47: Download complete
|
||||
e832d0ac2449: Verifying Checksum
|
||||
e832d0ac2449: Download complete
|
||||
f41db59aec9f: Download complete
|
||||
4a1facbdf857: Verifying Checksum
|
||||
4a1facbdf857: Download complete
|
||||
b1cc0c6d03ea: Verifying Checksum
|
||||
b1cc0c6d03ea: Download complete
|
||||
8136a02ba8b6: Download complete
|
||||
8136a02ba8b6: Pull complete
|
||||
b14dc82c93d9: Verifying Checksum
|
||||
b14dc82c93d9: Download complete
|
||||
f599001d1dac: Download complete
|
||||
66a08f34da9c: Verifying Checksum
|
||||
66a08f34da9c: Download complete
|
||||
2411167b6874: Verifying Checksum
|
||||
2411167b6874: Download complete
|
||||
a9b8ef092e47: Verifying Checksum
|
||||
a9b8ef092e47: Download complete
|
||||
0f36e99efdcd: Verifying Checksum
|
||||
0f36e99efdcd: Download complete
|
||||
32d0568ab58d: Verifying Checksum
|
||||
32d0568ab58d: Download complete
|
||||
4693bfabf3cd: Verifying Checksum
|
||||
4693bfabf3cd: Download complete
|
||||
7994811847da: Verifying Checksum
|
||||
7994811847da: Download complete
|
||||
d80f1ecbeb8c: Verifying Checksum
|
||||
d80f1ecbeb8c: Download complete
|
||||
8294aa869476: Verifying Checksum
|
||||
8294aa869476: Download complete
|
||||
1dae32d336bd: Verifying Checksum
|
||||
1dae32d336bd: Download complete
|
||||
d510763bc7fa: Verifying Checksum
|
||||
d510763bc7fa: Download complete
|
||||
c8097f43c701: Verifying Checksum
|
||||
c8097f43c701: Download complete
|
||||
3c84f5c1ca0b: Verifying Checksum
|
||||
3c84f5c1ca0b: Download complete
|
||||
b989116cb4ee: Verifying Checksum
|
||||
b989116cb4ee: Download complete
|
||||
cfe38ec4fb3a: Download complete
|
||||
2b5be6c4f7e6: Verifying Checksum
|
||||
2b5be6c4f7e6: Download complete
|
||||
1dae32d336bd: Pull complete
|
||||
4f4fb700ef54: Pull complete
|
||||
0e1d24786a23: Pull complete
|
||||
c83c61a504db: Pull complete
|
||||
81801e5f6a47: Pull complete
|
||||
e832d0ac2449: Pull complete
|
||||
f41db59aec9f: Pull complete
|
||||
4a1facbdf857: Pull complete
|
||||
b1cc0c6d03ea: Pull complete
|
||||
334119c098d4: Verifying Checksum
|
||||
334119c098d4: Download complete
|
||||
334119c098d4: Pull complete
|
||||
b14dc82c93d9: Pull complete
|
||||
f599001d1dac: Pull complete
|
||||
66a08f34da9c: Pull complete
|
||||
2411167b6874: Pull complete
|
||||
a9b8ef092e47: Pull complete
|
||||
0f36e99efdcd: Pull complete
|
||||
32d0568ab58d: Pull complete
|
||||
4693bfabf3cd: Pull complete
|
||||
7994811847da: Pull complete
|
||||
d80f1ecbeb8c: Pull complete
|
||||
8294aa869476: Pull complete
|
||||
d510763bc7fa: Pull complete
|
||||
2b5be6c4f7e6: Pull complete
|
||||
c8097f43c701: Pull complete
|
||||
3c84f5c1ca0b: Pull complete
|
||||
b989116cb4ee: Pull complete
|
||||
cfe38ec4fb3a: Pull complete
|
||||
Digest: sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1
|
||||
Status: Downloaded newer image for vllm/vllm-openai:latest
|
||||
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Traceback (most recent call last):
|
||||
File "/q/quant_a16_datafree.py", line 36, in <module>
|
||||
model, tok = ref.load_model(a.model)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 73, in load_model
|
||||
tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 747, in from_pretrained
|
||||
config = AutoConfig.from_pretrained(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/configuration_auto.py", line 419, in from_pretrained
|
||||
return config_class.from_dict(config_dict, **unused_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 878, in from_dict
|
||||
config = cls(**config_dict)
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 275, in init_with_validate
|
||||
initial_init(self, *args, **kwargs) # type: ignore [call-arg]
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 131, in __init__
|
||||
self.__post_init__(**additional_kwargs)
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/configuration_gemma4.py", line 348, in __post_init__
|
||||
self.text_config = Gemma4TextConfig(**self.text_config)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 276, in init_with_validate
|
||||
cls.validate(self) # type: ignore [attr-defined]
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 251, in validate
|
||||
validator(self)
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 476, in validate_architecture
|
||||
hasattr(self, "head_dim")
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 464, in __getattribute__
|
||||
return super().__getattribute__(key)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py", line 266, in __getattribute__
|
||||
raise AmbiguousGlobalPerLayerAttributeError(
|
||||
transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].head_dim). To read the global config value from config.head_dim anyway, set `allow_global_per_layer_attribute_access` to `True` on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.
|
||||
=== 2026-09-10T09:28:02-07:00 END rc=1 size=512
|
||||
=== 2026-09-10T10:40:09-07:00 START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned)
|
||||
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
|
||||
transformers 5.14.1
|
||||
loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 5430.71it/s]
|
||||
NVFP4 oneshot (DATA-FREE): scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16
|
||||
2026-09-10T17:40:48.5554 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
|
||||
2026-09-10T17:40:50.1494 | reset | INFO - Compression lifecycle reset
|
||||
2026-09-10T17:40:50.1782 | from_modifiers | INFO - Creating recipe from modifiers
|
||||
Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 2863.72it/s]
|
||||
2026-09-10T17:40:50.3684 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
|
||||
2026-09-10T17:40:50.3685 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
|
||||
2026-09-10T17:42:45.7622 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
|
||||
saving -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
Compressing model: 100%|██████████| 410/410 [00:11<00:00, 36.91it/s]
|
||||
Writing model shards: 100%|██████████| 2/2 [00:08<00:00, 4.01s/it]
|
||||
Dispatching model: 100%|██████████| 1763/1763 [00:00<00:00, 58662.76it/s]
|
||||
DONE
|
||||
=== 2026-09-10T10:43:16-07:00 END rc=0 size=19G
|
||||
@@ -0,0 +1,83 @@
|
||||
### verify_quant.py — new v2 heretic quant vs the 2026-08-21 known-good canonical quant
|
||||
$ sudo -n python3 verify_quant.py <new> <august-known-good>
|
||||
|
||||
======================================================================
|
||||
/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY)
|
||||
format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed
|
||||
text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4
|
||||
tensor dtypes by family:
|
||||
embeddings BF16x1
|
||||
language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors]
|
||||
norms BF16x361
|
||||
vision_tower BF16x356
|
||||
|
||||
======================================================================
|
||||
/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16
|
||||
group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY)
|
||||
format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed
|
||||
text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4
|
||||
tensor dtypes by family:
|
||||
embeddings BF16x1
|
||||
language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors]
|
||||
norms BF16x361
|
||||
vision_tower BF16x356
|
||||
|
||||
### post_quant_gemma4.py --check — v2 heretic output
|
||||
[CHECK] src=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
[CHECK] out=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
|
||||
-- step 1: MTP graft
|
||||
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
||||
-- step 2: restore processor_config.json + preprocessor_config.json
|
||||
processor_config.json already present and identical to source
|
||||
preprocessor_config.json already present and correct
|
||||
-- step 4: confirm saved tokenizer.json has truncation: null
|
||||
truncation is null -- clean
|
||||
|
||||
[CHECK] done rc=0
|
||||
|
||||
### post_quant_gemma4.py --check — A4B output (after the truncation fix)
|
||||
[CHECK] src=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16
|
||||
[CHECK] out=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
|
||||
-- step 1: MTP graft
|
||||
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
||||
-- step 2: restore processor_config.json + preprocessor_config.json
|
||||
processor_config.json already present and identical to source
|
||||
preprocessor_config.json already present and correct
|
||||
-- step 4: confirm saved tokenizer.json has truncation: null
|
||||
truncation is null -- clean
|
||||
|
||||
[CHECK] done rc=0
|
||||
|
||||
### post_quant_gemma4.py --check — 2026-08-21 known-good tree (POSITIVE CONTROL, must be clean)
|
||||
[CHECK] src=/tank/aimodels/meromero-v2-nvfp4-work/src
|
||||
[CHECK] out=/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16
|
||||
|
||||
-- step 1: MTP graft
|
||||
N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0
|
||||
-- step 2: restore processor_config.json + preprocessor_config.json
|
||||
processor_config.json already present and identical to source
|
||||
preprocessor_config.json already present and correct
|
||||
-- step 4: confirm saved tokenizer.json has truncation: null
|
||||
truncation is null -- clean
|
||||
|
||||
[CHECK] done rc=0
|
||||
|
||||
### output tree
|
||||
total 19762161
|
||||
drwxr-xr-x 2 infra-ops infra-ops 13 Sep 10 10:43 .
|
||||
drwxrwxr-x 62 llmuser llm 99 Sep 10 08:59 ..
|
||||
-rw-r--r-- 1 root root 16934 Sep 10 10:43 chat_template.jinja
|
||||
-rw-r--r-- 1 root root 19419 Sep 10 10:43 config.json
|
||||
-rw-r--r-- 1 root root 204 Sep 10 10:43 generation_config.json
|
||||
-rw------- 1 root root 19994044576 Sep 10 10:43 model-00001-of-00002.safetensors
|
||||
-rw------- 1 root root 452731960 Sep 10 10:43 model-00002-of-00002.safetensors
|
||||
-rw-r--r-- 1 root root 209808 Sep 10 10:43 model.safetensors.index.json
|
||||
-rw-r--r-- 1 root root 375 Sep 10 10:43 preprocessor_config.json
|
||||
-rw-r--r-- 1 root root 1689 Aug 12 02:28 processor_config.json
|
||||
-rw-r--r-- 1 root root 430 Sep 10 10:43 recipe.yaml
|
||||
-rw-r--r-- 1 root root 2819 Sep 10 10:43 tokenizer_config.json
|
||||
-rw-r--r-- 1 root root 32169780 Sep 10 10:43 tokenizer.json
|
||||
19G /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
# Two NVFP4A16 quants, 2026-09-10. Operator: "run our own quant. w4a16 vllm
|
||||
# servable, vision towers intact, mtp if applicable."
|
||||
# W4A16 -> --scheme NVFP4A16 (weight-only, NOT plain NVFP4/W4A4)
|
||||
# vision intact -> recipe ignore-list keeps vision/audio towers BF16
|
||||
# MTP -> N/A: verified 0 mtp tensors in BOTH bf16 sources
|
||||
# GPU1 not GPU0: the script onloads one layer at a time (GPU-light) but its own
|
||||
# docstring warns of OOM when the card is not fairly free. GPU0 has 4.6 GiB spare
|
||||
# (gen + mog-sec resident); GPU1 has ~19.3 GiB.
|
||||
set -uo pipefail
|
||||
WORK=/tank/aimodels/meromero-v2-nvfp4-work
|
||||
CALIB=/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl
|
||||
exec >> /home/infra-ops/quant/batch.log 2>&1
|
||||
|
||||
run_one () {
|
||||
local name="$1" src="$2" out="$3"
|
||||
echo "=== $(date -Is) START $name -> $out"
|
||||
docker rm -f meromero-quant >/dev/null 2>&1
|
||||
docker run --rm --name meromero-quant --gpus '"device=1"' --ipc host \
|
||||
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
-v /tank/aimodels:/tank/aimodels \
|
||||
--entrypoint bash vllm/vllm-openai:latest -c "
|
||||
set -e
|
||||
pip install -q llmcompressor==0.13.0 tiktoken sentencepiece 2>&1 | tail -1
|
||||
python3 $WORK/quant_nvfp4_gemma.py \
|
||||
--model '$src' --calib '$CALIB' \
|
||||
--num-samples 512 --seqlen 8192 --scheme NVFP4A16 \
|
||||
--out '$out'
|
||||
"
|
||||
local rc=$?
|
||||
echo "=== $(date -Is) END $name rc=$rc size=$(du -sh "$out" 2>/dev/null | cut -f1)"
|
||||
}
|
||||
|
||||
run_one A4B-heretic \
|
||||
/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16 \
|
||||
/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
run_one v2-31B-heretic \
|
||||
/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 \
|
||||
/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
echo "=== $(date -Is) BATCH DONE"
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# NVFP4A16 quant of the DogOnKeyboard v2-31B heretic (abliterated) Gemma-4.
|
||||
#
|
||||
# Attempt 5. Four things had to be fixed to get here and all four are load-bearing:
|
||||
# 1. config.json was missing num_global_key_value_heads / global_head_dim while
|
||||
# declaring attention_k_eq_v -- patched from zerofata's canonical values after
|
||||
# shape-verifying the checkpoint (full-attn k_proj [2048,5376] = 4x512).
|
||||
# 2. config.json carried a per_layer_config block that transformers 5.14.1 refuses
|
||||
# to read globally. Removed; it was exactly redundant with (1). See
|
||||
# patch_perlayer.py for the redundancy proof.
|
||||
# 3. No processor_config.json in the upload -> llmcompressor demands a processor
|
||||
# whenever a dataset is passed. Dropping the dataset removes the demand, and
|
||||
# costs nothing: NVFP4A16 is weight-only and runs a DataFreePipeline.
|
||||
# 4. The reference module argparses at import; quant_a16_datafree.py feeds it
|
||||
# placeholder argv and restores the real one.
|
||||
#
|
||||
# IMAGE PINNED BY DIGEST, deliberately. `vllm/vllm-openai:latest` was re-pulled
|
||||
# between attempt 3 and attempt 4 and moved transformers 5.12.1 -> 5.16.1, which is
|
||||
# why attempt 4's error looked like a new config problem and was not. Note the
|
||||
# effective transformers is 5.14.1 either way -- llmcompressor 0.13.0 pins it, and
|
||||
# it is 5.14.1 the config had to be made readable by.
|
||||
IMAGE=vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1
|
||||
SRC=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16
|
||||
OUT=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16
|
||||
set -uo pipefail
|
||||
exec >> /home/infra-ops/quant/v2.log 2>&1
|
||||
echo "=== $(date -Is) START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned)"
|
||||
docker rm -f meromero-quant >/dev/null 2>&1
|
||||
docker run --rm --name meromero-quant --gpus '"device=1"' --ipc host \
|
||||
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
-v /tank/aimodels:/tank/aimodels -v /home/infra-ops/quant:/q \
|
||||
--entrypoint bash "$IMAGE" -c "
|
||||
set -e
|
||||
pip install -q llmcompressor==0.13.0 tiktoken sentencepiece 2>&1 | tail -1
|
||||
python3 -c 'import transformers; print(\"transformers\", transformers.__version__, flush=True)'
|
||||
python3 /q/quant_a16_datafree.py --model $SRC --out $OUT --scheme NVFP4A16
|
||||
"
|
||||
rc=$? # captured BEFORE any other command -- an earlier wrapper read $? after an echo and always said 0
|
||||
echo "=== $(date -Is) END rc=$rc size=$(du -sh $OUT 2>/dev/null | cut -f1)"
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Do the CHECKPOINT's tensor shapes agree with num_global_key_value_heads=4?
|
||||
|
||||
Patching a config to satisfy a constructor is only safe if the weights already have
|
||||
the shape the patched value implies. If the abliteration reshaped attention, the
|
||||
patch would silence the error and produce a quietly wrong quant -- worse than the
|
||||
crash, because it ships.
|
||||
|
||||
For a full-attention layer, k_proj/v_proj out-features == num_kv_heads * head_dim.
|
||||
zerofata's canonical v2: num_global_key_value_heads=4, global_head_dim=512
|
||||
=> expected out-features 4 * 512 = 2048 on the GLOBAL (full-attention) layers.
|
||||
"""
|
||||
import json, sys
|
||||
from safetensors import safe_open
|
||||
from pathlib import Path
|
||||
|
||||
def probe(label, root, n_global_kv, global_head_dim, n_kv, head_dim):
|
||||
root = Path(root)
|
||||
idx = json.load(open(root / "model.safetensors.index.json"))["weight_map"]
|
||||
types = json.load(open(root / "config.json"))["text_config"]["layer_types"]
|
||||
full = [i for i, t in enumerate(types) if t == "full_attention"][:2]
|
||||
slide = [i for i, t in enumerate(types) if t == "sliding_attention"][:2]
|
||||
print(f" -- {label}")
|
||||
print(f" expected FULL k/v out-features = {n_global_kv} x {global_head_dim} = {n_global_kv*global_head_dim}"
|
||||
if n_global_kv and global_head_dim else " expected FULL = (config lacks the fields)")
|
||||
print(f" expected SLIDE k/v out-features = {n_kv} x {head_dim} = {n_kv*head_dim}")
|
||||
for tag, idxs in (("full ", full), ("slide", slide)):
|
||||
for li in idxs:
|
||||
for proj in ("k_proj", "v_proj"):
|
||||
key = f"model.language_model.layers.{li}.self_attn.{proj}.weight"
|
||||
if key not in idx:
|
||||
key = f"language_model.model.layers.{li}.self_attn.{proj}.weight"
|
||||
if key not in idx:
|
||||
cand = [k for k in idx if f"layers.{li}.self_attn.{proj}" in k]
|
||||
key = cand[0] if cand else None
|
||||
if not key:
|
||||
print(f" {tag} L{li} {proj}: KEY NOT FOUND"); continue
|
||||
with safe_open(root / idx[key], framework="pt") as f:
|
||||
shape = f.get_slice(key).get_shape()
|
||||
print(f" {tag} L{li} {proj}: shape {shape} out-features={shape[0]}")
|
||||
|
||||
cfg = json.load(open("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16/config.json"))["text_config"]
|
||||
good = json.load(open("/tank/aimodels/meromero-v2-nvfp4-work/src/config.json"))["text_config"]
|
||||
print(f" canonical (zerofata): num_global_key_value_heads={good.get('num_global_key_value_heads')} "
|
||||
f"global_head_dim={good.get('global_head_dim')} num_key_value_heads={good.get('num_key_value_heads')} head_dim={good.get('head_dim')}")
|
||||
probe("zerofata v2 (canonical)", "/tank/aimodels/meromero-v2-nvfp4-work/src",
|
||||
good.get("num_global_key_value_heads"), good.get("global_head_dim"),
|
||||
good.get("num_key_value_heads"), good.get("head_dim"))
|
||||
probe("DogOnKeyboard v2 (to patch)", "/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16",
|
||||
good.get("num_global_key_value_heads"), good.get("global_head_dim"),
|
||||
cfg.get("num_key_value_heads"), cfg.get("head_dim"))
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Reproduce the ACTUAL failing call, not a paraphrase of it.
|
||||
|
||||
quant_a16_datafree.py dies inside `AutoTokenizer.from_pretrained(path,
|
||||
trust_remote_code=True)`. A bare `AutoConfig.from_pretrained(dir)` does NOT
|
||||
reproduce it -- I checked, and all four config variants sailed through. So the
|
||||
trigger lives in the tokenizer path, and testing the config alone would have sent
|
||||
me off patching a file that was never the problem.
|
||||
|
||||
POSITIVE CONTROL, and it is the whole point of this script: zerofata's canonical
|
||||
v2 tree quantized cleanly on 2026-08-21. If it now fails on this same call, the
|
||||
config is exonerated and the toolchain moved under us -- `vllm/vllm-openai:latest`
|
||||
was re-pulled mid-campaign and carries transformers 5.16.1 where the successful
|
||||
August run had 5.12.1.
|
||||
"""
|
||||
import json, os, tempfile, traceback
|
||||
from pathlib import Path
|
||||
|
||||
import transformers
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
print(f"transformers {transformers.__version__}", flush=True)
|
||||
|
||||
HERETIC = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16")
|
||||
CANON = Path("/tank/aimodels/meromero-v2-nvfp4-work/src")
|
||||
|
||||
# Tokenizer loading reads config.json, so a variant needs the whole tree. Symlink
|
||||
# everything, then overwrite the one file under test.
|
||||
def tree(name, src, mutate=None):
|
||||
d = Path(tempfile.mkdtemp(prefix=f"tok-{name}-"))
|
||||
for f in src.iterdir():
|
||||
if f.is_file():
|
||||
os.symlink(f, d / f.name)
|
||||
if mutate is not None:
|
||||
cfg = json.loads((src / "config.json").read_text())
|
||||
mutate(cfg)
|
||||
(d / "config.json").unlink()
|
||||
(d / "config.json").write_text(json.dumps(cfg))
|
||||
return d
|
||||
|
||||
|
||||
def drop_plc(c):
|
||||
c["text_config"].pop("per_layer_config", None)
|
||||
|
||||
|
||||
def force_global(c):
|
||||
c["text_config"]["allow_global_per_layer_attribute_access"] = True
|
||||
|
||||
|
||||
cases = [
|
||||
("A-canonical-POSITIVE-CONTROL", tree("canon", CANON)),
|
||||
("B-heretic-asis", tree("heretic", HERETIC)),
|
||||
("C-heretic-drop-per_layer_config", tree("drop", HERETIC, drop_plc)),
|
||||
("D-heretic-force-global-access", tree("force", HERETIC, force_global)),
|
||||
("E-canonical-force-global-access", tree("canonforce", CANON, force_global)),
|
||||
]
|
||||
|
||||
for name, d in cases:
|
||||
print(f"\n=== {name} ===", flush=True)
|
||||
try:
|
||||
tok = AutoTokenizer.from_pretrained(d, trust_remote_code=True)
|
||||
except Exception as e:
|
||||
tb = traceback.format_exc().strip().splitlines()
|
||||
print(f" FAILED {type(e).__name__}")
|
||||
print(" " + "\n ".join(tb[-4:]))
|
||||
continue
|
||||
trunc = getattr(tok, "truncation_side", None)
|
||||
print(f" OK {type(tok).__name__} vocab={len(tok)} truncation_side={trunc}")
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Did the quant actually do what the recipe says, on the tensors it claims?
|
||||
|
||||
`rc=0` and a plausible file size prove neither. The two things the operator asked
|
||||
for -- W4A16, vision towers intact -- are properties of the tensor table, so read
|
||||
the tensor table. Parses safetensors headers directly (u64 length + JSON), so no
|
||||
torch, no GPU, and no 20 GB load.
|
||||
|
||||
Checks, per module family:
|
||||
* language-model Linears -> must be NVFP4-packed (uint8 blobs + *_scale companions)
|
||||
* vision / audio towers -> must still be BF16, i.e. PRESERVED not dropped
|
||||
* embeddings / lm_head / norms -> BF16 per the ignore list
|
||||
|
||||
Run it against a known-good tree as well. A checker that has only ever seen the
|
||||
tree it was written for cannot tell "correct" from "blind".
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import struct
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def tensors(root: Path):
|
||||
"""Yield (name, dtype, shape) for every tensor in a sharded or single-file tree."""
|
||||
idx = root / "model.safetensors.index.json"
|
||||
files = sorted({Path(v) for v in json.loads(idx.read_text())["weight_map"].values()}) \
|
||||
if idx.exists() else [Path("model.safetensors")]
|
||||
for f in files:
|
||||
p = root / f
|
||||
with p.open("rb") as fh:
|
||||
n = struct.unpack("<Q", fh.read(8))[0]
|
||||
head = json.loads(fh.read(n))
|
||||
for name, meta in head.items():
|
||||
if name == "__metadata__":
|
||||
continue
|
||||
yield name, meta["dtype"], meta["shape"]
|
||||
|
||||
|
||||
def family(name: str) -> str:
|
||||
if "vision_tower" in name or "embed_vision" in name:
|
||||
return "vision_tower"
|
||||
if "audio_tower" in name or "embed_audio" in name:
|
||||
return "audio_tower"
|
||||
if "multi_modal_projector" in name or "mm_projector" in name:
|
||||
return "projector"
|
||||
if "embed_tokens" in name:
|
||||
return "embeddings"
|
||||
if name.startswith("lm_head") or ".lm_head" in name:
|
||||
return "lm_head"
|
||||
if "norm" in name:
|
||||
return "norms"
|
||||
if "language_model" in name or ".layers." in name:
|
||||
return "language_model"
|
||||
return "other"
|
||||
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("trees", nargs="+")
|
||||
a = ap.parse_args()
|
||||
|
||||
for t in a.trees:
|
||||
root = Path(t)
|
||||
print(f"\n{'='*70}\n{root}")
|
||||
cfg = json.loads((root / "config.json").read_text())
|
||||
q = cfg.get("quantization_config", {})
|
||||
groups = q.get("config_groups", {})
|
||||
for gname, g in groups.items():
|
||||
w = g.get("weights", {})
|
||||
i = g.get("input_activations")
|
||||
print(f" {gname}: weights num_bits={w.get('num_bits')} type={w.get('type')} "
|
||||
f"strategy={w.get('strategy')} | input_activations="
|
||||
f"{'None (WEIGHT-ONLY)' if i is None else i}")
|
||||
print(f" format={q.get('format')} kv_cache_scheme={q.get('kv_cache_scheme')} "
|
||||
f"status={q.get('quantization_status')}")
|
||||
tc = cfg.get("text_config", {})
|
||||
print(f" text_config: per_layer_config={'PRESENT' if 'per_layer_config' in tc else 'absent'}"
|
||||
f" head_dim={tc.get('head_dim')} global_head_dim={tc.get('global_head_dim')}"
|
||||
f" num_key_value_heads={tc.get('num_key_value_heads')}"
|
||||
f" num_global_key_value_heads={tc.get('num_global_key_value_heads')}")
|
||||
|
||||
by = defaultdict(lambda: defaultdict(int))
|
||||
packed = defaultdict(int)
|
||||
for name, dt, shape in tensors(root):
|
||||
f = family(name)
|
||||
by[f][dt] += 1
|
||||
if name.endswith("weight_packed") or name.endswith("weight_scale"):
|
||||
packed[f] += 1
|
||||
print(" tensor dtypes by family:")
|
||||
for f in sorted(by):
|
||||
dts = ", ".join(f"{d}x{c}" for d, c in sorted(by[f].items()))
|
||||
note = f" [{packed[f]} packed/scale tensors]" if packed[f] else ""
|
||||
print(f" {f:16} {dts}{note}")
|
||||
Reference in New Issue
Block a user