diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 86b7b6d..145a2b3 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -471,6 +471,47 @@ Reference: `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (linearize_moe 11,520 expert Linears + post-steps; the published `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16` recipe replicated, 222→252 ignore entries with audio/norm/router regexes added). +**So for any `*A16` scheme, do not pass a dataset at all** — not a shorter one, none. It removes +the §3.14 tokenizer bake-in *and* llm-compressor's "initialize model processor ... required when a +dataset is provided" demand, which is fatal on any upload that ships no `processor_config.json`. +Both of those cost an attempt on MeroMero v2 (2026-09-10); dropping the corpus costs nothing, +because a `DataFreePipeline` was never going to read it. Driver: +`services/meromero-quant/quant_a16_datafree.py`. **Confirmed twice more the same day**: the A4B +heretic quant, run *with* the corpus, shipped `max_length: 8192` in its `tokenizer.json`; the v2 +dense, run without it, came out `truncation: null`. + +### 3.17 ⭐⭐ The transformers you measured is not the transformers that ran — llm-compressor pins it + +**Measured 2026-09-10, MeroMero v2-31B, and it cost a full misdiagnosis.** The quant died in +`AutoTokenizer.from_pretrained` with +`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`. The obvious reading +was that the source config carried a `per_layer_config` key the canonical one lacked, so that key +was the defect. It was not. + +`pip install llmcompressor==0.13.0` **downgrades transformers underneath you** — 5.16.1 → 5.14.1 in +the `vllm/vllm-openai` image. The config had been serialized *by* 5.16.1, which materializes +`per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard +but not the gemma4 resolver, so it refuses the global read. Under the image's own 5.16.1 the very +same config loads fine, which is exactly what makes this class expensive: **the version you print +at the top of the script is not the version the quant runs on.** + +- **Print the version AFTER the install**, in the same container, and put it in the log. + `python3 -c 'import transformers; print(transformers.__version__)'` as a pipeline step, not a + thing you check by hand once. +- **Pin the image by digest for the length of a campaign.** `:latest` was re-pulled between + attempts 3 and 4 of this run and moved the toolchain mid-diagnosis, so the same command produced + a different error for reasons that had nothing to do with the change under test. +- **Reproduce the actual failing call, not a paraphrase.** A bare `AutoConfig.from_pretrained` did + not reproduce this at all — the trigger was reached through `AutoTokenizer`. Testing the config + in isolation would have "cleared" it. +- **Keep a known-good tree as the positive control.** zerofata's canonical v2, quantized cleanly + three weeks earlier, is what separated "this config is malformed" from "this toolchain moved". + Without it, four green variants and one red one are just noise. + +Related but distinct from §3.4, which is about version deadlocks you can *see*. This one is a +silent downgrade inside a line you already trusted. Instruments: `services/meromero-quant/` +(`tok_repro.py`, `perlayer_test.py`). + ### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer **Symptom (on a newer transformers, at startup, on a vision model):** @@ -579,6 +620,10 @@ Never optional, always in this order, and the last one **verifies rather than as Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`. +On Gemma-4 steps 1 and 3 are N/A — the family ships no MTP head at all — which leaves 2 and 4, and +4 is the one that fires. `services/meromero-quant/post_quant_gemma4.py` runs them idempotently with +a `--check` mode; point it at a tree you already trust before you trust its verdict on a new one. + ### 4.4 Test on a temp port, never on the live seat Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and @@ -698,6 +743,7 @@ above, and where the two disagree, **this file wins**. |---|---| | `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. | | `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) | +| `services/meromero-quant/` | NVFP4A16 on Gemma-4 (MeroMero A4B + v2-31B ablits): the five-attempt failure chain, the pinned-transformers trap (§3.17), and the GPU-free verification instruments. | | `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) | | `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it | | `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 | diff --git a/persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md b/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md similarity index 52% rename from persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md rename to persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md index 5494725..6157e3f 100644 --- a/persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md +++ b/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md @@ -1,7 +1,8 @@ -# `[2026-09-10]` MeroMero acquisition — the A4B quant landed, the v2 dense has failed FOUR times +# `[2026-09-10]` MeroMero — both quants landed; the v2 dense took five attempts -Operator wanted a MeroMero seat. Getting there cost four quant attempts and corrected two of my own -wrong hypotheses, so the failure chain is the durable part. +Operator wanted a MeroMero seat. Getting there cost five quant attempts and corrected three wrong +hypotheses (two of the previous session's, one of mine), so the failure chain is the durable part. +**Both outputs now exist and are verified against a known-good tree. Neither is serving yet.** ## The family, because I got it wrong first @@ -60,13 +61,62 @@ playbook §3.16: weight-only A16 runs a `DataFreePipeline` and never touches the truncation cap into the shipped tokenizer. Removing it kills both for zero loss. 3. **My own bug**: the reference module runs argparse with `required=True` at IMPORT, so blanking `sys.argv` still exited 2. Placeholder args, real argv restored after. -4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — OPEN. The - DogOnKeyboard config carries a `per_layer_config` key that zerofata's lacks (it was the one key in - the diff I noted and did not chase). transformers refuses global access to `head_dim` on a - heterogeneous config; the fix is likely `allow_global_per_layer_attribute_access=True`, with the - warning that a caller assuming homogeneity may then read the wrong value. +4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — RESOLVED, + and **not what it looked like**. See "The attempt-4 trap" below. +5. Clean. `rc=0`, 19 G, 3m07s. ⚠ **My wrapper reported `rc=0` on a failed run** because it read `$?` after an `echo`. A wrapper that reports success on failure is the false-reassurance class; fixed to capture `$?` immediately. + +## The attempt-4 trap — the toolchain moved, the config was fine + +The standing hypothesis was that DogOnKeyboard's `per_layer_config` key was the defect and +`allow_global_per_layer_attribute_access=True` was the fix. Both halves were wrong, and the second +half would have shipped a risk for no reason. + +**`pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 → 5.14.1.** The config was +serialized *by* 5.16.1, which materializes `per_layer_config` out of `global_head_dim` + +`layer_types`; 5.14.1 has the heterogeneity guard but not the gemma4 resolver, so it refuses the +global read. Under the image's own 5.16.1 the identical config loads fine. On top of that, +`vllm/vllm-openai:latest` was re-pulled *during attempt 4 and in no earlier run* — the pull line is +in that block alone — so the error changed for reasons unrelated to anything under test. + +Two things made this findable, and neither was inspection: + +- **Reproducing the real call.** A bare `AutoConfig.from_pretrained` does not reproduce it; the + trigger is reached through `AutoTokenizer`. Testing the config alone would have cleared it. +- **A known-good positive control.** zerofata's canonical v2, quantized cleanly on 2026-08-21, is + what separated "this upload is malformed" from "this toolchain moved". Four green variants and one + red one are noise without it. + +**Fix: drop `per_layer_config`, don't force global access.** It is exactly redundant — keys are +precisely the ten `full_attention` layer indices, sole value `(512, 4)`, verbatim the global fields. +Forcing instead leaves the config heterogeneous and makes `config.head_dim` answer 256 to every +caller including the ones building the 512-wide layers; geometry survived it in a meta-device check, +but llmcompressor's onloading is an unaudited caller and that is what the warning is about. The +patch re-proves the redundancy at apply time and refuses if it ever stops holding. + +## What landed, and what is verified + +- `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G**, and its tensor table is **identical family for + family and count for count to the 2026-08-21 canonical quant**: 410 U8 packed + 410 F8_E4M3 + + 410 F32 scales on the LM Linears, **356 BF16 vision-tower tensors preserved**, + `input_activations=None` (genuinely A16). Shard sizes match that tree byte for byte. +- CPU load-and-generate: 0 tensors left on meta, decompresses, emits coherent prose. n=1, greedy, + 24 tokens — an "is it wired up" check and nothing more. +- ⚠ **The A4B output had the §3.14 truncation cap baked in** (`max_length: 8192`), because it was + quantized *with* the corpus. Caught and fixed; backup at `tokenizer.json.bak-pre-truncfix`. The v2, + run data-free, came out `truncation: null`. + +## Still owed + +**The §4.4 serve test has NOT run.** GPU1 has 19.9 GB free against 19.5 GB of weights, so it cannot +happen without displacing a live seat — operator's call. Until it does, *"vllm servable"* is +unverified for this tree, and the dense 31B's 60-layer / kv-16 geometry still does not fit 262k on +GPU1 beside the current tenants regardless. + +Instruments and the full write-up: `services/meromero-quant/`. General lessons: +`docs/pfi/model-quantization-playbook.md` §3.16, **§3.17 (new)**, §4.3. + Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]] diff --git a/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md b/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md index bda3529..5a7a489 100644 --- a/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md +++ b/persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md @@ -116,4 +116,4 @@ resolved at run start AND end. Handoff bundle for adjudication at `/mnt/smithy/handoff/r49/`. -Related: [[2026-09-10-meromero-acquisition-and-four-quant-failures]] +Related: [[2026-09-10-meromero-quants-and-the-pinned-transformers-trap]] diff --git a/persistent-memory.md b/persistent-memory.md index 5fc5d54..2398f3a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-10 10:25 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; MeroMero A4B quantized, v2 dense blocked at attempt 4; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_ +_Last updated: 2026-09-10 11:00 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; **MeroMero BOTH quants landed** — v2 dense on attempt 5, serve test still owed; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -141,12 +141,19 @@ _As of 2026-09-10 10:25 PT._ ### MeroMero seats -- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, quantized in-house, W4A16. - ⚠ Missing `preprocessor_config.json`; §4.3 post-steps owed before it serves. -- ⛔ **v2-31B dense quant has failed FOUR times**, currently on - `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute` — DogOnKeyboard's - config carries a `per_layer_config` key zerofata's lacks. Likely fix - `allow_global_per_layer_attribute_access=True`. → `persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md` +- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, W4A16, §4.3 post-steps DONE. + ⚠ It had the §3.14 **truncation cap baked into `tokenizer.json`** (`max_length: 8192`, because it + was quantized *with* the corpus). Fixed; backup `tokenizer.json.bak-pre-truncfix`. +- ✅ `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G, landed on attempt 5.** Tensor table identical + family-for-family to the 2026-08-21 canonical quant; **356 BF16 vision tensors preserved**; + `input_activations=None` (genuinely A16). CPU load+generate coherent, 0 tensors on meta. + ⚠ Attempt 4's `AmbiguousGlobalPerLayerAttributeError` was **NOT a config defect** — llmcompressor + 0.13.0 downgrades transformers 5.16.1→5.14.1, and `:latest` moved mid-campaign. Fix was to DROP + the redundant `per_layer_config`, not to force global access. + → `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md`, + `services/meromero-quant/`, playbook **§3.17 (new)** +- ⛔ **NEITHER quant has had its §4.4 serve test** — GPU1 has 19.9 GB free against 19.5 GB of v2 + weights, so it needs a live seat displaced. Operator's call; *"vllm servable"* unverified until then. - ⚠ **The A4B is 30 layers / kv 8 — Pfish-6's geometry**, so it fits 262k in the existing KV budget. The dense 31B is 60/16, ~4x KV per token, and does NOT fit 262k on GPU1 beside the other seats. - Pfish-6 remains the standing seat on ana-ml2 `:8021`. Nothing repointed, no `rp-fast` alias exists. @@ -174,7 +181,7 @@ _As of 2026-09-10 10:25 PT._ - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` - `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`. -- `[2026-09-10]` **MeroMero: A4B MoE quantized in-house at W4A16; the v2 dense has failed four times and is OPEN.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* → `persistent-memory.d/2026-09-10-meromero-acquisition-and-four-quant-failures.md` +- `[2026-09-10]` **MeroMero: BOTH quants landed in-house at W4A16 — A4B first try, v2 dense on attempt 5.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* The durable lesson is **§3.17**: `pip install llmcompressor` silently pins transformers down a version, so attempt 4's error was a moved toolchain, not the malformed upload it looked like — a known-good positive control is what told them apart. Serve test still owed. → `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md` - `[2026-09-10]` **althing 3.6.2 deployed — post office + both heralds — and the fleet has TWO herald nodes, not seven.** Ask the post office's `nodes` table, not the box inventory. Cost a self-inflicted ~12 min bus outage. → `persistent-memory.d/2026-09-10-althing-362-rollout.md` - `[2026-09-10]` **A grep over a log that records your greps counts itself.** I reported forseti's drop defect as reproducing here with 3 drops in 21 s; the session had **zero**. Searching transcripts writes the search term into them. Filter by `"type":"system"` provenance, never content. Generalises to any instrument that can see itself. Auto-memory `feedback_grep_over_a_log_that_records_your_greps`. - `[2026-09-10]` **Operator-directed purges: 466 GB (qwopus + huihui 122B bf16) and 107.8 GB Docker on ana-ml2.** Serving/rollback artifacts and qwopus's MTP head verified intact after. ⚠ `/tank` is OUTSIDE restic, so both were final. diff --git a/services/meromero-quant/README.md b/services/meromero-quant/README.md new file mode 100644 index 0000000..766b107 --- /dev/null +++ b/services/meromero-quant/README.md @@ -0,0 +1,138 @@ +# MeroMero NVFP4A16 quants (Gemma-4) — instruments and the failure chain + +Two in-house W4A16 quants of the abliterated MeroMero models, 2026-09-10. The +operator's brief was three clauses long: *"w4a16 vllm servable, vision towers +intact, mtp if applicable."* Every published quant of these models is W4A4 (our own +measured long-context collapse) or, for v2, does not exist at all — 0 of 27 v2 repos. + +| output | source | result | +|---|---|---| +| `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` | `DogOnKeyboard` A4B ablit | **16 G**, first try, 2m08s | +| `G4-MeroMero-v2-31B-heretic-NVFP4A16` | `DogOnKeyboard` v2-31B ablit | **19 G**, attempt **5** | + +Both live on ana-ml2 under `/tank/aimodels/`. Neither is serving yet — see *Owed*. + +## What the v2 dense cost, and why each layer mattered + +Five attempts, five different causes. The order matters because each one masked the +next. + +1. **`num_key_value_heads` is None** at `Gemma4TextAttention.__init__`. The uploader + set `attention_k_eq_v: true` but omitted `num_global_key_value_heads` and + `global_head_dim` — a malformed upload, not a toolchain problem. Patched from + zerofata's canonical values (4 / 512) after shape-verifying the checkpoint + (`shape_verify.py`): full-attn `k_proj [2048,5376]` = 4×512, sliding + `[4096,5376]` = 16×256, identical to canonical. + ⚠ **A 2-layer truncation test PASSED and hid this.** The failing branch is chosen + per layer type and only `full_attention` layers take it. Testing each layer type + individually found it in seconds. +2. **`initialize model processor ... required when a dataset is provided`.** This + upload ships no `processor_config.json`. Rather than supply one, the dataset was + dropped entirely — NVFP4A16 is weight-only and runs a `DataFreePipeline`, so the + corpus was never read anyway (playbook §3.16), and passing one also bakes a + truncation cap into the shipped tokenizer (§3.14). Removing it kills both for + zero loss. `quant_a16_datafree.py`. +3. **Our own bug**: the reference module argparses at import with `required=True`, + so blanking `sys.argv` still exited 2. Placeholder argv, real one restored after. +4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`.** + See below — this one was not what it looked like. +5. Clean. `rc=0`, 19 G, 3m07s. + +## The attempt-4 trap: the transformers you measured is not the one that ran + +The obvious reading was "DogOnKeyboard's config carries a `per_layer_config` key +zerofata's lacks, so that key is the defect." Two measurements said otherwise. + +`tok_repro.py` reproduces the **actual failing call** rather than a paraphrase of it +— a bare `AutoConfig.from_pretrained` does not reproduce it, and testing that +instead would have sent us patching a file that was never the problem — with +zerofata's canonical tree, which quantized cleanly on 2026-08-21, as the positive +control. Against the container's shipped transformers **5.16.1**, every variant +passes, the unmodified heretic config included. Run the same script *after* +`pip install llmcompressor==0.13.0` and transformers is **5.14.1**: canonical +passes, heretic fails. **llmcompressor pins transformers and silently downgrades +it**, so the version printed before the install is not the version that runs. + +Compounding it, `vllm/vllm-openai:latest` was re-pulled during attempt 4 and not +before it — the pull line appears in that run's log block and in no earlier one — so +the toolchain moved mid-campaign (the previous session recorded 5.12.1 in-container +while diagnosing attempt 1). That is why attempt 4's error read as a *new config +problem* and was not one. **`run_v2_quant.sh` now pins the image by digest.** + +`per_layer_config` was in fact a 5.16.1 serialization artifact, and an exactly +redundant one: its keys are precisely the ten `full_attention` layer indices +[5,11,…,59] and its sole distinct value is `(head_dim 512, num_key_value_heads 4)` — +verbatim what `global_head_dim: 512` / `num_global_key_value_heads: 4` already say. +`patch_perlayer.py` re-proves that redundancy at patch time and refuses to drop the +key if it ever stops holding. + +**Why drop it rather than set `allow_global_per_layer_attribute_access=True`.** The +forcing flag leaves the config heterogeneous and makes `config.head_dim` answer 256 +to every caller — including the ones building the 512-wide full-attention layers. +`perlayer_test.py` builds all four variants on the meta device and reads the k_proj +widths back: geometry survived the flag, so it was not wrong, but llmcompressor's +own onloading is a caller nobody here has audited and the flag's warning names +exactly that hazard. The lossless option verified identically, so there was no +reason to take the risk. + +## Verification — the tensor table, not the exit code + +`rc=0` and a plausible file size prove neither of the operator's two checkable +requirements. `verify_quant.py` parses the safetensors headers directly (no torch, +no GPU, no 20 GB load) and reports dtypes by module family. The new quant is +**identical, family for family and count for count, to the 2026-08-21 known-good +canonical quant** — 410 U8 packed + 410 F8_E4M3 scales + 410 F32 global scales on the +LM Linears, **356 BF16 vision-tower tensors preserved**, `input_activations=None` +(genuinely A16, not A4). Shard sizes match that tree byte for byte. Full transcript +in `raw/verification.txt`. + +MTP is N/A and was checked on the sources, not assumed: Gemma-4 ships no MTP head at +all — 0 mtp tensors and no mtp config keys in either bf16 source or in any published +quant. The "mtp if applicable" clause is a no-op for this family. + +`cpu_smoke.py` then loads the finished tree with no GPU at all, confirms **0 tensors +left on the meta device** (a hole `from_pretrained` will not always raise on), +decompresses, and generates: + +> *"A lighthouse is a tower with a bright light used to guide ships at sea and warn +> them of dangerous coastlines."* + +24 greedy tokens, 3.4 s/tok on CPU. That is an "is it wired up" check and is offered +as nothing more — n=1 says nothing about quality, and it says nothing about whether +vLLM's sm_120 NVFP4 kernels serve the thing. + +## Owed + +- **§4.4 serve test on a temp port.** Not run. GPU1 has 19.9 GB free against 19.5 GB + of weights, so it cannot happen without displacing a live seat, which is the + operator's call. Until it runs, *"vllm servable"* is unverified for this tree. +- The A4B output **had the §3.14 truncation cap baked in** (`max_length: 8192`) — it + was quantized *with* the calibration corpus, before the data-free path existed. + Fixed 2026-09-10 by `post_quant_gemma4.py`; backup at + `tokenizer.json.bak-pre-truncfix`. Both trees now pass `--check` clean, as does the + August tree the checker is calibrated against. + +## Files + +| file | what it does | +|---|---| +| `run_v2_quant.sh` | attempt-5 runner, image pinned by digest | +| `run_quant_batch.sh` | the original two-model batch (A4B succeeded here, v2 did not) | +| `quant_a16_datafree.py` | NVFP4A16 driver with no dataset, wrapping the reference recipe | +| `patch_perlayer.py` | drops `per_layer_config`, re-proving its redundancy first | +| `shape_verify.py` | do the checkpoint's k/v shapes agree with the patched config? | +| `perlayer_test.py` | meta-device geometry across all four config variants | +| `tok_repro.py` | reproduces the real failing call; canonical tree as positive control | +| `post_quant_gemma4.py` | playbook §4.3 post-steps, idempotent, `--check` mode | +| `verify_quant.py` | dtypes by module family, straight from safetensors headers | +| `cpu_smoke.py` | GPU-free load + generate | +| `raw/` | run logs and the verification transcript, so the claims can be re-derived | + +`raw/` holds `batch-quant-run-2026-09-10.txt` (attempt 1, and the A4B success), +`v2-quant-run-2026-09-10.txt` (attempts 2-5), `cpu-smoke-2026-09-10.txt`, and +`verification.txt`. Progress-bar redraws are collapsed to one line per bar, final +state; nothing else is edited. (`.txt` rather than `.log` because the repo ignores +`*.log` — same convention as `scripts/training-probes/`.) + +General lessons live in `docs/pfi/model-quantization-playbook.md` (§3.4, §3.14, +§3.16, §3.17, §4.3) — read that first, and where it disagrees with this file, it wins. diff --git a/services/meromero-quant/cpu_smoke.py b/services/meromero-quant/cpu_smoke.py new file mode 100644 index 0000000..f1d7e4f --- /dev/null +++ b/services/meromero-quant/cpu_smoke.py @@ -0,0 +1,57 @@ +"""GPU-free load-and-generate smoke test for the quantized tree. + +§4.4 says test on a temp port, never on the live seat -- but GPU1 has 19.9 GB free +against 19.5 GB of weights, so a vLLM serve test cannot happen without displacing a +live seat, and that is not my call. This is what CAN be established without one: +that the checkpoint's tensor names map cleanly onto the architecture (no missing or +unexpected keys), that compressed-tensors can decompress it, and that it emits +plausible tokens rather than garbage. + +What it does NOT establish: that vLLM's sm_120 NVFP4 kernels serve it, or anything +about long-context quality. Those still need the GPU. Saying so is part of the +result -- a smoke test whose limits go unstated gets read as more than it is. + +Deliberately greedy and short. This is an "is it wired up" check, not an eval; n=1 +proves nothing about quality and is not offered as if it did. +""" +import sys +import time + +import torch +from transformers import AutoTokenizer + +path = sys.argv[1] +print(f"tree: {path}", flush=True) + +t0 = time.time() +tok = AutoTokenizer.from_pretrained(path) +print(f"tokenizer OK ({time.time()-t0:.1f}s) truncation_side={tok.truncation_side}", flush=True) + +t0 = time.time() +from transformers import AutoModelForImageTextToText as M +model = M.from_pretrained(path, dtype=torch.bfloat16, device_map=None) +print(f"model loaded on CPU ({time.time()-t0:.1f}s) {type(model).__name__}", flush=True) + +n = sum(p.numel() for p in model.parameters()) +print(f"parameters: {n/1e9:.2f} B", flush=True) + +# Any tensor still sitting on meta means a weight the checkpoint never supplied -- +# from_pretrained does not always raise on that, it just leaves the hole. +meta = [k for k, v in model.state_dict().items() if v.is_meta] +print(f"tensors still on meta device: {len(meta)}" + + (f" *** {meta[:5]}" if meta else " (none -- every weight was materialised)"), flush=True) + +msgs = [{"role": "user", "content": "In one sentence, what is a lighthouse for?"}] +enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt") +# transformers 5.x hands back a BatchEncoding here, not a bare tensor. +ids = enc["input_ids"] if hasattr(enc, "keys") else enc +print(f"prompt tokens: {ids.shape[-1]}", flush=True) + +t0 = time.time() +with torch.inference_mode(): + out = model.generate(ids, max_new_tokens=24, do_sample=False) +dt = time.time() - t0 +text = tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True) +print(f"generated {out.shape[-1]-ids.shape[-1]} tokens in {dt:.1f}s " + f"({dt/max(1,out.shape[-1]-ids.shape[-1]):.1f}s/tok, CPU)", flush=True) +print(f"OUTPUT: {text!r}", flush=True) diff --git a/services/meromero-quant/patch_perlayer.py b/services/meromero-quant/patch_perlayer.py new file mode 100644 index 0000000..fa1bdb7 --- /dev/null +++ b/services/meromero-quant/patch_perlayer.py @@ -0,0 +1,74 @@ +"""Remove the redundant `per_layer_config` block that blocks attempt 4. + +Why this and not `allow_global_per_layer_attribute_access=True`: + + * `per_layer_config` here carries NO information. Its keys are exactly the ten + full_attention layer indices [5,11,...,59] and its only distinct value is + (head_dim 512, num_key_value_heads 4) -- which `global_head_dim: 512` and + `num_global_key_value_heads: 4`, already in this config, say verbatim. + Removing it is lossless, and it is the difference in what the two toolchain + versions can read: transformers 5.16.1 (which authored this file) emits the + per-layer form; llmcompressor 0.13.0 PINS transformers to 5.14.1, which has + the heterogeneity guard but not the gemma4 resolver, so it refuses the read. + * Forcing global access leaves the config heterogeneous and makes + `config.head_dim` answer 256 to every caller -- including the ones building + the 512-wide full-attention layers. Geometry survived that in my meta-device + check, but llmcompressor's own onloading code is a caller I have not audited, + and it is precisely what transformers' warning is about. No reason to take + that when the lossless option verifies identically. + +Verified on transformers 5.14.1, the version the quant actually runs: this config +then reports head_dim/num_key_value_heads/global_head_dim/num_global_key_value_heads +identical to zerofata's canonical tree -- the config that quantized successfully on +2026-08-21 -- and builds k_proj (2048, 5376) on layer 5 and (4096, 5376) on layer 0, +matching the checkpoint. +""" +import json +import shutil +from pathlib import Path + +CFG = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16/config.json") +BAK = CFG.with_name("config.json.bak-pre-perlayer-20260910") + +NOTE = ( + " | infra-ops 2026-09-10 (2nd patch): removed text_config.per_layer_config, " + "a transformers-5.16.1 serialization artifact that llmcompressor 0.13.0's " + "pinned transformers 5.14.1 cannot read (AmbiguousGlobalPerLayerAttributeError " + "on head_dim). It was exactly redundant with global_head_dim=512 / " + "num_global_key_value_heads=4 -- keys were the 10 full_attention layers, sole " + "value (512, 4). Config now matches zerofata's canonical shape. Original at " + "config.json.bak-pre-perlayer-20260910." +) + +cfg = json.loads(CFG.read_text()) +t = cfg["text_config"] +plc = t.get("per_layer_config") + +if plc is None: + print("per_layer_config already absent -- nothing to do") + raise SystemExit(0) + +# Re-prove the redundancy here rather than trusting the earlier session: a patch +# that silences an error on a config it did not actually verify is how a quietly +# wrong quant ships. +full = {i for i, x in enumerate(t["layer_types"]) if x == "full_attention"} +assert {int(k) for k in plc} == full, f"per_layer_config keys {sorted(plc)} != full-attn layers {sorted(full)}" +vals = {(v["head_dim"], v["num_key_value_heads"]) for v in plc.values()} +assert vals == {(t["global_head_dim"], t["num_global_key_value_heads"])}, \ + f"per_layer_config carries {vals}, not the global (512, 4) -- NOT redundant, do not drop" +print(f"redundancy re-verified: {len(plc)} entries, all {vals.pop()}, " + f"== (global_head_dim, num_global_key_value_heads)") + +if not BAK.exists(): + shutil.copy2(CFG, BAK) + print(f"backed up -> {BAK.name}") +else: + print(f"backup {BAK.name} already exists, left alone") + +t.pop("per_layer_config") +cfg["_patched_by"] = cfg.get("_patched_by", "") + NOTE + +tmp = CFG.with_suffix(".json.tmp") +tmp.write_text(json.dumps(cfg, indent=2) + "\n") +tmp.replace(CFG) +print(f"patched {CFG}") diff --git a/services/meromero-quant/perlayer_test.py b/services/meromero-quant/perlayer_test.py new file mode 100644 index 0000000..189c1b3 --- /dev/null +++ b/services/meromero-quant/perlayer_test.py @@ -0,0 +1,100 @@ +"""Attempt-4 blocker: is dropping `per_layer_config` the correct fix, or must we +force `allow_global_per_layer_attribute_access`? + +The two candidates are NOT equivalent: + * DROP -> config becomes homogeneous in transformers' eyes and the global + `global_head_dim` / `num_global_key_value_heads` fields describe the + full-attention layers, exactly as zerofata's canonical config does. + * FORCE -> config stays heterogeneous; `config.head_dim` starts answering 256 to + every caller, including the ones building the 512-wide full-attention + layers. That is the hazard transformers' own warning names. + +So this is not a "did the traceback go away" test. It builds the model on the meta +device from each candidate and reads the ACTUAL k_proj widths back, against the +checkpoint's measured 2048 (full) / 4096 (sliding). A candidate that constructs but +mis-shapes a layer is a worse outcome than the crash, because it would ship. + +Positive control: zerofata's canonical config, which we already quantized +successfully on 2026-08-21, MUST pass every check here. If it doesn't, the +instrument is broken and none of the negatives mean anything. +""" +import json, shutil, tempfile, traceback +from pathlib import Path + +import torch +import transformers +from transformers import AutoConfig + +print(f"transformers {transformers.__version__} torch {torch.__version__}", flush=True) + +HERETIC = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16") +CANON = Path("/tank/aimodels/meromero-v2-nvfp4-work/src") + +# Measured off the checkpoints by shape_verify.py; both trees agree. +EXPECT = {"full_attention": 2048, "sliding_attention": 4096} + + +def variant(name, cfg_dict): + d = Path(tempfile.mkdtemp(prefix=f"cfg-{name}-")) + (d / "config.json").write_text(json.dumps(cfg_dict)) + return name, d + + +heretic = json.loads((HERETIC / "config.json").read_text()) +canon = json.loads((CANON / "config.json").read_text()) + +dropped = json.loads(json.dumps(heretic)) +dropped["text_config"].pop("per_layer_config") + +forced = json.loads(json.dumps(heretic)) +forced["text_config"]["allow_global_per_layer_attribute_access"] = True + +variants = [ + variant("A-canonical-POSITIVE-CONTROL", canon), + variant("B-heretic-asis", heretic), + variant("C-heretic-drop-per_layer_config", dropped), + variant("D-heretic-force-global-access", forced), +] + +for name, d in variants: + print(f"\n=== {name} ===", flush=True) + try: + cfg = AutoConfig.from_pretrained(d) + except Exception as e: + print(f" CONFIG FAILED: {type(e).__name__}: {str(e)[:160]}") + continue + t = cfg.text_config + fields = {} + for k in ("head_dim", "num_key_value_heads", "global_head_dim", + "num_global_key_value_heads"): + try: + fields[k] = getattr(t, k, "") + except Exception as e: + fields[k] = f"<{type(e).__name__}>" + print(f" config OK: {fields}") + + try: + from transformers import Gemma4ForConditionalGeneration as M + with torch.device("meta"): + model = M(cfg) + except Exception: + print(" MODEL BUILD FAILED:") + print(" " + traceback.format_exc().strip().replace("\n", "\n ")[-1200:]) + continue + + layer_types = t.layer_types + probes = [next(i for i, x in enumerate(layer_types) if x == "full_attention"), + next(i for i, x in enumerate(layer_types) if x == "sliding_attention")] + layers = model.model.language_model.layers + verdict = [] + for li in probes: + got = tuple(layers[li].self_attn.k_proj.weight.shape) + want = EXPECT[layer_types[li]] + ok = got[0] == want + verdict.append(ok) + print(f" L{li:>2} {layer_types[li]:<18} k_proj {got} " + f"want out={want} {'OK' if ok else '*** MISMATCH ***'}") + print(f" => {'GEOMETRY MATCHES CHECKPOINT' if all(verdict) else 'GEOMETRY WRONG'}") + +for _, d in variants: + shutil.rmtree(d, ignore_errors=True) diff --git a/services/meromero-quant/post_quant_gemma4.py b/services/meromero-quant/post_quant_gemma4.py new file mode 100644 index 0000000..6192bf3 --- /dev/null +++ b/services/meromero-quant/post_quant_gemma4.py @@ -0,0 +1,109 @@ +"""Playbook §4.3 post-steps for a Gemma-4 NVFP4 output tree. + +Steps 1 and 3 (MTP graft, `re:^mtp.*` re-injection) are N/A on Gemma-4 -- it ships +no MTP head at all, verified as 0 mtp tensors in both bf16 sources. That leaves: + + step 2 restore processor_config.json + preprocessor_config.json + step 4 confirm the saved tokenizer.json has truncation: null + +Step 4 is not a formality here. The A4B output was quantized WITH the calibration +dataset, and build_calib calls the fast tokenizer with truncation=True, +max_length=8192 -- which mutates the Rust backend in place, and save_pretrained +then bakes the cap into the shipped tokenizer.json. It is latent on the +transformers that wrote it and fatal on a newer one. The fix edits the one +`truncation` key rather than copying the source file wholesale, so nothing else in +a 32 MB tokenizer can quietly change underneath it. + +Derivation of preprocessor_config.json is `processor_config.json["image_processor"]` +verbatim; that reproduces the 2026-08-21 known-good output byte for byte. + +Idempotent, and reports per step whether it CHANGED or was already correct. +Run with --check to verify without writing. +""" +import argparse +import json +import shutil +from pathlib import Path + +ap = argparse.ArgumentParser() +ap.add_argument("--src", required=True, help="bf16 source tree") +ap.add_argument("--out", required=True, help="quantized output tree") +ap.add_argument("--check", action="store_true", help="report only, write nothing") +a = ap.parse_args() +src, out = Path(a.src), Path(a.out) +mode = "CHECK" if a.check else "APPLY" +print(f"[{mode}] src={src}\n[{mode}] out={out}\n") + +rc = 0 + + +def step(n, desc): + print(f"-- step {n}: {desc}") + + +step(1, "MTP graft") +mtp = [k for k in json.loads((out / "config.json").read_text()).get( + "quantization_config", {}).get("ignore", []) if "mtp" in k.lower()] +idx = out / "model.safetensors.index.json" +tensors = json.loads(idx.read_text())["weight_map"] if idx.exists() else {} +n_mtp = sum(1 for k in tensors if k.startswith("mtp")) +print(f" N/A for Gemma-4 (no MTP head). mtp tensors in output index: {n_mtp}; " + f"mtp entries in ignore list: {len(mtp)}") +if n_mtp: + print(" *** unexpected mtp tensors -- step 3 would become live, investigate") + rc = 1 + +step(2, "restore processor_config.json + preprocessor_config.json") +spc = src / "processor_config.json" +if not spc.exists(): + print(f" *** source has no processor_config.json -- cannot restore") + rc = 1 +else: + opc = out / "processor_config.json" + if opc.exists() and opc.read_bytes() == spc.read_bytes(): + print(" processor_config.json already present and identical to source") + elif a.check: + print(f" processor_config.json MISSING/differs -> would copy from source") + else: + shutil.copy2(spc, opc) + print(" processor_config.json CHANGED (copied from source)") + + want = json.dumps(dict(json.loads(spc.read_text())["image_processor"]), indent=1) + opre = out / "preprocessor_config.json" + if opre.exists() and opre.read_text() == want: + print(" preprocessor_config.json already present and correct") + elif a.check: + print(" preprocessor_config.json MISSING/differs -> would derive from image_processor") + else: + opre.write_text(want) + print(" preprocessor_config.json CHANGED (derived from processor_config" + "['image_processor'])") + +step(4, "confirm saved tokenizer.json has truncation: null") +tj = out / "tokenizer.json" +tok = json.loads(tj.read_text()) +trunc = tok.get("truncation") +if trunc is None: + print(" truncation is null -- clean") +else: + print(f" *** truncation BAKED IN: {trunc}") + stok = json.loads((src / "tokenizer.json").read_text()) + others = [k for k in set(tok) | set(stok) + if k != "truncation" and tok.get(k) != stok.get(k)] + print(f" other top-level keys differing from source: {others or 'none'}") + if a.check: + print(" would set truncation -> null") + rc = 1 + else: + bak = tj.with_name("tokenizer.json.bak-pre-truncfix") + if not bak.exists(): + shutil.copy2(tj, bak) + print(f" backed up -> {bak.name}") + tok["truncation"] = None + tmp = tj.with_suffix(".json.tmp") + tmp.write_text(json.dumps(tok, ensure_ascii=False, indent=2)) + tmp.replace(tj) + print(" truncation CHANGED -> null") + +print(f"\n[{mode}] done rc={rc}") +raise SystemExit(rc) diff --git a/services/meromero-quant/quant_a16_datafree.py b/services/meromero-quant/quant_a16_datafree.py new file mode 100644 index 0000000..c4707d0 --- /dev/null +++ b/services/meromero-quant/quant_a16_datafree.py @@ -0,0 +1,48 @@ +"""NVFP4A16 without a calibration dataset. + +Playbook §3.16, measured 2026-09-08 on this architecture: with scheme NVFP4A16 +llm-compressor logs `Inferred DataFreePipeline` and NEVER touches the dataset. +Passing one is therefore pure liability, and it cost two failures here: + + * llmcompressor demands a model PROCESSOR whenever a dataset is provided, which + is what killed the v2 pass (`DogOnKeyboard` ships no processor_config.json). + * building the calib set calls the fast tokenizer with truncation=True, which + mutates the Rust backend in place and `save_pretrained` then BAKES that cap + into the shipped tokenizer.json -- playbook §3.14, fatal on a newer + transformers for a vision model. + +Dropping the dataset removes both for zero loss, because the quant is data-free. +Everything else -- targets, ignore list, save path -- matches the reference script. +""" +import argparse, json, sys, importlib.util +spec = importlib.util.spec_from_file_location( + "ref", "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py") +ref = importlib.util.module_from_spec(spec) +# The reference module runs argparse at IMPORT with required=True args, so an +# empty argv still exits 2. Feed placeholders; our own parse happens after. +_real_argv = sys.argv +sys.argv = ["ref", "--model", "/dev/null", "--calib", "/dev/null", "--out", "/dev/null"] +spec.loader.exec_module(ref) + +sys.argv = _real_argv +ap = argparse.ArgumentParser() +ap.add_argument("--model", required=True) +ap.add_argument("--out", required=True) +ap.add_argument("--scheme", default="NVFP4A16") +a = ap.parse_args() + +assert a.scheme.endswith("A16"), f"{a.scheme} is not weight-only; it needs calibration data" +print(f"loading {a.model}", flush=True) +model, tok = ref.load_model(a.model) + +from llmcompressor import oneshot +from llmcompressor.modifiers.quantization import QuantizationModifier +recipe = QuantizationModifier(targets="Linear", scheme=a.scheme, ignore=ref.IGNORE) +print(f"NVFP4 oneshot (DATA-FREE): scheme={a.scheme}, Linear-only, " + f"vision/audio/projector/embed/lm_head/norms kept BF16", flush=True) +oneshot(model=model, recipe=recipe) + +print(f"saving -> {a.out}", flush=True) +model.save_pretrained(a.out, save_compressed=True) +tok.save_pretrained(a.out) +print("DONE", flush=True) diff --git a/services/meromero-quant/raw/batch-quant-run-2026-09-10.txt b/services/meromero-quant/raw/batch-quant-run-2026-09-10.txt new file mode 100644 index 0000000..4ff7398 --- /dev/null +++ b/services/meromero-quant/raw/batch-quant-run-2026-09-10.txt @@ -0,0 +1,98 @@ +=== 2026-09-10T08:13:44-07:00 START A4B-heretic -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 +[notice] To update, run: python3.12 -m pip install --upgrade pip +loading /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16 +Loading weights: 100%|██████████| 1013/1013 [00:00<00:00, 4035.87it/s] +building calibration (<= 512 @ seq 8192) + 512 calibration rows +NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16 +2026-09-10T15:14:09.8205 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning. +2026-09-10T15:14:12.3242 | reset | INFO - Compression lifecycle reset +2026-09-10T15:14:12.8109 | apply_recipe_modifiers | WARNING - Detected an MoE model which has not been linearized. First load model `with llmcompressor.modeling.moe.linearize.load_quantizable_moe` before passing to `oneshot`. Falling back to post-load linearization. +2026-09-10T15:14:13.2653 | linearize_moe | WARNING - MoE is being linearized after loading in order to support efficient calibration of experts. However, this may be inefficient if the model checkpoint is already linearized (2D -> 3D -> 2D). Consider registering a load converter for faster load times. See https://docs.vllm.ai/projects/llm-compressor/en/latest/developer-tutorials/add-moe-support +Linearizing experts: 100%|██████████| 30/30 [00:35<00:00, 1.17s/it] +2026-09-10T15:14:48.4792 | from_modifiers | INFO - Creating recipe from modifiers +Applying quantization config: 100%|██████████| 11755/11755 [00:01<00:00, 8851.89it/s] +2026-09-10T15:14:50.3917 | initialize | INFO - Compression lifecycle initialized for 1 modifiers +2026-09-10T15:14:50.3920 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier` +2026-09-10T15:15:14.4903 | finalize | INFO - Compression lifecycle finalized for 1 modifiers +saving -> /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 +Compressing model: 100%|██████████| 11755/11755 [00:11<00:00, 980.90it/s] +Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.96s/it] +Dispatching model: 100%|██████████| 16828/16828 [00:00<00:00, 43870.85it/s] +DONE. serve --quantization compressed-tensors (multimodal: vision+audio kept BF16; NO --language-model-only). No spec-decode; Gemma-4 has no MTP. +=== 2026-09-10T08:15:52-07:00 END A4B-heretic rc=0 size=16G +=== 2026-09-10T08:15:52-07:00 START v2-31B-heretic -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 +[notice] To update, run: python3.12 -m pip install --upgrade pip +loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 +Traceback (most recent call last): + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 76, in load_model + model = M.from_pretrained( + ^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained + model = cls(config, *model_args, **model_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__ + self.model = Gemma4Model(config) + ^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__ + language_model = AutoModel.from_config(config=config.text_config) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config + return model_class._from_config(config, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config + model = cls(config, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__ + [Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)] + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__ + self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__ + self.num_key_value_groups = config.num_attention_heads // num_key_value_heads + ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~ +TypeError: unsupported operand type(s) for //: 'int' and 'NoneType' + +During handling of the above exception, another exception occurred: + +Traceback (most recent call last): + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in + sys.exit(main()) + ^^^^^^ + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 99, in main + model, tok = load_model(a.model) + ^^^^^^^^^^^^^^^^^^^ + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 82, in load_model + model = M.from_pretrained( + ^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 406, in from_pretrained + return model_class.from_pretrained( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 4283, in from_pretrained + model = cls(config, *model_args, **model_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2452, in __init__ + self.model = Gemma4Model(config) + ^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2132, in __init__ + language_model = AutoModel.from_config(config=config.text_config) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 250, in from_config + return model_class._from_config(config, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 1620, in _from_config + model = cls(config, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1605, in __init__ + [Gemma4TextDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)] + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1375, in __init__ + self.self_attn = Gemma4TextAttention(config=config, layer_idx=layer_idx) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 1193, in __init__ + self.num_key_value_groups = config.num_attention_heads // num_key_value_heads + ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~~~~ +TypeError: unsupported operand type(s) for //: 'int' and 'NoneType' +=== 2026-09-10T08:16:18-07:00 END v2-31B-heretic rc=1 size= +=== 2026-09-10T08:16:18-07:00 BATCH DONE diff --git a/services/meromero-quant/raw/cpu-smoke-2026-09-10.txt b/services/meromero-quant/raw/cpu-smoke-2026-09-10.txt new file mode 100644 index 0000000..49ec3f2 --- /dev/null +++ b/services/meromero-quant/raw/cpu-smoke-2026-09-10.txt @@ -0,0 +1,13 @@ +WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. +tree: /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 +tokenizer OK (1.6s) truncation_side=right +Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 19976.82it/s] +Compressing model: 100%|██████████| 410/410 [00:01<00:00, 293.98it/s] +Loading weights: 100%|██████████| 2008/2008 [00:00<00:00, 4966.04it/s] +model loaded on CPU (2.7s) Gemma4ForConditionalGeneration +parameters: 18.46 B +tensors still on meta device: 0 (none -- every weight was materialised) +prompt tokens: 23 +Decompressing model: 100%|██████████| 410/410 [01:10<00:00, 5.85it/s] +generated 24 tokens in 82.1s (3.4s/tok, CPU) +OUTPUT: 'A lighthouse is a tower with a bright light used to guide ships at sea and warn them of dangerous coastlines.' diff --git a/services/meromero-quant/raw/v2-quant-run-2026-09-10.txt b/services/meromero-quant/raw/v2-quant-run-2026-09-10.txt new file mode 100644 index 0000000..a63fbba --- /dev/null +++ b/services/meromero-quant/raw/v2-quant-run-2026-09-10.txt @@ -0,0 +1,265 @@ +=== 2026-09-10T08:59:18-07:00 START v2-31B-heretic (post config patch) +[notice] To update, run: python3.12 -m pip install --upgrade pip +loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 +Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 2518.29it/s] +building calibration (<= 512 @ seq 8192) + 512 calibration rows +NVFP4 oneshot: scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16 +2026-09-10T15:59:48.3371 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning. +Traceback (most recent call last): + File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 68, in pre_process + model_args.processor = initialize_processor_from_path( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 184, in initialize_processor_from_path + processor = AutoProcessor.from_pretrained( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/processing_auto.py", line 327, in from_pretrained + return processor_class.from_pretrained( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1715, in from_pretrained + args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1844, in _get_arguments_from_pretrained + sub_processor = auto_processor_class.from_pretrained( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/feature_extraction_auto.py", line 300, in from_pretrained + config_dict, _ = FeatureExtractionMixin.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/feature_extraction_utils.py", line 529, in get_feature_extractor_dict + raise OSError( +OSError: Can't load feature extractor for '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16' is the correct path to a directory containing a preprocessor_config.json file + +The above exception was the direct cause of the following exception: + +Traceback (most recent call last): + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 128, in + sys.exit(main()) + ^^^^^^ + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 111, in main + oneshot( + File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 468, in oneshot + one_shot = Oneshot(**local_args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/oneshot.py", line 178, in __init__ + pre_process(model_args, dataset_args, output_dir) + File "/usr/local/lib/python3.12/dist-packages/llmcompressor/entrypoints/utils.py", line 73, in pre_process + raise RuntimeError( +RuntimeError: An error occurred when attempting to initialize model processor, which is required when a dataset is provided. To resolve, create and pass in a processor directly to `oneshot`/`train`. +=== 2026-09-10T08:59:54-07:00 END rc=0 size=512 +=== 2026-09-10T09:21:40-07:00 START v2-31B-heretic (config patched + processor + data-free) +[notice] To update, run: python3.12 -m pip install --upgrade pip +usage: ref [-h] --model MODEL --out OUT [--scheme SCHEME] +ref: error: the following arguments are required: --model, --out +=== 2026-09-10T09:21:56-07:00 END rc=2 size=512 +=== 2026-09-10T09:23:50-07:00 START v2-31B-heretic (config patched + processor + data-free) +Unable to find image 'vllm/vllm-openai:latest' locally +latest: Pulling from vllm/vllm-openai +cf57d2112d89: Already exists +c567a87f21d2: Already exists +0b3b5bd92824: Already exists +70f87b6ed43e: Already exists +526d5438c009: Already exists +79ed78d42ca9: Pulling fs layer +5785dfb2d94d: Pulling fs layer +8136a02ba8b6: Pulling fs layer +1dae32d336bd: Pulling fs layer +4f4fb700ef54: Pulling fs layer +0e1d24786a23: Pulling fs layer +c83c61a504db: Pulling fs layer +81801e5f6a47: Pulling fs layer +e832d0ac2449: Pulling fs layer +f41db59aec9f: Pulling fs layer +4a1facbdf857: Pulling fs layer +b1cc0c6d03ea: Pulling fs layer +334119c098d4: Pulling fs layer +b14dc82c93d9: Pulling fs layer +f599001d1dac: Pulling fs layer +66a08f34da9c: Pulling fs layer +2411167b6874: Pulling fs layer +a9b8ef092e47: Pulling fs layer +0f36e99efdcd: Pulling fs layer +32d0568ab58d: Pulling fs layer +c83c61a504db: Waiting +81801e5f6a47: Waiting +4693bfabf3cd: Pulling fs layer +e832d0ac2449: Waiting +7994811847da: Pulling fs layer +d80f1ecbeb8c: Pulling fs layer +1dae32d336bd: Waiting +8294aa869476: Pulling fs layer +f41db59aec9f: Waiting +d510763bc7fa: Pulling fs layer +4f4fb700ef54: Waiting +2b5be6c4f7e6: Pulling fs layer +0e1d24786a23: Waiting +c8097f43c701: Pulling fs layer +4a1facbdf857: Waiting +3c84f5c1ca0b: Pulling fs layer +b1cc0c6d03ea: Waiting +b989116cb4ee: Pulling fs layer +f599001d1dac: Waiting +334119c098d4: Waiting +cfe38ec4fb3a: Pulling fs layer +2411167b6874: Waiting +b14dc82c93d9: Waiting +a9b8ef092e47: Waiting +66a08f34da9c: Waiting +8294aa869476: Waiting +d510763bc7fa: Waiting +3c84f5c1ca0b: Waiting +2b5be6c4f7e6: Waiting +4693bfabf3cd: Waiting +0f36e99efdcd: Waiting +c8097f43c701: Waiting +b989116cb4ee: Waiting +32d0568ab58d: Waiting +cfe38ec4fb3a: Waiting +7994811847da: Waiting +d80f1ecbeb8c: Waiting +79ed78d42ca9: Verifying Checksum +79ed78d42ca9: Download complete +5785dfb2d94d: Verifying Checksum +5785dfb2d94d: Download complete +79ed78d42ca9: Pull complete +5785dfb2d94d: Pull complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +0e1d24786a23: Verifying Checksum +0e1d24786a23: Download complete +c83c61a504db: Verifying Checksum +c83c61a504db: Download complete +81801e5f6a47: Verifying Checksum +81801e5f6a47: Download complete +e832d0ac2449: Verifying Checksum +e832d0ac2449: Download complete +f41db59aec9f: Download complete +4a1facbdf857: Verifying Checksum +4a1facbdf857: Download complete +b1cc0c6d03ea: Verifying Checksum +b1cc0c6d03ea: Download complete +8136a02ba8b6: Download complete +8136a02ba8b6: Pull complete +b14dc82c93d9: Verifying Checksum +b14dc82c93d9: Download complete +f599001d1dac: Download complete +66a08f34da9c: Verifying Checksum +66a08f34da9c: Download complete +2411167b6874: Verifying Checksum +2411167b6874: Download complete +a9b8ef092e47: Verifying Checksum +a9b8ef092e47: Download complete +0f36e99efdcd: Verifying Checksum +0f36e99efdcd: Download complete +32d0568ab58d: Verifying Checksum +32d0568ab58d: Download complete +4693bfabf3cd: Verifying Checksum +4693bfabf3cd: Download complete +7994811847da: Verifying Checksum +7994811847da: Download complete +d80f1ecbeb8c: Verifying Checksum +d80f1ecbeb8c: Download complete +8294aa869476: Verifying Checksum +8294aa869476: Download complete +1dae32d336bd: Verifying Checksum +1dae32d336bd: Download complete +d510763bc7fa: Verifying Checksum +d510763bc7fa: Download complete +c8097f43c701: Verifying Checksum +c8097f43c701: Download complete +3c84f5c1ca0b: Verifying Checksum +3c84f5c1ca0b: Download complete +b989116cb4ee: Verifying Checksum +b989116cb4ee: Download complete +cfe38ec4fb3a: Download complete +2b5be6c4f7e6: Verifying Checksum +2b5be6c4f7e6: Download complete +1dae32d336bd: Pull complete +4f4fb700ef54: Pull complete +0e1d24786a23: Pull complete +c83c61a504db: Pull complete +81801e5f6a47: Pull complete +e832d0ac2449: Pull complete +f41db59aec9f: Pull complete +4a1facbdf857: Pull complete +b1cc0c6d03ea: Pull complete +334119c098d4: Verifying Checksum +334119c098d4: Download complete +334119c098d4: Pull complete +b14dc82c93d9: Pull complete +f599001d1dac: Pull complete +66a08f34da9c: Pull complete +2411167b6874: Pull complete +a9b8ef092e47: Pull complete +0f36e99efdcd: Pull complete +32d0568ab58d: Pull complete +4693bfabf3cd: Pull complete +7994811847da: Pull complete +d80f1ecbeb8c: Pull complete +8294aa869476: Pull complete +d510763bc7fa: Pull complete +2b5be6c4f7e6: Pull complete +c8097f43c701: Pull complete +3c84f5c1ca0b: Pull complete +b989116cb4ee: Pull complete +cfe38ec4fb3a: Pull complete +Digest: sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1 +Status: Downloaded newer image for vllm/vllm-openai:latest +WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. +loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 +Traceback (most recent call last): + File "/q/quant_a16_datafree.py", line 36, in + model, tok = ref.load_model(a.model) + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/tank/aimodels/meromero-v2-nvfp4-work/quant_nvfp4_gemma.py", line 73, in load_model + tok = AutoTokenizer.from_pretrained(path, trust_remote_code=True) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 747, in from_pretrained + config = AutoConfig.from_pretrained( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/configuration_auto.py", line 419, in from_pretrained + return config_class.from_dict(config_dict, **unused_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 878, in from_dict + config = cls(**config_dict) + ^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 275, in init_with_validate + initial_init(self, *args, **kwargs) # type: ignore [call-arg] + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 131, in __init__ + self.__post_init__(**additional_kwargs) + File "/usr/local/lib/python3.12/dist-packages/transformers/models/gemma4/configuration_gemma4.py", line 348, in __post_init__ + self.text_config = Gemma4TextConfig(**self.text_config) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 276, in init_with_validate + cls.validate(self) # type: ignore [attr-defined] + ^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/dataclasses.py", line 251, in validate + validator(self) + File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 476, in validate_architecture + hasattr(self, "head_dim") + File "/usr/local/lib/python3.12/dist-packages/transformers/configuration_utils.py", line 464, in __getattribute__ + return super().__getattribute__(key) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/heterogeneity/configuration_utils.py", line 266, in __getattribute__ + raise AmbiguousGlobalPerLayerAttributeError( +transformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].head_dim). To read the global config value from config.head_dim anyway, set `allow_global_per_layer_attribute_access` to `True` on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly. +=== 2026-09-10T09:28:02-07:00 END rc=1 size=512 +=== 2026-09-10T10:40:09-07:00 START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned) +WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. +transformers 5.14.1 +loading /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 +Loading weights: 100%|██████████| 1188/1188 [00:00<00:00, 5430.71it/s] +NVFP4 oneshot (DATA-FREE): scheme=NVFP4A16, Linear-only, vision/audio/projector/embed/lm_head/norms kept BF16 +2026-09-10T17:40:48.5554 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning. +2026-09-10T17:40:50.1494 | reset | INFO - Compression lifecycle reset +2026-09-10T17:40:50.1782 | from_modifiers | INFO - Creating recipe from modifiers +Applying quantization config: 100%|██████████| 410/410 [00:00<00:00, 2863.72it/s] +2026-09-10T17:40:50.3684 | initialize | INFO - Compression lifecycle initialized for 1 modifiers +2026-09-10T17:40:50.3685 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier` +2026-09-10T17:42:45.7622 | finalize | INFO - Compression lifecycle finalized for 1 modifiers +saving -> /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 +Compressing model: 100%|██████████| 410/410 [00:11<00:00, 36.91it/s] +Writing model shards: 100%|██████████| 2/2 [00:08<00:00, 4.01s/it] +Dispatching model: 100%|██████████| 1763/1763 [00:00<00:00, 58662.76it/s] +DONE +=== 2026-09-10T10:43:16-07:00 END rc=0 size=19G diff --git a/services/meromero-quant/raw/verification.txt b/services/meromero-quant/raw/verification.txt new file mode 100644 index 0000000..750cd36 --- /dev/null +++ b/services/meromero-quant/raw/verification.txt @@ -0,0 +1,83 @@ +### verify_quant.py — new v2 heretic quant vs the 2026-08-21 known-good canonical quant +$ sudo -n python3 verify_quant.py + +====================================================================== +/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 + group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY) + format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed + text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4 + tensor dtypes by family: + embeddings BF16x1 + language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors] + norms BF16x361 + vision_tower BF16x356 + +====================================================================== +/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16 + group_0: weights num_bits=4 type=float strategy=tensor_group | input_activations=None (WEIGHT-ONLY) + format=nvfp4-pack-quantized kv_cache_scheme=None status=compressed + text_config: per_layer_config=absent head_dim=256 global_head_dim=512 num_key_value_heads=16 num_global_key_value_heads=4 + tensor dtypes by family: + embeddings BF16x1 + language_model BF16x60, F32x410, F8_E4M3x410, U8x410 [820 packed/scale tensors] + norms BF16x361 + vision_tower BF16x356 + +### post_quant_gemma4.py --check — v2 heretic output +[CHECK] src=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 +[CHECK] out=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 + +-- step 1: MTP graft + N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0 +-- step 2: restore processor_config.json + preprocessor_config.json + processor_config.json already present and identical to source + preprocessor_config.json already present and correct +-- step 4: confirm saved tokenizer.json has truncation: null + truncation is null -- clean + +[CHECK] done rc=0 + +### post_quant_gemma4.py --check — A4B output (after the truncation fix) +[CHECK] src=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16 +[CHECK] out=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 + +-- step 1: MTP graft + N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0 +-- step 2: restore processor_config.json + preprocessor_config.json + processor_config.json already present and identical to source + preprocessor_config.json already present and correct +-- step 4: confirm saved tokenizer.json has truncation: null + truncation is null -- clean + +[CHECK] done rc=0 + +### post_quant_gemma4.py --check — 2026-08-21 known-good tree (POSITIVE CONTROL, must be clean) +[CHECK] src=/tank/aimodels/meromero-v2-nvfp4-work/src +[CHECK] out=/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16 + +-- step 1: MTP graft + N/A for Gemma-4 (no MTP head). mtp tensors in output index: 0; mtp entries in ignore list: 0 +-- step 2: restore processor_config.json + preprocessor_config.json + processor_config.json already present and identical to source + preprocessor_config.json already present and correct +-- step 4: confirm saved tokenizer.json has truncation: null + truncation is null -- clean + +[CHECK] done rc=0 + +### output tree +total 19762161 +drwxr-xr-x 2 infra-ops infra-ops 13 Sep 10 10:43 . +drwxrwxr-x 62 llmuser llm 99 Sep 10 08:59 .. +-rw-r--r-- 1 root root 16934 Sep 10 10:43 chat_template.jinja +-rw-r--r-- 1 root root 19419 Sep 10 10:43 config.json +-rw-r--r-- 1 root root 204 Sep 10 10:43 generation_config.json +-rw------- 1 root root 19994044576 Sep 10 10:43 model-00001-of-00002.safetensors +-rw------- 1 root root 452731960 Sep 10 10:43 model-00002-of-00002.safetensors +-rw-r--r-- 1 root root 209808 Sep 10 10:43 model.safetensors.index.json +-rw-r--r-- 1 root root 375 Sep 10 10:43 preprocessor_config.json +-rw-r--r-- 1 root root 1689 Aug 12 02:28 processor_config.json +-rw-r--r-- 1 root root 430 Sep 10 10:43 recipe.yaml +-rw-r--r-- 1 root root 2819 Sep 10 10:43 tokenizer_config.json +-rw-r--r-- 1 root root 32169780 Sep 10 10:43 tokenizer.json +19G /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 diff --git a/services/meromero-quant/run_quant_batch.sh b/services/meromero-quant/run_quant_batch.sh new file mode 100755 index 0000000..69907bb --- /dev/null +++ b/services/meromero-quant/run_quant_batch.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Two NVFP4A16 quants, 2026-09-10. Operator: "run our own quant. w4a16 vllm +# servable, vision towers intact, mtp if applicable." +# W4A16 -> --scheme NVFP4A16 (weight-only, NOT plain NVFP4/W4A4) +# vision intact -> recipe ignore-list keeps vision/audio towers BF16 +# MTP -> N/A: verified 0 mtp tensors in BOTH bf16 sources +# GPU1 not GPU0: the script onloads one layer at a time (GPU-light) but its own +# docstring warns of OOM when the card is not fairly free. GPU0 has 4.6 GiB spare +# (gen + mog-sec resident); GPU1 has ~19.3 GiB. +set -uo pipefail +WORK=/tank/aimodels/meromero-v2-nvfp4-work +CALIB=/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl +exec >> /home/infra-ops/quant/batch.log 2>&1 + +run_one () { + local name="$1" src="$2" out="$3" + echo "=== $(date -Is) START $name -> $out" + docker rm -f meromero-quant >/dev/null 2>&1 + docker run --rm --name meromero-quant --gpus '"device=1"' --ipc host \ + -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ + -v /tank/aimodels:/tank/aimodels \ + --entrypoint bash vllm/vllm-openai:latest -c " + set -e + pip install -q llmcompressor==0.13.0 tiktoken sentencepiece 2>&1 | tail -1 + python3 $WORK/quant_nvfp4_gemma.py \ + --model '$src' --calib '$CALIB' \ + --num-samples 512 --seqlen 8192 --scheme NVFP4A16 \ + --out '$out' + " + local rc=$? + echo "=== $(date -Is) END $name rc=$rc size=$(du -sh "$out" 2>/dev/null | cut -f1)" +} + +run_one A4B-heretic \ + /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-bf16 \ + /tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 +run_one v2-31B-heretic \ + /tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 \ + /tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 +echo "=== $(date -Is) BATCH DONE" diff --git a/services/meromero-quant/run_v2_quant.sh b/services/meromero-quant/run_v2_quant.sh new file mode 100755 index 0000000..131511b --- /dev/null +++ b/services/meromero-quant/run_v2_quant.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# NVFP4A16 quant of the DogOnKeyboard v2-31B heretic (abliterated) Gemma-4. +# +# Attempt 5. Four things had to be fixed to get here and all four are load-bearing: +# 1. config.json was missing num_global_key_value_heads / global_head_dim while +# declaring attention_k_eq_v -- patched from zerofata's canonical values after +# shape-verifying the checkpoint (full-attn k_proj [2048,5376] = 4x512). +# 2. config.json carried a per_layer_config block that transformers 5.14.1 refuses +# to read globally. Removed; it was exactly redundant with (1). See +# patch_perlayer.py for the redundancy proof. +# 3. No processor_config.json in the upload -> llmcompressor demands a processor +# whenever a dataset is passed. Dropping the dataset removes the demand, and +# costs nothing: NVFP4A16 is weight-only and runs a DataFreePipeline. +# 4. The reference module argparses at import; quant_a16_datafree.py feeds it +# placeholder argv and restores the real one. +# +# IMAGE PINNED BY DIGEST, deliberately. `vllm/vllm-openai:latest` was re-pulled +# between attempt 3 and attempt 4 and moved transformers 5.12.1 -> 5.16.1, which is +# why attempt 4's error looked like a new config problem and was not. Note the +# effective transformers is 5.14.1 either way -- llmcompressor 0.13.0 pins it, and +# it is 5.14.1 the config had to be made readable by. +IMAGE=vllm/vllm-openai@sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1 +SRC=/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16 +OUT=/tank/aimodels/G4-MeroMero-v2-31B-heretic-NVFP4A16 +set -uo pipefail +exec >> /home/infra-ops/quant/v2.log 2>&1 +echo "=== $(date -Is) START v2-31B-heretic attempt 5 (per_layer_config dropped, image pinned)" +docker rm -f meromero-quant >/dev/null 2>&1 +docker run --rm --name meromero-quant --gpus '"device=1"' --ipc host \ + -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ + -v /tank/aimodels:/tank/aimodels -v /home/infra-ops/quant:/q \ + --entrypoint bash "$IMAGE" -c " + set -e + pip install -q llmcompressor==0.13.0 tiktoken sentencepiece 2>&1 | tail -1 + python3 -c 'import transformers; print(\"transformers\", transformers.__version__, flush=True)' + python3 /q/quant_a16_datafree.py --model $SRC --out $OUT --scheme NVFP4A16 + " +rc=$? # captured BEFORE any other command -- an earlier wrapper read $? after an echo and always said 0 +echo "=== $(date -Is) END rc=$rc size=$(du -sh $OUT 2>/dev/null | cut -f1)" diff --git a/services/meromero-quant/shape_verify.py b/services/meromero-quant/shape_verify.py new file mode 100644 index 0000000..444722a --- /dev/null +++ b/services/meromero-quant/shape_verify.py @@ -0,0 +1,50 @@ +"""Do the CHECKPOINT's tensor shapes agree with num_global_key_value_heads=4? + +Patching a config to satisfy a constructor is only safe if the weights already have +the shape the patched value implies. If the abliteration reshaped attention, the +patch would silence the error and produce a quietly wrong quant -- worse than the +crash, because it ships. + +For a full-attention layer, k_proj/v_proj out-features == num_kv_heads * head_dim. +zerofata's canonical v2: num_global_key_value_heads=4, global_head_dim=512 + => expected out-features 4 * 512 = 2048 on the GLOBAL (full-attention) layers. +""" +import json, sys +from safetensors import safe_open +from pathlib import Path + +def probe(label, root, n_global_kv, global_head_dim, n_kv, head_dim): + root = Path(root) + idx = json.load(open(root / "model.safetensors.index.json"))["weight_map"] + types = json.load(open(root / "config.json"))["text_config"]["layer_types"] + full = [i for i, t in enumerate(types) if t == "full_attention"][:2] + slide = [i for i, t in enumerate(types) if t == "sliding_attention"][:2] + print(f" -- {label}") + print(f" expected FULL k/v out-features = {n_global_kv} x {global_head_dim} = {n_global_kv*global_head_dim}" + if n_global_kv and global_head_dim else " expected FULL = (config lacks the fields)") + print(f" expected SLIDE k/v out-features = {n_kv} x {head_dim} = {n_kv*head_dim}") + for tag, idxs in (("full ", full), ("slide", slide)): + for li in idxs: + for proj in ("k_proj", "v_proj"): + key = f"model.language_model.layers.{li}.self_attn.{proj}.weight" + if key not in idx: + key = f"language_model.model.layers.{li}.self_attn.{proj}.weight" + if key not in idx: + cand = [k for k in idx if f"layers.{li}.self_attn.{proj}" in k] + key = cand[0] if cand else None + if not key: + print(f" {tag} L{li} {proj}: KEY NOT FOUND"); continue + with safe_open(root / idx[key], framework="pt") as f: + shape = f.get_slice(key).get_shape() + print(f" {tag} L{li} {proj}: shape {shape} out-features={shape[0]}") + +cfg = json.load(open("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16/config.json"))["text_config"] +good = json.load(open("/tank/aimodels/meromero-v2-nvfp4-work/src/config.json"))["text_config"] +print(f" canonical (zerofata): num_global_key_value_heads={good.get('num_global_key_value_heads')} " + f"global_head_dim={good.get('global_head_dim')} num_key_value_heads={good.get('num_key_value_heads')} head_dim={good.get('head_dim')}") +probe("zerofata v2 (canonical)", "/tank/aimodels/meromero-v2-nvfp4-work/src", + good.get("num_global_key_value_heads"), good.get("global_head_dim"), + good.get("num_key_value_heads"), good.get("head_dim")) +probe("DogOnKeyboard v2 (to patch)", "/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16", + good.get("num_global_key_value_heads"), good.get("global_head_dim"), + cfg.get("num_key_value_heads"), cfg.get("head_dim")) diff --git a/services/meromero-quant/tok_repro.py b/services/meromero-quant/tok_repro.py new file mode 100644 index 0000000..7f51d72 --- /dev/null +++ b/services/meromero-quant/tok_repro.py @@ -0,0 +1,67 @@ +"""Reproduce the ACTUAL failing call, not a paraphrase of it. + +quant_a16_datafree.py dies inside `AutoTokenizer.from_pretrained(path, +trust_remote_code=True)`. A bare `AutoConfig.from_pretrained(dir)` does NOT +reproduce it -- I checked, and all four config variants sailed through. So the +trigger lives in the tokenizer path, and testing the config alone would have sent +me off patching a file that was never the problem. + +POSITIVE CONTROL, and it is the whole point of this script: zerofata's canonical +v2 tree quantized cleanly on 2026-08-21. If it now fails on this same call, the +config is exonerated and the toolchain moved under us -- `vllm/vllm-openai:latest` +was re-pulled mid-campaign and carries transformers 5.16.1 where the successful +August run had 5.12.1. +""" +import json, os, tempfile, traceback +from pathlib import Path + +import transformers +from transformers import AutoTokenizer + +print(f"transformers {transformers.__version__}", flush=True) + +HERETIC = Path("/tank/aimodels/G4-MeroMero-v2-31B-heretic-bf16") +CANON = Path("/tank/aimodels/meromero-v2-nvfp4-work/src") + +# Tokenizer loading reads config.json, so a variant needs the whole tree. Symlink +# everything, then overwrite the one file under test. +def tree(name, src, mutate=None): + d = Path(tempfile.mkdtemp(prefix=f"tok-{name}-")) + for f in src.iterdir(): + if f.is_file(): + os.symlink(f, d / f.name) + if mutate is not None: + cfg = json.loads((src / "config.json").read_text()) + mutate(cfg) + (d / "config.json").unlink() + (d / "config.json").write_text(json.dumps(cfg)) + return d + + +def drop_plc(c): + c["text_config"].pop("per_layer_config", None) + + +def force_global(c): + c["text_config"]["allow_global_per_layer_attribute_access"] = True + + +cases = [ + ("A-canonical-POSITIVE-CONTROL", tree("canon", CANON)), + ("B-heretic-asis", tree("heretic", HERETIC)), + ("C-heretic-drop-per_layer_config", tree("drop", HERETIC, drop_plc)), + ("D-heretic-force-global-access", tree("force", HERETIC, force_global)), + ("E-canonical-force-global-access", tree("canonforce", CANON, force_global)), +] + +for name, d in cases: + print(f"\n=== {name} ===", flush=True) + try: + tok = AutoTokenizer.from_pretrained(d, trust_remote_code=True) + except Exception as e: + tb = traceback.format_exc().strip().splitlines() + print(f" FAILED {type(e).__name__}") + print(" " + "\n ".join(tb[-4:])) + continue + trunc = getattr(tok, "truncation_side", None) + print(f" OK {type(tok).__name__} vocab={len(tok)} truncation_side={trunc}") diff --git a/services/meromero-quant/verify_quant.py b/services/meromero-quant/verify_quant.py new file mode 100644 index 0000000..ddd7c16 --- /dev/null +++ b/services/meromero-quant/verify_quant.py @@ -0,0 +1,92 @@ +"""Did the quant actually do what the recipe says, on the tensors it claims? + +`rc=0` and a plausible file size prove neither. The two things the operator asked +for -- W4A16, vision towers intact -- are properties of the tensor table, so read +the tensor table. Parses safetensors headers directly (u64 length + JSON), so no +torch, no GPU, and no 20 GB load. + +Checks, per module family: + * language-model Linears -> must be NVFP4-packed (uint8 blobs + *_scale companions) + * vision / audio towers -> must still be BF16, i.e. PRESERVED not dropped + * embeddings / lm_head / norms -> BF16 per the ignore list + +Run it against a known-good tree as well. A checker that has only ever seen the +tree it was written for cannot tell "correct" from "blind". +""" +import argparse +import json +import struct +from collections import defaultdict +from pathlib import Path + + +def tensors(root: Path): + """Yield (name, dtype, shape) for every tensor in a sharded or single-file tree.""" + idx = root / "model.safetensors.index.json" + files = sorted({Path(v) for v in json.loads(idx.read_text())["weight_map"].values()}) \ + if idx.exists() else [Path("model.safetensors")] + for f in files: + p = root / f + with p.open("rb") as fh: + n = struct.unpack(" str: + if "vision_tower" in name or "embed_vision" in name: + return "vision_tower" + if "audio_tower" in name or "embed_audio" in name: + return "audio_tower" + if "multi_modal_projector" in name or "mm_projector" in name: + return "projector" + if "embed_tokens" in name: + return "embeddings" + if name.startswith("lm_head") or ".lm_head" in name: + return "lm_head" + if "norm" in name: + return "norms" + if "language_model" in name or ".layers." in name: + return "language_model" + return "other" + + +ap = argparse.ArgumentParser() +ap.add_argument("trees", nargs="+") +a = ap.parse_args() + +for t in a.trees: + root = Path(t) + print(f"\n{'='*70}\n{root}") + cfg = json.loads((root / "config.json").read_text()) + q = cfg.get("quantization_config", {}) + groups = q.get("config_groups", {}) + for gname, g in groups.items(): + w = g.get("weights", {}) + i = g.get("input_activations") + print(f" {gname}: weights num_bits={w.get('num_bits')} type={w.get('type')} " + f"strategy={w.get('strategy')} | input_activations=" + f"{'None (WEIGHT-ONLY)' if i is None else i}") + print(f" format={q.get('format')} kv_cache_scheme={q.get('kv_cache_scheme')} " + f"status={q.get('quantization_status')}") + tc = cfg.get("text_config", {}) + print(f" text_config: per_layer_config={'PRESENT' if 'per_layer_config' in tc else 'absent'}" + f" head_dim={tc.get('head_dim')} global_head_dim={tc.get('global_head_dim')}" + f" num_key_value_heads={tc.get('num_key_value_heads')}" + f" num_global_key_value_heads={tc.get('num_global_key_value_heads')}") + + by = defaultdict(lambda: defaultdict(int)) + packed = defaultdict(int) + for name, dt, shape in tensors(root): + f = family(name) + by[f][dt] += 1 + if name.endswith("weight_packed") or name.endswith("weight_scale"): + packed[f] += 1 + print(" tensor dtypes by family:") + for f in sorted(by): + dts = ", ".join(f"{d}x{c}" for d, c in sorted(by[f].items())) + note = f" [{packed[f]} packed/scale tensors]" if packed[f] else "" + print(f" {f:16} {dts}{note}")