esh-pfi-infrastructure/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md

[2026-09-10] MeroMero — both quants landed; the v2 dense took five attempts

Operator wanted a MeroMero seat. Getting there cost five quant attempts and corrected three wrong hypotheses (two of the previous session's, one of mine), so the failure chain is the durable part. Both outputs now exist and are verified against a known-good tree. Neither is serving yet.

The family, because I got it wrong first

I called the unsuffixed 31B "v1" because v2 is v2-31B. The operator caught it. Creation dates from the upstream author settle it:

2026-04-15  zerofata/G4-MeroMero-26B-A4B   30 layers, kv 8   MoE 128 experts  ← the ORIGINAL
2026-04-30  zerofata/G4-MeroMero-31B       60 layers, kv 16  DENSE            ← a separate line
2026-07-31  zerofata/G4-MeroMero-v2-31B    60 layers, kv 16  DENSE

Not a v1→v2 sequence at one size: an A4B MoE first, then a dense 31B line that later got a v2. ⚠ The -v1- repo 401s, and I read that as "v1 must be the unsuffixed one" instead of checking dates.

The geometry is why it matters. The A4B is 30 layers / kv 8 — identical to Pfish-6 (a gemma4 26B-A4B) — so it drops into the existing KV budget at 262k. The dense 31B is 60 layers / kv 16, ~4x the KV per token, and did NOT fit 262k on GPU1 alongside the other seats.

Published quants: none of them are our scheme

pekkAi/G4-MeroMero-31B-NVFP4                        true v1 dense, W4A4
llmfan46/G4-MeroMero-31B-uncensored-heretic-NVFP4   abliterated dense, W4A16_NVFP4  (pulled, 19 GB)
pekkAi/G4-MeroMero-26B-A4B-it-...-heretic-NVFP4     abliterated A4B, W4A4 + FP8 KV baked in
NVFP4 of v2-31B                                     DOES NOT EXIST, from anyone (0 of 27 v2 repos)

W4A4 carries our own measured long-context collapse (Granite, 30k ctx), which is why every seat we run is A16. So the operator ruled: pull both ablits bf16 and quantize in-house at W4A16.

MTP — checked on the SOURCES, not inferred

Gemma-4 ships no MTP head at all: 0 mtp tensors and no mtp config keys in the A4B bf16, the v2 bf16, Pfish-6, or the published quants. The "mtp if applicable" clause is a no-op here; the MTP work in our history is Qwen-family.

A4B — SUCCEEDED

G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16, 16 G, 7 files, 2m08s. That time confirms playbook §3.16: weight-only A16 runs a DataFreePipeline and never touches the dataset. ⚠ Output is missing preprocessor_config.json — §4.3 post-step 2 still owed before serving.

v2 dense — FOUR failures, each a different layer

1. num_key_value_heads is None at Gemma4TextAttention.__init__. I hypothesised a transformers version deadlock (config authored by 5.16.1, container on 5.12.1) — wrong: all three configs construct fine on 5.12.1. Then I hypothesised the stale torch_dtype kwarg reshaping the config — also wrong: every kwarg combination returns 16. ⚠ A 2-layer truncation PASSED and hid it. The failing local is chosen per layer type; only full_attention layers take the branch. Testing each layer type individually found it in seconds. Real cause: DogOnKeyboard's config sets attention_k_eq_v: true but omits num_global_key_value_heads and global_head_dim — a malformed upload, not a toolchain issue. Patched from zerofata's canonical values (4 / 512) after shape-verifying the checkpoint: full-attn k_proj [2048,5376] = 4×512, sliding [4096,5376] = 16×256, identical to canonical. Original at config.json.bak-pre-kvfields-20260910.
2. initialize model processor ... required when a dataset is provided — DogOnKeyboard ships no processor_config.json (the A4B does; that is why it succeeded). Copied from canonical, AND dropped the dataset entirely, since §3.16 says A16 ignores it and §3.14 says building it bakes a truncation cap into the shipped tokenizer. Removing it kills both for zero loss.
3. My own bug: the reference module runs argparse with required=True at IMPORT, so blanking sys.argv still exited 2. Placeholder args, real argv restored after.
4. AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute — RESOLVED, and not what it looked like. See "The attempt-4 trap" below.
5. Clean. rc=0, 19 G, 3m07s.

⚠ My wrapper reported rc=0 on a failed run because it read $? after an echo. A wrapper that reports success on failure is the false-reassurance class; fixed to capture $? immediately.

The attempt-4 trap — the toolchain moved, the config was fine

The standing hypothesis was that DogOnKeyboard's per_layer_config key was the defect and allow_global_per_layer_attribute_access=True was the fix. Both halves were wrong, and the second half would have shipped a risk for no reason.

pip install llmcompressor==0.13.0 downgrades transformers 5.16.1 → 5.14.1. The config was serialized by 5.16.1, which materializes per_layer_config out of global_head_dim + layer_types; 5.14.1 has the heterogeneity guard but not the gemma4 resolver, so it refuses the global read. Under the image's own 5.16.1 the identical config loads fine. On top of that, vllm/vllm-openai:latest was re-pulled during attempt 4 and in no earlier run — the pull line is in that block alone — so the error changed for reasons unrelated to anything under test.

Two things made this findable, and neither was inspection:

• Reproducing the real call. A bare AutoConfig.from_pretrained does not reproduce it; the trigger is reached through AutoTokenizer. Testing the config alone would have cleared it.
• A known-good positive control. zerofata's canonical v2, quantized cleanly on 2026-08-21, is what separated "this upload is malformed" from "this toolchain moved". Four green variants and one red one are noise without it.

Fix: drop per_layer_config, don't force global access. It is exactly redundant — keys are precisely the ten full_attention layer indices, sole value (512, 4), verbatim the global fields. Forcing instead leaves the config heterogeneous and makes config.head_dim answer 256 to every caller including the ones building the 512-wide layers; geometry survived it in a meta-device check, but llmcompressor's onloading is an unaudited caller and that is what the warning is about. The patch re-proves the redundancy at apply time and refuses if it ever stops holding.

What landed, and what is verified

• G4-MeroMero-v2-31B-heretic-NVFP4A16 — 19 G, and its tensor table is identical family for family and count for count to the 2026-08-21 canonical quant: 410 U8 packed + 410 F8_E4M3 + 410 F32 scales on the LM Linears, 356 BF16 vision-tower tensors preserved, input_activations=None (genuinely A16). Shard sizes match that tree byte for byte.
• CPU load-and-generate: 0 tensors left on meta, decompresses, emits coherent prose. n=1, greedy, 24 tokens — an "is it wired up" check and nothing more.
• ⚠ The A4B output had the §3.14 truncation cap baked in (max_length: 8192), because it was quantized with the corpus. Caught and fixed; backup at tokenizer.json.bak-pre-truncfix. The v2, run data-free, came out truncation: null.

Still owed

The §4.4 serve test has NOT run. GPU1 has 19.9 GB free against 19.5 GB of weights, so it cannot happen without displacing a live seat — operator's call. Until it does, "vllm servable" is unverified for this tree, and the dense 31B's 60-layer / kv-16 geometry still does not fit 262k on GPU1 beside the current tenants regardless.

Instruments and the full write-up: services/meromero-quant/. General lessons: docs/pfi/model-quantization-playbook.md §3.16, §3.17 (new), §4.3.

The A4B reached a live seat while broken — and looked healthy doing it

Operator, later the same day: "replace that a4b moe over pfish-6 — remove the pfish-6 alias and create an alias for char-rp-fast." The A4B went onto the erp-seat seat (ana-ml2 :8021) and served NaN.

Cause: the morning's batch used the dense recipe for a MoE model. Its IGNORE list has no re:.*router.*, so all 30 MoE routers were quantized to NVFP4, and a 4-bit router does not degrade expert selection — it changes which experts run.

Nothing in the normal startup path showed it. Quant rc=0. Healthcheck green in 210 s. Engine log reported the correct KV pool. /v1/models correct. Every completion came back finish_reason=length with the full token count and content: null — the model was generating a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, which ruled out the chat template and the reasoning parser. The one signal that named it: logprobs: 1 → HTTP 400 Out of range float values are not JSON compliant: nan.

⚠⚠ The durable lesson is about the CONTROL, not the router. That broken tree HAD been structurally diffed before it shipped — and passed — against a verified-good dense 31B quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing that control could not distinguish. A positive control is only worth what it can distinguish; "same family" is not "same architecture class." Diffing instead against Pfish-6 — a known-good NVFP4A16 quant of the same 26B-A4B MoE — gave the answer in one line: 222 ignore entries against 252, the 30 missing being exactly model.language_model.layers.N.router.proj.

Re-quantized with services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose --dry-run asserts 11,520 expert Linears and refuses a router in the quantize set, both before any GPU time. 90 s. Live seat then passed prose / vision / tool-call / logprobs. Broken tree parked as ...-NVFP4A16.BROKEN-routers-quantized-20260910.

§4.4's temp port was not reachable, and the fallback mattered. 15.9 GiB of weights + KV + multimodal encoder-cache profiling does not fit in ~19 GiB free beside GPU1's six other tenants: gpu-memory-utilization 0.20 refused admission (18.26 free vs 18.99 wanted) and 0.185 OOM'd in encoder profiling. Substitute: named .env backup, prove the seat on its real port while no gateway alias points at it, move the alias last. That ordering is the only reason a NaN-serving seat never reached a consumer. Cost: ~16 min of seat downtime, twice; zero broken aliases.

Runbook: services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md. Playbook §3.15 (failure signature + the control-class rule), §4.4 (ask for logprobs once).

Related: 2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot