vh
9a916a759f
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
164 lines
10 KiB
Markdown
164 lines
10 KiB
Markdown
# `[2026-09-10]` MeroMero — both quants landed; the v2 dense took five attempts
|
||
|
||
Operator wanted a MeroMero seat. Getting there cost five quant attempts and corrected three wrong
|
||
hypotheses (two of the previous session's, one of mine), so the failure chain is the durable part.
|
||
**Both outputs now exist and are verified against a known-good tree. Neither is serving yet.**
|
||
|
||
## The family, because I got it wrong first
|
||
|
||
I called the unsuffixed 31B "v1" because v2 is `v2-31B`. The operator caught it. Creation dates from
|
||
the upstream author settle it:
|
||
|
||
2026-04-15 zerofata/G4-MeroMero-26B-A4B 30 layers, kv 8 MoE 128 experts ← the ORIGINAL
|
||
2026-04-30 zerofata/G4-MeroMero-31B 60 layers, kv 16 DENSE ← a separate line
|
||
2026-07-31 zerofata/G4-MeroMero-v2-31B 60 layers, kv 16 DENSE
|
||
|
||
Not a v1→v2 sequence at one size: an A4B MoE first, then a dense 31B line that later got a v2.
|
||
⚠ The `-v1-` repo 401s, and I read that as "v1 must be the unsuffixed one" instead of checking dates.
|
||
|
||
**The geometry is why it matters.** The A4B is 30 layers / kv 8 — *identical to Pfish-6* (a gemma4
|
||
26B-A4B) — so it drops into the existing KV budget at 262k. The dense 31B is 60 layers / kv 16, ~4x
|
||
the KV per token, and did NOT fit 262k on GPU1 alongside the other seats.
|
||
|
||
## Published quants: none of them are our scheme
|
||
|
||
pekkAi/G4-MeroMero-31B-NVFP4 true v1 dense, W4A4
|
||
llmfan46/G4-MeroMero-31B-uncensored-heretic-NVFP4 abliterated dense, W4A16_NVFP4 (pulled, 19 GB)
|
||
pekkAi/G4-MeroMero-26B-A4B-it-...-heretic-NVFP4 abliterated A4B, W4A4 + FP8 KV baked in
|
||
NVFP4 of v2-31B DOES NOT EXIST, from anyone (0 of 27 v2 repos)
|
||
|
||
W4A4 carries our own measured long-context collapse (Granite, 30k ctx), which is why every seat we
|
||
run is A16. So the operator ruled: pull both ablits bf16 and quantize in-house at W4A16.
|
||
|
||
## MTP — checked on the SOURCES, not inferred
|
||
|
||
Gemma-4 ships **no MTP head at all**: 0 mtp tensors and no mtp config keys in the A4B bf16, the v2
|
||
bf16, Pfish-6, or the published quants. The "mtp if applicable" clause is a no-op here; the MTP work
|
||
in our history is Qwen-family.
|
||
|
||
## A4B — SUCCEEDED
|
||
|
||
`G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`, 16 G, 7 files, **2m08s**. That time confirms
|
||
playbook §3.16: weight-only A16 runs a `DataFreePipeline` and never touches the dataset.
|
||
⚠ Output is missing `preprocessor_config.json` — §4.3 post-step 2 still owed before serving.
|
||
|
||
## v2 dense — FOUR failures, each a different layer
|
||
|
||
1. **`num_key_value_heads` is None** at `Gemma4TextAttention.__init__`. I hypothesised a transformers
|
||
version deadlock (config authored by 5.16.1, container on 5.12.1) — **wrong**: all three configs
|
||
construct fine on 5.12.1. Then I hypothesised the stale `torch_dtype` kwarg reshaping the config —
|
||
**also wrong**: every kwarg combination returns 16.
|
||
⚠ **A 2-layer truncation PASSED and hid it.** The failing local is chosen per layer type; only
|
||
`full_attention` layers take the branch. Testing each layer type individually found it in seconds.
|
||
Real cause: DogOnKeyboard's config sets `attention_k_eq_v: true` but **omits
|
||
`num_global_key_value_heads` and `global_head_dim`** — a malformed upload, not a toolchain issue.
|
||
Patched from zerofata's canonical values (4 / 512) after **shape-verifying** the checkpoint:
|
||
full-attn k_proj `[2048,5376]` = 4×512, sliding `[4096,5376]` = 16×256, identical to canonical.
|
||
Original at `config.json.bak-pre-kvfields-20260910`.
|
||
2. **`initialize model processor ... required when a dataset is provided`** — DogOnKeyboard ships no
|
||
`processor_config.json` (the A4B does; that is why it succeeded). Copied from canonical, AND
|
||
dropped the dataset entirely, since §3.16 says A16 ignores it and §3.14 says building it bakes a
|
||
truncation cap into the shipped tokenizer. Removing it kills both for zero loss.
|
||
3. **My own bug**: the reference module runs argparse with `required=True` at IMPORT, so blanking
|
||
`sys.argv` still exited 2. Placeholder args, real argv restored after.
|
||
4. **`AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`** — RESOLVED,
|
||
and **not what it looked like**. See "The attempt-4 trap" below.
|
||
5. Clean. `rc=0`, 19 G, 3m07s.
|
||
|
||
⚠ **My wrapper reported `rc=0` on a failed run** because it read `$?` after an `echo`. A wrapper that
|
||
reports success on failure is the false-reassurance class; fixed to capture `$?` immediately.
|
||
|
||
|
||
## The attempt-4 trap — the toolchain moved, the config was fine
|
||
|
||
The standing hypothesis was that DogOnKeyboard's `per_layer_config` key was the defect and
|
||
`allow_global_per_layer_attribute_access=True` was the fix. Both halves were wrong, and the second
|
||
half would have shipped a risk for no reason.
|
||
|
||
**`pip install llmcompressor==0.13.0` downgrades transformers 5.16.1 → 5.14.1.** The config was
|
||
serialized *by* 5.16.1, which materializes `per_layer_config` out of `global_head_dim` +
|
||
`layer_types`; 5.14.1 has the heterogeneity guard but not the gemma4 resolver, so it refuses the
|
||
global read. Under the image's own 5.16.1 the identical config loads fine. On top of that,
|
||
`vllm/vllm-openai:latest` was re-pulled *during attempt 4 and in no earlier run* — the pull line is
|
||
in that block alone — so the error changed for reasons unrelated to anything under test.
|
||
|
||
Two things made this findable, and neither was inspection:
|
||
|
||
- **Reproducing the real call.** A bare `AutoConfig.from_pretrained` does not reproduce it; the
|
||
trigger is reached through `AutoTokenizer`. Testing the config alone would have cleared it.
|
||
- **A known-good positive control.** zerofata's canonical v2, quantized cleanly on 2026-08-21, is
|
||
what separated "this upload is malformed" from "this toolchain moved". Four green variants and one
|
||
red one are noise without it.
|
||
|
||
**Fix: drop `per_layer_config`, don't force global access.** It is exactly redundant — keys are
|
||
precisely the ten `full_attention` layer indices, sole value `(512, 4)`, verbatim the global fields.
|
||
Forcing instead leaves the config heterogeneous and makes `config.head_dim` answer 256 to every
|
||
caller including the ones building the 512-wide layers; geometry survived it in a meta-device check,
|
||
but llmcompressor's onloading is an unaudited caller and that is what the warning is about. The
|
||
patch re-proves the redundancy at apply time and refuses if it ever stops holding.
|
||
|
||
## What landed, and what is verified
|
||
|
||
- `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G**, and its tensor table is **identical family for
|
||
family and count for count to the 2026-08-21 canonical quant**: 410 U8 packed + 410 F8_E4M3 +
|
||
410 F32 scales on the LM Linears, **356 BF16 vision-tower tensors preserved**,
|
||
`input_activations=None` (genuinely A16). Shard sizes match that tree byte for byte.
|
||
- CPU load-and-generate: 0 tensors left on meta, decompresses, emits coherent prose. n=1, greedy,
|
||
24 tokens — an "is it wired up" check and nothing more.
|
||
- ⚠ **The A4B output had the §3.14 truncation cap baked in** (`max_length: 8192`), because it was
|
||
quantized *with* the corpus. Caught and fixed; backup at `tokenizer.json.bak-pre-truncfix`. The v2,
|
||
run data-free, came out `truncation: null`.
|
||
|
||
## Still owed
|
||
|
||
**The §4.4 serve test has NOT run.** GPU1 has 19.9 GB free against 19.5 GB of weights, so it cannot
|
||
happen without displacing a live seat — operator's call. Until it does, *"vllm servable"* is
|
||
unverified for this tree, and the dense 31B's 60-layer / kv-16 geometry still does not fit 262k on
|
||
GPU1 beside the current tenants regardless.
|
||
|
||
Instruments and the full write-up: `services/meromero-quant/`. General lessons:
|
||
`docs/pfi/model-quantization-playbook.md` §3.16, **§3.17 (new)**, §4.3.
|
||
|
||
|
||
## The A4B reached a live seat while broken — and looked healthy doing it
|
||
|
||
Operator, later the same day: *"replace that a4b moe over pfish-6 — remove the pfish-6 alias and
|
||
create an alias for char-rp-fast."* The A4B went onto the `erp-seat` seat (ana-ml2 `:8021`) and
|
||
**served NaN**.
|
||
|
||
Cause: the morning's batch used the **dense** recipe for a **MoE** model. Its IGNORE list has no
|
||
`re:.*router.*`, so all 30 MoE routers were quantized to NVFP4, and a 4-bit router does not degrade
|
||
expert selection — it changes which experts run.
|
||
|
||
**Nothing in the normal startup path showed it.** Quant `rc=0`. Healthcheck green in 210 s. Engine
|
||
log reported the correct KV pool. `/v1/models` correct. Every completion came back
|
||
`finish_reason=length` with the **full** token count and `content: null` — the model was generating
|
||
a full budget of tokens that decoded to the empty string. Raw `/v1/completions` was empty too, which
|
||
ruled out the chat template and the reasoning parser. The one signal that named it: `logprobs: 1`
|
||
→ HTTP 400 `Out of range float values are not JSON compliant: nan`.
|
||
|
||
⚠⚠ **The durable lesson is about the CONTROL, not the router.** That broken tree HAD been
|
||
structurally diffed before it shipped — and passed — against a verified-good **dense** 31B quant of
|
||
the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the
|
||
single thing that control could not distinguish. **A positive control is only worth what it can
|
||
distinguish; "same family" is not "same architecture class."** Diffing instead against **Pfish-6**
|
||
— a known-good NVFP4A16 quant of the same 26B-A4B MoE — gave the answer in one line: 222 ignore
|
||
entries against 252, the 30 missing being exactly `model.language_model.layers.N.router.proj`.
|
||
|
||
Re-quantized with `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts
|
||
11,520 expert Linears and refuses a router in the quantize set, both before any GPU time. 90 s.
|
||
Live seat then passed prose / vision / tool-call / logprobs. Broken tree parked as
|
||
`...-NVFP4A16.BROKEN-routers-quantized-20260910`.
|
||
|
||
**§4.4's temp port was not reachable, and the fallback mattered.** 15.9 GiB of weights + KV +
|
||
multimodal encoder-cache profiling does not fit in ~19 GiB free beside GPU1's six other tenants:
|
||
`gpu-memory-utilization 0.20` refused admission (18.26 free vs 18.99 wanted) and `0.185` OOM'd in
|
||
encoder profiling. Substitute: named `.env` backup, prove the seat on its real port **while no
|
||
gateway alias points at it**, move the alias last. That ordering is the only reason a NaN-serving
|
||
seat never reached a consumer. Cost: ~16 min of seat downtime, twice; zero broken aliases.
|
||
|
||
Runbook: `services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md`. Playbook §3.15 (failure signature +
|
||
the control-class rule), §4.4 (ask for logprobs once).
|
||
|
||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|