Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
This commit is contained in:
@@ -291,6 +291,43 @@ hit it exactly, the recipe is wrong and the failure is silent.
|
||||
⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
|
||||
error does not average out downstream, it changes which weights run at all.
|
||||
|
||||
**And here is what that actually looks like when it ships — measured 2026-09-10 on the MeroMero
|
||||
26B-A4B, which reached a live seat before anyone noticed.** The A4B was quantized with the *dense*
|
||||
recipe (`services/meromero-quant/quant_a16_datafree.py`), whose IGNORE list has no `re:.*router.*`
|
||||
entry. All 30 routers went to NVFP4. Then:
|
||||
|
||||
- the quant **completed cleanly**, 16 G, no warning;
|
||||
- the tensor table looked **plausible** — 11,755 quantized modules against 11,725 in the
|
||||
known-good build, a 0.26% difference nobody eyeballs;
|
||||
- vLLM **started, passed its healthcheck, and reported the correct KV pool**;
|
||||
- every request returned `finish_reason: "length"` with the **full completion_tokens count** —
|
||||
120 of 120, 600 of 600 — and `content: null`. The model was generating, and every token decoded
|
||||
to the empty string;
|
||||
- the give-away was **`logprobs` coming back NaN**, which surfaced only because a diagnostic asked
|
||||
for them and vLLM refused to serialize the response: `Out of range float values are not JSON
|
||||
compliant: nan`.
|
||||
|
||||
⚠⚠ **So the router mistake has no symptom you would catch by watching a seat come up.** It has
|
||||
exactly one cheap tell, and §4.4 now carries it: ask for `logprobs` once.
|
||||
|
||||
**Two guards, both cheap, both would have caught this before the seat:**
|
||||
|
||||
1. **Use the architecture-class-correct recipe and let its guard fire.**
|
||||
`services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` refuses outright — *"⚠ REFUSING: a
|
||||
router/vision/audio Linear is in the quantize set"* — and asserts `layers × experts × 3` before
|
||||
any GPU time. Its `--dry-run` does the whole check with no GPU and no save. The dense recipe has
|
||||
neither guard and will happily eat a MoE.
|
||||
2. **Diff `quantization_config.ignore` against a known-good quant of the SAME ARCHITECTURE CLASS.**
|
||||
The broken build had **222** ignore entries against the good build's **252**; the 30 missing
|
||||
were exactly `model.language_model.layers.N.router.proj`. That diff is a two-line script and it
|
||||
names the defect precisely.
|
||||
|
||||
⚠ **The control has to match the architecture class, not just the family.** The broken A4B *was*
|
||||
structurally diffed before it shipped — against a verified-good **dense** 31B quant of the same
|
||||
Gemma-4 family. A dense model has no routers, so the one thing that was wrong was the one thing
|
||||
that control could not see, and the comparison came back clean. A positive control is only worth
|
||||
what it can distinguish; "same family" is not "same architecture class".
|
||||
|
||||
### 3.4 Toolchain version deadlocks
|
||||
|
||||
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
|
||||
@@ -629,6 +666,25 @@ a `--check` mode; point it at a tree you already trust before you trust its verd
|
||||
Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
|
||||
only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.
|
||||
|
||||
**Ask for `logprobs` once, on the temp port, before the alias moves.** A seat can pass its
|
||||
healthcheck, report the right KV pool, and answer every request with the full token count while
|
||||
every token decodes to the empty string — that is what a router-quantized MoE does (§3.15). NaN
|
||||
logits are invisible to `/health`, invisible to the token counts, and invisible to a tensor-table
|
||||
diff against the wrong control; a single `logprobs: 1` request surfaces them immediately, because
|
||||
vLLM cannot even serialize the response (`Out of range float values are not JSON compliant: nan`).
|
||||
Add it to the smoke set: **served name, one prose completion, one image if the model is
|
||||
multimodal, one tool call, and one `logprobs` request.**
|
||||
|
||||
⚠ **A co-resident temp port is not always reachable, and the fallback is reversibility, not
|
||||
skipping the test.** Measured 2026-09-10: with 19 GiB free on a shared card, a 16 G A4B refused
|
||||
admission at `gpu-memory-utilization 0.20` (18.26 free vs 18.99 requested), and at 0.185 it got
|
||||
past admission and past the KV reservation only to OOM in **multimodal encoder-cache profiling**
|
||||
(`profiled with 3 video items of the maximum feature size`) — a cost easy to forget when budgeting
|
||||
a vision model. When the card genuinely cannot hold both, the substitute is: back up the host
|
||||
`.env` to a named file first, prove the new seat on its real port **while no gateway alias points
|
||||
at it**, and move the alias last. That ordering is what kept a NaN-serving seat away from every
|
||||
consumer; the seat itself was down ~16 minutes and nothing downstream saw a broken alias.
|
||||
|
||||
---
|
||||
|
||||
## 5. The acceptance gate — and how measurement lies to you
|
||||
|
||||
Reference in New Issue
Block a user