Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
This commit is contained in:
@@ -119,4 +119,45 @@ GPU1 beside the current tenants regardless.
|
||||
Instruments and the full write-up: `services/meromero-quant/`. General lessons:
|
||||
`docs/pfi/model-quantization-playbook.md` §3.16, **§3.17 (new)**, §4.3.
|
||||
|
||||
|
||||
## The A4B reached a live seat while broken — and looked healthy doing it
|
||||
|
||||
Operator, later the same day: *"replace that a4b moe over pfish-6 — remove the pfish-6 alias and
|
||||
create an alias for char-rp-fast."* The A4B went onto the `erp-seat` seat (ana-ml2 `:8021`) and
|
||||
**served NaN**.
|
||||
|
||||
Cause: the morning's batch used the **dense** recipe for a **MoE** model. Its IGNORE list has no
|
||||
`re:.*router.*`, so all 30 MoE routers were quantized to NVFP4, and a 4-bit router does not degrade
|
||||
expert selection — it changes which experts run.
|
||||
|
||||
**Nothing in the normal startup path showed it.** Quant `rc=0`. Healthcheck green in 210 s. Engine
|
||||
log reported the correct KV pool. `/v1/models` correct. Every completion came back
|
||||
`finish_reason=length` with the **full** token count and `content: null` — the model was generating
|
||||
a full budget of tokens that decoded to the empty string. Raw `/v1/completions` was empty too, which
|
||||
ruled out the chat template and the reasoning parser. The one signal that named it: `logprobs: 1`
|
||||
→ HTTP 400 `Out of range float values are not JSON compliant: nan`.
|
||||
|
||||
⚠⚠ **The durable lesson is about the CONTROL, not the router.** That broken tree HAD been
|
||||
structurally diffed before it shipped — and passed — against a verified-good **dense** 31B quant of
|
||||
the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the
|
||||
single thing that control could not distinguish. **A positive control is only worth what it can
|
||||
distinguish; "same family" is not "same architecture class."** Diffing instead against **Pfish-6**
|
||||
— a known-good NVFP4A16 quant of the same 26B-A4B MoE — gave the answer in one line: 222 ignore
|
||||
entries against 252, the 30 missing being exactly `model.language_model.layers.N.router.proj`.
|
||||
|
||||
Re-quantized with `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts
|
||||
11,520 expert Linears and refuses a router in the quantize set, both before any GPU time. 90 s.
|
||||
Live seat then passed prose / vision / tool-call / logprobs. Broken tree parked as
|
||||
`...-NVFP4A16.BROKEN-routers-quantized-20260910`.
|
||||
|
||||
**§4.4's temp port was not reachable, and the fallback mattered.** 15.9 GiB of weights + KV +
|
||||
multimodal encoder-cache profiling does not fit in ~19 GiB free beside GPU1's six other tenants:
|
||||
`gpu-memory-utilization 0.20` refused admission (18.26 free vs 18.99 wanted) and `0.185` OOM'd in
|
||||
encoder profiling. Substitute: named `.env` backup, prove the seat on its real port **while no
|
||||
gateway alias points at it**, move the alias last. That ordering is the only reason a NaN-serving
|
||||
seat never reached a consumer. Cost: ~16 min of seat downtime, twice; zero broken aliases.
|
||||
|
||||
Runbook: `services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md`. Playbook §3.15 (failure signature +
|
||||
the control-class rule), §4.4 (ask for logprobs once).
|
||||
|
||||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|
||||
|
||||
Reference in New Issue
Block a user