vh
9a916a759f
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
6.2 KiB
char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (2026-09-10)
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast."
Result: G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021
behind gateway alias char-rp-fast. Pfish-6 is gone from the gateway. It took two
attempts, because the first quant was broken in a way that looks exactly like a healthy seat.
⚠ The failure worth remembering: a 4-bit MoE router serves NaN and passes its healthcheck
The A4B built that morning used services/meromero-quant/quant_a16_datafree.py — the dense
v2-31B recipe. Its IGNORE list has no re:.*router.* entry, so all 30 MoE routers were quantized
to NVFP4. A 4-bit router does not degrade expert selection, it changes which experts run
(playbook §3.15).
What that looked like on the seat, in order of how convincing each signal was:
| signal | what it said |
|---|---|
| quant exit code | rc=0, 16 G, no warning |
docker healthcheck |
healthy in 210 s |
| engine log | KV pool 534,649 tokens, 2.04x — exactly right |
/v1/models |
correct served name, 262,144 context |
| every completion | finish_reason: "length", full completion_tokens (120/120, 600/600) |
content |
null. Every time. |
raw /v1/completions |
text: '' — so it was not the chat template or the reasoning parser |
logprobs: 1 |
HTTP 400 Out of range float values are not JSON compliant: nan |
The model was generating a full budget of tokens that decoded to the empty string, and the only
thing that named the fault was asking for logprobs. seat_verify.py now carries that as check 6.
What actually found it was not the CPU forward (started, then abandoned as too slow): it was
diffing quantization_config.ignore against Pfish-6 — a known-good NVFP4A16 quant of the
same architecture class. 222 entries against 252, and the 30 missing were precisely
model.language_model.layers.N.router.proj.
⚠⚠ The broken tree HAD been structurally diffed before it shipped — against a verified-good DENSE 31B quant of the same Gemma-4 family, which came back clean. A dense model has no routers, so the one thing that was wrong was the one thing that control could not see. A positive control is only worth what it can distinguish; "same family" is not "same architecture class."
Fix: re-quantize with quant_nvfp4a16_gemma4_moe.py, whose --dry-run asserts
layers × experts × 3 = 11,520 expert Linears and refuses if a router lands in the quantize set,
both before any GPU time. 90 seconds end to end. The broken tree is parked on ana-ml2 as
...-NVFP4A16.BROKEN-routers-quantized-20260910. Do not serve it.
Why the seat went dark for ~16 minutes instead of not at all
Playbook §4.4 wants a temp port. It was not reachable, twice, and the numbers are worth keeping:
--gpu-memory-utilization 0.20→ admission refused:Free memory on device cuda:0 (18.26/94.97 GiB) on startup is less than desired GPU memory utilization (0.2, 18.99 GiB).0.185+--kv-cache-memory 1.5 GB+--max-model-len 8192+--enforce-eager→ past admission, past the KV reservation, thentorch.OutOfMemoryErrorduring multimodal encoder-cache profiling (profiled with 3 video items of the maximum feature size). That profiling cost is easy to forget when budgeting a vision model.
15.9 GiB of weights plus a KV pool plus vision profiling does not fit in the ~19 GiB free beside the other six GPU1 tenants. So the substitute was reversibility and ordering:
- back the host
.envup to a named file first (.env.pfish6.bak-20260910); - swap
.env,up -d, and prove the seat on its real port while no gateway alias points at it; - move the gateway alias last.
That ordering is why the NaN-serving seat never reached a consumer — char-rp-fast did not exist
yet and Pfish-6 still resolved to nothing else. The cost was ~16 minutes of that one seat being
down, twice, and nothing downstream saw a broken alias.
The swap, as steps
# on ana-ml2, /opt/docker/compose/erp-seat
cp -n .env .env.pfish6.bak-20260910 # ROLLBACK LIVES HERE
# point ERP_MODEL / ERP_SERVED_NAME / ERP_CHAT_TEMPLATE at the new tree
sudo docker compose up -d # ~210 s to healthy
# verify BEFORE touching the gateway
python3 seat_verify.py http://127.0.0.1:8021/v1 <served-name>
# gateway (canonical: stacks/litellm/conf/config.yaml)
scripts/deploy-stack.sh ana-docker litellm --conf
ssh ana-docker 'cd /opt/docker/compose/litellm && sudo docker compose restart litellm'
Rollback to Pfish-6 is cp .env.pfish6.bak-20260910 .env && sudo docker compose up -d,
~4 minutes. /tank/aimodels/erp-tune-v6-nvfp4a16 is untouched.
What was checked, and what was not
Verified on the live seat (raw/char-rp-fast-seat-verification-2026-09-10.txt): served name and
262,144 context; KV 534,649 tokens / 2.04x; clean prose with no <|channel>thought leak and no
reasoning field; a solid-colour image read correctly, so vision is tested rather than inferred
from a tensor count; an auto tool_choice call parsed with correct arguments; finite logprobs.
Through the gateway with the shared all-agents-local key: char-rp-fast answers, Pfish-6
returns an explicit 400 Invalid model name rather than a substitution, and char-rp /
char-rp-reasoning are both unaffected.
Audited before removing the alias: 0 of 17 LiteLLM keys named Pfish-6 in their model
allowlist, so nothing was orphaned (1 of 17 is unrestricted and reaches whatever the gateway
serves). ⚠ The first attempt at that audit passed size=200 and got a silent 422, which the
script reported as "scanned 0 keys" — an empty result and a rejected query look identical if you
do not check.
Not established: anything about quality. No RP eval, no long-context check, no A/B against
Pfish-6 or char-rp. The samplers are the author's card values (Temp 0.8–1.0, MinP 0.05), not
tuned here. n=1 smoke output is not evidence about writing.
⚠ Pre-existing doc rot noticed and not fixed: the char-rp comment block in
stacks/litellm/conf/config.yaml still describes its :8016 seat as MeroMero-v2. That has been
stale since the 2026-08-24 swap to stock Gemma-4.