vh
9a916a759f
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
114 lines
6.2 KiB
Markdown
114 lines
6.2 KiB
Markdown
# char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (2026-09-10)
|
||
|
||
Operator: *"replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an
|
||
alias for char-rp-fast."*
|
||
|
||
Result: `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` is live on ana-ml2 `:8021`
|
||
behind gateway alias `char-rp-fast`. `Pfish-6` is gone from the gateway. It took two
|
||
attempts, because the first quant was broken in a way that looks exactly like a healthy seat.
|
||
|
||
## ⚠ The failure worth remembering: a 4-bit MoE router serves NaN and passes its healthcheck
|
||
|
||
The A4B built that morning used `services/meromero-quant/quant_a16_datafree.py` — the **dense**
|
||
v2-31B recipe. Its IGNORE list has no `re:.*router.*` entry, so all 30 MoE routers were quantized
|
||
to NVFP4. A 4-bit router does not degrade expert selection, it *changes which experts run*
|
||
(playbook §3.15).
|
||
|
||
What that looked like on the seat, in order of how convincing each signal was:
|
||
|
||
| signal | what it said |
|
||
|---|---|
|
||
| quant exit code | `rc=0`, 16 G, no warning |
|
||
| `docker` healthcheck | healthy in 210 s |
|
||
| engine log | KV pool 534,649 tokens, 2.04x — exactly right |
|
||
| `/v1/models` | correct served name, 262,144 context |
|
||
| every completion | `finish_reason: "length"`, **full** `completion_tokens` (120/120, 600/600) |
|
||
| `content` | `null`. Every time. |
|
||
| raw `/v1/completions` | `text: ''` — so it was not the chat template or the reasoning parser |
|
||
| **`logprobs: 1`** | **HTTP 400 `Out of range float values are not JSON compliant: nan`** |
|
||
|
||
The model was generating a full budget of tokens that decoded to the empty string, and the only
|
||
thing that named the fault was asking for logprobs. `seat_verify.py` now carries that as check 6.
|
||
|
||
**What actually found it** was not the CPU forward (started, then abandoned as too slow): it was
|
||
diffing `quantization_config.ignore` against **Pfish-6** — a known-good NVFP4A16 quant of the
|
||
*same architecture class*. 222 entries against 252, and the 30 missing were precisely
|
||
`model.language_model.layers.N.router.proj`.
|
||
|
||
⚠⚠ **The broken tree HAD been structurally diffed before it shipped — against a verified-good
|
||
DENSE 31B quant of the same Gemma-4 family, which came back clean.** A dense model has no
|
||
routers, so the one thing that was wrong was the one thing that control could not see. **A
|
||
positive control is only worth what it can distinguish; "same family" is not "same architecture
|
||
class."**
|
||
|
||
Fix: re-quantize with `quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts
|
||
`layers × experts × 3 = 11,520` expert Linears and refuses if a router lands in the quantize set,
|
||
both before any GPU time. 90 seconds end to end. The broken tree is parked on ana-ml2 as
|
||
`...-NVFP4A16.BROKEN-routers-quantized-20260910`. **Do not serve it.**
|
||
|
||
## Why the seat went dark for ~16 minutes instead of not at all
|
||
|
||
Playbook §4.4 wants a temp port. It was not reachable, twice, and the numbers are worth keeping:
|
||
|
||
- `--gpu-memory-utilization 0.20` → **admission refused**: `Free memory on device cuda:0
|
||
(18.26/94.97 GiB) on startup is less than desired GPU memory utilization (0.2, 18.99 GiB)`.
|
||
- `0.185` + `--kv-cache-memory 1.5 GB` + `--max-model-len 8192` + `--enforce-eager` → past
|
||
admission, past the KV reservation, then `torch.OutOfMemoryError` during **multimodal
|
||
encoder-cache profiling** (`profiled with 3 video items of the maximum feature size`). That
|
||
profiling cost is easy to forget when budgeting a vision model.
|
||
|
||
15.9 GiB of weights plus a KV pool plus vision profiling does not fit in the ~19 GiB free beside
|
||
the other six GPU1 tenants. So the substitute was **reversibility and ordering**:
|
||
|
||
1. back the host `.env` up to a *named* file first (`.env.pfish6.bak-20260910`);
|
||
2. swap `.env`, `up -d`, and prove the seat on its real port **while no gateway alias points at
|
||
it**;
|
||
3. move the gateway alias **last**.
|
||
|
||
That ordering is why the NaN-serving seat never reached a consumer — `char-rp-fast` did not exist
|
||
yet and `Pfish-6` still resolved to nothing else. The cost was ~16 minutes of that one seat being
|
||
down, twice, and nothing downstream saw a broken alias.
|
||
|
||
## The swap, as steps
|
||
|
||
```bash
|
||
# on ana-ml2, /opt/docker/compose/erp-seat
|
||
cp -n .env .env.pfish6.bak-20260910 # ROLLBACK LIVES HERE
|
||
# point ERP_MODEL / ERP_SERVED_NAME / ERP_CHAT_TEMPLATE at the new tree
|
||
sudo docker compose up -d # ~210 s to healthy
|
||
|
||
# verify BEFORE touching the gateway
|
||
python3 seat_verify.py http://127.0.0.1:8021/v1 <served-name>
|
||
|
||
# gateway (canonical: stacks/litellm/conf/config.yaml)
|
||
scripts/deploy-stack.sh ana-docker litellm --conf
|
||
ssh ana-docker 'cd /opt/docker/compose/litellm && sudo docker compose restart litellm'
|
||
```
|
||
|
||
**Rollback to Pfish-6** is `cp .env.pfish6.bak-20260910 .env && sudo docker compose up -d`,
|
||
~4 minutes. `/tank/aimodels/erp-tune-v6-nvfp4a16` is untouched.
|
||
|
||
## What was checked, and what was not
|
||
|
||
Verified on the live seat (`raw/char-rp-fast-seat-verification-2026-09-10.txt`): served name and
|
||
262,144 context; KV 534,649 tokens / 2.04x; clean prose with no `<|channel>thought` leak and no
|
||
reasoning field; **a solid-colour image read correctly**, so vision is tested rather than inferred
|
||
from a tensor count; an auto `tool_choice` call parsed with correct arguments; finite logprobs.
|
||
Through the gateway with the shared `all-agents-local` key: `char-rp-fast` answers, `Pfish-6`
|
||
returns an explicit `400 Invalid model name` rather than a substitution, and `char-rp` /
|
||
`char-rp-reasoning` are both unaffected.
|
||
|
||
Audited before removing the alias: **0 of 17 LiteLLM keys** named `Pfish-6` in their model
|
||
allowlist, so nothing was orphaned (1 of 17 is unrestricted and reaches whatever the gateway
|
||
serves). ⚠ The first attempt at that audit passed `size=200` and got a silent `422`, which the
|
||
script reported as "scanned 0 keys" — an empty result and a rejected query look identical if you
|
||
do not check.
|
||
|
||
**Not established:** anything about quality. No RP eval, no long-context check, no A/B against
|
||
Pfish-6 or `char-rp`. The samplers are the author's card values (Temp 0.8–1.0, MinP 0.05), not
|
||
tuned here. n=1 smoke output is not evidence about writing.
|
||
|
||
⚠ Pre-existing doc rot noticed and **not** fixed: the `char-rp` comment block in
|
||
`stacks/litellm/conf/config.yaml` still describes its `:8016` seat as MeroMero-v2. That has been
|
||
stale since the 2026-08-24 swap to stock Gemma-4.
|