Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
This commit is contained in:
@@ -0,0 +1,113 @@
|
||||
# char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (2026-09-10)
|
||||
|
||||
Operator: *"replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an
|
||||
alias for char-rp-fast."*
|
||||
|
||||
Result: `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` is live on ana-ml2 `:8021`
|
||||
behind gateway alias `char-rp-fast`. `Pfish-6` is gone from the gateway. It took two
|
||||
attempts, because the first quant was broken in a way that looks exactly like a healthy seat.
|
||||
|
||||
## ⚠ The failure worth remembering: a 4-bit MoE router serves NaN and passes its healthcheck
|
||||
|
||||
The A4B built that morning used `services/meromero-quant/quant_a16_datafree.py` — the **dense**
|
||||
v2-31B recipe. Its IGNORE list has no `re:.*router.*` entry, so all 30 MoE routers were quantized
|
||||
to NVFP4. A 4-bit router does not degrade expert selection, it *changes which experts run*
|
||||
(playbook §3.15).
|
||||
|
||||
What that looked like on the seat, in order of how convincing each signal was:
|
||||
|
||||
| signal | what it said |
|
||||
|---|---|
|
||||
| quant exit code | `rc=0`, 16 G, no warning |
|
||||
| `docker` healthcheck | healthy in 210 s |
|
||||
| engine log | KV pool 534,649 tokens, 2.04x — exactly right |
|
||||
| `/v1/models` | correct served name, 262,144 context |
|
||||
| every completion | `finish_reason: "length"`, **full** `completion_tokens` (120/120, 600/600) |
|
||||
| `content` | `null`. Every time. |
|
||||
| raw `/v1/completions` | `text: ''` — so it was not the chat template or the reasoning parser |
|
||||
| **`logprobs: 1`** | **HTTP 400 `Out of range float values are not JSON compliant: nan`** |
|
||||
|
||||
The model was generating a full budget of tokens that decoded to the empty string, and the only
|
||||
thing that named the fault was asking for logprobs. `seat_verify.py` now carries that as check 6.
|
||||
|
||||
**What actually found it** was not the CPU forward (started, then abandoned as too slow): it was
|
||||
diffing `quantization_config.ignore` against **Pfish-6** — a known-good NVFP4A16 quant of the
|
||||
*same architecture class*. 222 entries against 252, and the 30 missing were precisely
|
||||
`model.language_model.layers.N.router.proj`.
|
||||
|
||||
⚠⚠ **The broken tree HAD been structurally diffed before it shipped — against a verified-good
|
||||
DENSE 31B quant of the same Gemma-4 family, which came back clean.** A dense model has no
|
||||
routers, so the one thing that was wrong was the one thing that control could not see. **A
|
||||
positive control is only worth what it can distinguish; "same family" is not "same architecture
|
||||
class."**
|
||||
|
||||
Fix: re-quantize with `quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts
|
||||
`layers × experts × 3 = 11,520` expert Linears and refuses if a router lands in the quantize set,
|
||||
both before any GPU time. 90 seconds end to end. The broken tree is parked on ana-ml2 as
|
||||
`...-NVFP4A16.BROKEN-routers-quantized-20260910`. **Do not serve it.**
|
||||
|
||||
## Why the seat went dark for ~16 minutes instead of not at all
|
||||
|
||||
Playbook §4.4 wants a temp port. It was not reachable, twice, and the numbers are worth keeping:
|
||||
|
||||
- `--gpu-memory-utilization 0.20` → **admission refused**: `Free memory on device cuda:0
|
||||
(18.26/94.97 GiB) on startup is less than desired GPU memory utilization (0.2, 18.99 GiB)`.
|
||||
- `0.185` + `--kv-cache-memory 1.5 GB` + `--max-model-len 8192` + `--enforce-eager` → past
|
||||
admission, past the KV reservation, then `torch.OutOfMemoryError` during **multimodal
|
||||
encoder-cache profiling** (`profiled with 3 video items of the maximum feature size`). That
|
||||
profiling cost is easy to forget when budgeting a vision model.
|
||||
|
||||
15.9 GiB of weights plus a KV pool plus vision profiling does not fit in the ~19 GiB free beside
|
||||
the other six GPU1 tenants. So the substitute was **reversibility and ordering**:
|
||||
|
||||
1. back the host `.env` up to a *named* file first (`.env.pfish6.bak-20260910`);
|
||||
2. swap `.env`, `up -d`, and prove the seat on its real port **while no gateway alias points at
|
||||
it**;
|
||||
3. move the gateway alias **last**.
|
||||
|
||||
That ordering is why the NaN-serving seat never reached a consumer — `char-rp-fast` did not exist
|
||||
yet and `Pfish-6` still resolved to nothing else. The cost was ~16 minutes of that one seat being
|
||||
down, twice, and nothing downstream saw a broken alias.
|
||||
|
||||
## The swap, as steps
|
||||
|
||||
```bash
|
||||
# on ana-ml2, /opt/docker/compose/erp-seat
|
||||
cp -n .env .env.pfish6.bak-20260910 # ROLLBACK LIVES HERE
|
||||
# point ERP_MODEL / ERP_SERVED_NAME / ERP_CHAT_TEMPLATE at the new tree
|
||||
sudo docker compose up -d # ~210 s to healthy
|
||||
|
||||
# verify BEFORE touching the gateway
|
||||
python3 seat_verify.py http://127.0.0.1:8021/v1 <served-name>
|
||||
|
||||
# gateway (canonical: stacks/litellm/conf/config.yaml)
|
||||
scripts/deploy-stack.sh ana-docker litellm --conf
|
||||
ssh ana-docker 'cd /opt/docker/compose/litellm && sudo docker compose restart litellm'
|
||||
```
|
||||
|
||||
**Rollback to Pfish-6** is `cp .env.pfish6.bak-20260910 .env && sudo docker compose up -d`,
|
||||
~4 minutes. `/tank/aimodels/erp-tune-v6-nvfp4a16` is untouched.
|
||||
|
||||
## What was checked, and what was not
|
||||
|
||||
Verified on the live seat (`raw/char-rp-fast-seat-verification-2026-09-10.txt`): served name and
|
||||
262,144 context; KV 534,649 tokens / 2.04x; clean prose with no `<|channel>thought` leak and no
|
||||
reasoning field; **a solid-colour image read correctly**, so vision is tested rather than inferred
|
||||
from a tensor count; an auto `tool_choice` call parsed with correct arguments; finite logprobs.
|
||||
Through the gateway with the shared `all-agents-local` key: `char-rp-fast` answers, `Pfish-6`
|
||||
returns an explicit `400 Invalid model name` rather than a substitution, and `char-rp` /
|
||||
`char-rp-reasoning` are both unaffected.
|
||||
|
||||
Audited before removing the alias: **0 of 17 LiteLLM keys** named `Pfish-6` in their model
|
||||
allowlist, so nothing was orphaned (1 of 17 is unrestricted and reaches whatever the gateway
|
||||
serves). ⚠ The first attempt at that audit passed `size=200` and got a silent `422`, which the
|
||||
script reported as "scanned 0 keys" — an empty result and a rejected query look identical if you
|
||||
do not check.
|
||||
|
||||
**Not established:** anything about quality. No RP eval, no long-context check, no A/B against
|
||||
Pfish-6 or `char-rp`. The samplers are the author's card values (Temp 0.8–1.0, MinP 0.05), not
|
||||
tuned here. n=1 smoke output is not evidence about writing.
|
||||
|
||||
⚠ Pre-existing doc rot noticed and **not** fixed: the `char-rp` comment block in
|
||||
`stacks/litellm/conf/config.yaml` still describes its `:8016` seat as MeroMero-v2. That has been
|
||||
stale since the 2026-08-24 swap to stock Gemma-4.
|
||||
@@ -0,0 +1,39 @@
|
||||
### char-rp-fast seat verification — ana-ml2 :8021, 2026-09-10
|
||||
### model: G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 (MoE-recipe re-quant)
|
||||
|
||||
$ docker logs vllm-erp-seat | grep 'GPU KV cache size'
|
||||
(EngineCore pid=663) INFO 09-10 18:30:39 [kv_cache_utils.py:1869] GPU KV cache size: 534,649 tokens, Maximum concurrency for 262,144 tokens per request: 2.04x
|
||||
|
||||
$ python3 seat_verify.py http://127.0.0.1:8021/v1 <served-name>
|
||||
== 1. served name + context
|
||||
served: ['G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16']
|
||||
max_model_len: {'G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16': 262144}
|
||||
OK 'G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16' is served
|
||||
|
||||
== 3. prose, non-thinking (the <|channel>thought leak)
|
||||
content (277 chars): 'Oil-slicked puddles mirror the fractured glow of a flickering neon sign, casting distorted crimson light across the uneven cobblestones. The sharp, metallic tang of wet iron clings to the air as water cascades rhythmical'
|
||||
reasoning_content: None
|
||||
OK clean prose in content, no reasoning, no channel prefix
|
||||
|
||||
== 4. vision (towers preserved, tested not inferred)
|
||||
answer: 'Blue' (image was solid RGB(30,60,200) = blue)
|
||||
OK image was decoded and read correctly
|
||||
|
||||
== 5. tool call (auto)
|
||||
tool_calls: [{"id": "chatcmpl-tool-ba6a1874968381f1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Anaheim\"}"}}]
|
||||
content: ''
|
||||
OK parsed a get_weather call, arguments='{"city": "Anaheim"}'
|
||||
|
||||
== 6. logprobs (NaN logits, the router-quant tell)
|
||||
text: '</b></b></b></b></b></b></b><b>'
|
||||
token_logprobs: [-1.3935617208480835, -0.1289057433605194, -0.006735478527843952, -0.006430173758417368, -0.005962086841464043]
|
||||
OK finite logprobs, non-empty raw text
|
||||
|
||||
============================================================
|
||||
ALL CHECKS PASSED
|
||||
|
||||
### ignore-list diff vs Pfish-6 (the known-good MoE quant of the SAME architecture class)
|
||||
Pfish-6 (known good) : 252 ignore entries
|
||||
A4B re-quant (live) : 252 ignore entries identical to Pfish-6: True
|
||||
A4B FIRST quant (bad) : 222 ignore entries missing vs good: 30
|
||||
the missing ones : ['model.language_model.layers.0.router.proj', 'model.language_model.layers.1.router.proj', 'model.language_model.layers.10.router.proj'] ... (all 30 are layers.N.router.proj)
|
||||
@@ -0,0 +1,210 @@
|
||||
"""Verify the swapped char-rp-fast seat before the gateway alias points at it.
|
||||
|
||||
The order matters: the seat is proven on its direct port FIRST, and only then does
|
||||
`char-rp-fast` start resolving. That way no consumer ever sees a half-working alias
|
||||
-- which is the reason playbook §4.4 wants a temp port. A temp port was not
|
||||
reachable here (18.26 GiB free against 15.9 GiB of weights plus a 8.5 GiB KV pool),
|
||||
so the substitute is: prove it on :8021 while nothing routes to it, and keep the
|
||||
one-flip rollback to Pfish-6 intact until it passes.
|
||||
|
||||
Five checks, and each one exists because this seat family has broken in that exact
|
||||
way before:
|
||||
|
||||
1. served name + context -- a stale served-name is a silent substitution
|
||||
2. KV pool -- Pfish-6's 9.114 GB pinning should transfer, because
|
||||
the architecture is identical field for field; if the
|
||||
token count moved, that assumption was wrong
|
||||
3. prose, non-thinking -- the `<|channel>thought` leak into content, which
|
||||
stacks/gemma4-charrp/README.md warns about and which
|
||||
was measured 3/3 on this recipe without the parser pin
|
||||
4. vision -- the "vision towers intact" claim, tested rather than
|
||||
inferred from a tensor count
|
||||
5. tool call (auto) -- the seat advertises gemma4 tool parsing
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import zlib
|
||||
|
||||
BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8021/v1"
|
||||
MODEL = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
KEY = sys.argv[3] if len(sys.argv) > 3 else None
|
||||
|
||||
fails = []
|
||||
|
||||
|
||||
def post(path, body, timeout=180):
|
||||
req = urllib.request.Request(
|
||||
BASE + path, data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json",
|
||||
**({"Authorization": f"Bearer {KEY}"} if KEY else {})})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def get(path, timeout=30):
|
||||
req = urllib.request.Request(
|
||||
BASE + path,
|
||||
headers={**({"Authorization": f"Bearer {KEY}"} if KEY else {})})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def png(rgb, w=64, h=64):
|
||||
"""Minimal solid-colour PNG, built here so the test needs no asset on disk."""
|
||||
raw = b"".join(b"\x00" + bytes(rgb) * w for _ in range(h))
|
||||
|
||||
def chunk(tag, data):
|
||||
c = tag + data
|
||||
return struct.pack(">I", len(data)) + c + struct.pack(">I", zlib.crc32(c))
|
||||
|
||||
return (b"\x89PNG\r\n\x1a\n"
|
||||
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
||||
+ chunk(b"IDAT", zlib.compress(raw))
|
||||
+ chunk(b"IEND", b""))
|
||||
|
||||
|
||||
# ---- 1. served name + context -------------------------------------------------
|
||||
print("== 1. served name + context")
|
||||
models = get("/models")
|
||||
ids = [m["id"] for m in models["data"]]
|
||||
mlen = {m["id"]: m.get("max_model_len") for m in models["data"]}
|
||||
print(f" served: {ids}")
|
||||
print(f" max_model_len: {mlen}")
|
||||
if MODEL:
|
||||
if MODEL in ids:
|
||||
print(f" OK '{MODEL}' is served")
|
||||
else:
|
||||
fails.append(f"'{MODEL}' not in served names {ids}")
|
||||
print(f" *** '{MODEL}' NOT SERVED")
|
||||
target = MODEL if MODEL in ids else ids[0]
|
||||
if "Pfish-6" in ids:
|
||||
fails.append("Pfish-6 is STILL served -- the swap did not take")
|
||||
print(" *** Pfish-6 still served")
|
||||
|
||||
# ---- 3. prose, non-thinking ---------------------------------------------------
|
||||
print("\n== 3. prose, non-thinking (the <|channel>thought leak)")
|
||||
r = post("/chat/completions", {
|
||||
"model": target,
|
||||
"messages": [{"role": "user", "content":
|
||||
"Describe a rain-slicked alley at night in two sentences."}],
|
||||
"max_tokens": 120,
|
||||
})
|
||||
msg = r["choices"][0]["message"]
|
||||
content = msg.get("content") or ""
|
||||
reasoning = msg.get("reasoning_content") or msg.get("reasoning")
|
||||
print(f" content ({len(content)} chars): {content[:220]!r}")
|
||||
print(f" reasoning_content: {reasoning!r}")
|
||||
if not content.strip():
|
||||
fails.append("prose: content is empty")
|
||||
print(" *** content EMPTY")
|
||||
elif "<|channel" in content or "channel>thought" in content:
|
||||
fails.append("prose: <|channel>thought prefix leaked into content")
|
||||
print(" *** CHANNEL PREFIX LEAKED into content")
|
||||
elif reasoning:
|
||||
fails.append(f"prose: reasoning_content populated with enable_thinking=false ({len(reasoning)} chars)")
|
||||
print(" *** reasoning_content populated despite enable_thinking=false")
|
||||
else:
|
||||
print(" OK clean prose in content, no reasoning, no channel prefix")
|
||||
|
||||
# ---- 4. vision ----------------------------------------------------------------
|
||||
print("\n== 4. vision (towers preserved, tested not inferred)")
|
||||
blue = base64.b64encode(png((30, 60, 200))).decode()
|
||||
try:
|
||||
r = post("/chat/completions", {
|
||||
"model": target,
|
||||
"messages": [{"role": "user", "content": [
|
||||
{"type": "text", "text":
|
||||
"This image is one flat colour. Name that colour in one word."},
|
||||
{"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{blue}"}},
|
||||
]}],
|
||||
"max_tokens": 24,
|
||||
"temperature": 0,
|
||||
})
|
||||
v = (r["choices"][0]["message"].get("content") or "").strip()
|
||||
print(f" answer: {v!r} (image was solid RGB(30,60,200) = blue)")
|
||||
if "blue" in v.lower():
|
||||
print(" OK image was decoded and read correctly")
|
||||
elif v:
|
||||
fails.append(f"vision: answered {v!r} for a solid blue image")
|
||||
print(" *** answered, but not blue -- vision path suspect")
|
||||
else:
|
||||
fails.append("vision: empty answer")
|
||||
print(" *** empty answer")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:300]
|
||||
fails.append(f"vision: HTTP {e.code} {body}")
|
||||
print(f" *** HTTP {e.code}: {body}")
|
||||
|
||||
# ---- 5. tool call -------------------------------------------------------------
|
||||
print("\n== 5. tool call (auto)")
|
||||
try:
|
||||
r = post("/chat/completions", {
|
||||
"model": target,
|
||||
"messages": [{"role": "user", "content": "What is the weather in Anaheim?"}],
|
||||
"tools": [{"type": "function", "function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city.",
|
||||
"parameters": {"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"]}}}],
|
||||
"tool_choice": "auto",
|
||||
"max_tokens": 120,
|
||||
})
|
||||
m = r["choices"][0]["message"]
|
||||
tc = m.get("tool_calls")
|
||||
print(f" tool_calls: {json.dumps(tc)[:240] if tc else None}")
|
||||
print(f" content: {(m.get('content') or '')[:120]!r}")
|
||||
if tc and tc[0]["function"]["name"] == "get_weather":
|
||||
args = tc[0]["function"].get("arguments")
|
||||
print(f" OK parsed a get_weather call, arguments={args!r}")
|
||||
else:
|
||||
fails.append("tool call: no parsed get_weather tool_call")
|
||||
print(" *** no parsed tool call (auto tool_choice)")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:300]
|
||||
fails.append(f"tool call: HTTP {e.code} {body}")
|
||||
print(f" *** HTTP {e.code}: {body}")
|
||||
|
||||
# ---- 6. NaN logits -----------------------------------------------------------
|
||||
print("\n== 6. logprobs (NaN logits, the router-quant tell)")
|
||||
try:
|
||||
r = post("/completions", {
|
||||
"model": target, "prompt": "Rain on asphalt at midnight.",
|
||||
"max_tokens": 8, "temperature": 0, "logprobs": 1,
|
||||
})
|
||||
txt = r["choices"][0].get("text")
|
||||
lp = r["choices"][0].get("logprobs") or {}
|
||||
vals = lp.get("token_logprobs") or []
|
||||
print(f" text: {txt!r}")
|
||||
print(f" token_logprobs: {vals[:5]}")
|
||||
if not (txt or "").strip():
|
||||
fails.append("logprobs: raw completion decoded to the empty string -- generating, but no text")
|
||||
print(" *** EMPTY raw completion: tokens generated that decode to nothing")
|
||||
elif any(v is None or v != v for v in vals):
|
||||
fails.append("logprobs: NaN/None in token_logprobs")
|
||||
print(" *** NaN in token_logprobs")
|
||||
else:
|
||||
print(" OK finite logprobs, non-empty raw text")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:300]
|
||||
# vLLM cannot serialize NaN, so the 400 IS the positive finding here.
|
||||
if "nan" in body.lower():
|
||||
fails.append("logprobs: NaN logits -- vLLM refused to serialize them. "
|
||||
"On a MoE this is the router-quantized signature (playbook §3.15)")
|
||||
print(f" *** NaN LOGITS: {body}")
|
||||
else:
|
||||
fails.append(f"logprobs: HTTP {e.code} {body}")
|
||||
print(f" *** HTTP {e.code}: {body}")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
if fails:
|
||||
print(f"FAILED ({len(fails)}):")
|
||||
for f in fails:
|
||||
print(f" - {f}")
|
||||
sys.exit(1)
|
||||
print("ALL CHECKS PASSED")
|
||||
Reference in New Issue
Block a user