From 9a916a759fe6f9333acd1f5229a22d784b357f06 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 10 Sep 2026 11:34:05 -0700 Subject: [PATCH] Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here. --- docs/pfi/model-quantization-playbook.md | 56 +++++ ...quants-and-the-pinned-transformers-trap.md | 41 ++++ persistent-memory.md | 42 +++- .../RUNBOOK-char-rp-fast-swap.md | 113 ++++++++++ ...r-rp-fast-seat-verification-2026-09-10.txt | 39 ++++ services/erp-seat-quant/seat_verify.py | 210 ++++++++++++++++++ stacks/erp-seat/.env.example | 15 +- stacks/erp-seat/compose.yaml | 57 +++-- stacks/litellm/conf/config.yaml | 56 +++-- 9 files changed, 588 insertions(+), 41 deletions(-) create mode 100644 services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md create mode 100644 services/erp-seat-quant/raw/char-rp-fast-seat-verification-2026-09-10.txt create mode 100644 services/erp-seat-quant/seat_verify.py diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 145a2b3..24b927d 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -291,6 +291,43 @@ hit it exactly, the recipe is wrong and the failure is silent. ⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that error does not average out downstream, it changes which weights run at all. +**And here is what that actually looks like when it ships — measured 2026-09-10 on the MeroMero +26B-A4B, which reached a live seat before anyone noticed.** The A4B was quantized with the *dense* +recipe (`services/meromero-quant/quant_a16_datafree.py`), whose IGNORE list has no `re:.*router.*` +entry. All 30 routers went to NVFP4. Then: + +- the quant **completed cleanly**, 16 G, no warning; +- the tensor table looked **plausible** — 11,755 quantized modules against 11,725 in the + known-good build, a 0.26% difference nobody eyeballs; +- vLLM **started, passed its healthcheck, and reported the correct KV pool**; +- every request returned `finish_reason: "length"` with the **full completion_tokens count** — + 120 of 120, 600 of 600 — and `content: null`. The model was generating, and every token decoded + to the empty string; +- the give-away was **`logprobs` coming back NaN**, which surfaced only because a diagnostic asked + for them and vLLM refused to serialize the response: `Out of range float values are not JSON + compliant: nan`. + +⚠⚠ **So the router mistake has no symptom you would catch by watching a seat come up.** It has +exactly one cheap tell, and §4.4 now carries it: ask for `logprobs` once. + +**Two guards, both cheap, both would have caught this before the seat:** + +1. **Use the architecture-class-correct recipe and let its guard fire.** + `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` refuses outright — *"⚠ REFUSING: a + router/vision/audio Linear is in the quantize set"* — and asserts `layers × experts × 3` before + any GPU time. Its `--dry-run` does the whole check with no GPU and no save. The dense recipe has + neither guard and will happily eat a MoE. +2. **Diff `quantization_config.ignore` against a known-good quant of the SAME ARCHITECTURE CLASS.** + The broken build had **222** ignore entries against the good build's **252**; the 30 missing + were exactly `model.language_model.layers.N.router.proj`. That diff is a two-line script and it + names the defect precisely. + +⚠ **The control has to match the architecture class, not just the family.** The broken A4B *was* +structurally diffed before it shipped — against a verified-good **dense** 31B quant of the same +Gemma-4 family. A dense model has no routers, so the one thing that was wrong was the one thing +that control could not see, and the comparison came back clean. A positive control is only worth +what it can distinguish; "same family" is not "same architecture class". + ### 3.4 Toolchain version deadlocks Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors, @@ -629,6 +666,25 @@ a `--check` mode; point it at a tree you already trust before you trust its verd Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line. +**Ask for `logprobs` once, on the temp port, before the alias moves.** A seat can pass its +healthcheck, report the right KV pool, and answer every request with the full token count while +every token decodes to the empty string — that is what a router-quantized MoE does (§3.15). NaN +logits are invisible to `/health`, invisible to the token counts, and invisible to a tensor-table +diff against the wrong control; a single `logprobs: 1` request surfaces them immediately, because +vLLM cannot even serialize the response (`Out of range float values are not JSON compliant: nan`). +Add it to the smoke set: **served name, one prose completion, one image if the model is +multimodal, one tool call, and one `logprobs` request.** + +⚠ **A co-resident temp port is not always reachable, and the fallback is reversibility, not +skipping the test.** Measured 2026-09-10: with 19 GiB free on a shared card, a 16 G A4B refused +admission at `gpu-memory-utilization 0.20` (18.26 free vs 18.99 requested), and at 0.185 it got +past admission and past the KV reservation only to OOM in **multimodal encoder-cache profiling** +(`profiled with 3 video items of the maximum feature size`) — a cost easy to forget when budgeting +a vision model. When the card genuinely cannot hold both, the substitute is: back up the host +`.env` to a named file first, prove the new seat on its real port **while no gateway alias points +at it**, and move the alias last. That ordering is what kept a NaN-serving seat away from every +consumer; the seat itself was down ~16 minutes and nothing downstream saw a broken alias. + --- ## 5. The acceptance gate — and how measurement lies to you diff --git a/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md b/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md index 6157e3f..1cdcc95 100644 --- a/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md +++ b/persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md @@ -119,4 +119,45 @@ GPU1 beside the current tenants regardless. Instruments and the full write-up: `services/meromero-quant/`. General lessons: `docs/pfi/model-quantization-playbook.md` §3.16, **§3.17 (new)**, §4.3. + +## The A4B reached a live seat while broken — and looked healthy doing it + +Operator, later the same day: *"replace that a4b moe over pfish-6 — remove the pfish-6 alias and +create an alias for char-rp-fast."* The A4B went onto the `erp-seat` seat (ana-ml2 `:8021`) and +**served NaN**. + +Cause: the morning's batch used the **dense** recipe for a **MoE** model. Its IGNORE list has no +`re:.*router.*`, so all 30 MoE routers were quantized to NVFP4, and a 4-bit router does not degrade +expert selection — it changes which experts run. + +**Nothing in the normal startup path showed it.** Quant `rc=0`. Healthcheck green in 210 s. Engine +log reported the correct KV pool. `/v1/models` correct. Every completion came back +`finish_reason=length` with the **full** token count and `content: null` — the model was generating +a full budget of tokens that decoded to the empty string. Raw `/v1/completions` was empty too, which +ruled out the chat template and the reasoning parser. The one signal that named it: `logprobs: 1` +→ HTTP 400 `Out of range float values are not JSON compliant: nan`. + +⚠⚠ **The durable lesson is about the CONTROL, not the router.** That broken tree HAD been +structurally diffed before it shipped — and passed — against a verified-good **dense** 31B quant of +the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the +single thing that control could not distinguish. **A positive control is only worth what it can +distinguish; "same family" is not "same architecture class."** Diffing instead against **Pfish-6** +— a known-good NVFP4A16 quant of the same 26B-A4B MoE — gave the answer in one line: 222 ignore +entries against 252, the 30 missing being exactly `model.language_model.layers.N.router.proj`. + +Re-quantized with `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts +11,520 expert Linears and refuses a router in the quantize set, both before any GPU time. 90 s. +Live seat then passed prose / vision / tool-call / logprobs. Broken tree parked as +`...-NVFP4A16.BROKEN-routers-quantized-20260910`. + +**§4.4's temp port was not reachable, and the fallback mattered.** 15.9 GiB of weights + KV + +multimodal encoder-cache profiling does not fit in ~19 GiB free beside GPU1's six other tenants: +`gpu-memory-utilization 0.20` refused admission (18.26 free vs 18.99 wanted) and `0.185` OOM'd in +encoder profiling. Substitute: named `.env` backup, prove the seat on its real port **while no +gateway alias points at it**, move the alias last. That ordering is the only reason a NaN-serving +seat never reached a consumer. Cost: ~16 min of seat downtime, twice; zero broken aliases. + +Runbook: `services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md`. Playbook §3.15 (failure signature + +the control-class rule), §4.4 (ask for logprobs once). + Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]] diff --git a/persistent-memory.md b/persistent-memory.md index 2398f3a..8a11870 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-10 11:00 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; **MeroMero BOTH quants landed** — v2 dense on attempt 5, serve test still owed; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_ +_Last updated: 2026-09-10 11:35 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; **MeroMero BOTH quants landed and the A4B is LIVE on :8021 as `char-rp-fast`, `Pfish-6` alias removed** — its first quant served NaN and looked healthy; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -141,9 +141,20 @@ _As of 2026-09-10 10:25 PT._ ### MeroMero seats -- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, W4A16, §4.3 post-steps DONE. - ⚠ It had the §3.14 **truncation cap baked into `tokenizer.json`** (`max_length: 8192`, because it - was quantized *with* the corpus). Fixed; backup `tokenizer.json.bak-pre-truncfix`. +- ✅ **LIVE on ana-ml2 `:8021` as gateway alias `char-rp-fast`** (operator, 2026-09-10: *"replace + that a4b moe over pfish-6 — remove the pfish-6 alias and create an alias for char-rp-fast"*). + `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`, 16 G, served under its own true name on the + **`erp-seat` stack** (name kept: asset-engine derives liveness from the compose project name). + Verified end to end — prose, **vision (reads a solid-colour image)**, auto tool call, finite + logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. + ⚠⚠ **ITS FIRST QUANT SERVED NaN AND PASSED ITS HEALTHCHECK DOING IT.** Built with the DENSE + recipe (no `re:.*router.*` in IGNORE) → all 30 MoE routers quantized to 4 bits → expert selection + destroyed. Every request returned `finish_reason=length` with the FULL token count and + `content: null`; the only tell was **NaN logprobs**. Re-quantized with + `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (its guard refuses exactly that). Broken + tree parked at `...-NVFP4A16.BROKEN-routers-quantized-20260910` — **do not serve it**. + ⚠ Also had the §3.14 truncation cap baked in (`max_length: 8192`, quantized *with* the corpus); + the MoE recipe's own post-step resets it. - ✅ `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G, landed on attempt 5.** Tensor table identical family-for-family to the 2026-08-21 canonical quant; **356 BF16 vision tensors preserved**; `input_activations=None` (genuinely A16). CPU load+generate coherent, 0 tensors on meta. @@ -152,11 +163,27 @@ _As of 2026-09-10 10:25 PT._ the redundant `per_layer_config`, not to force global access. → `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md`, `services/meromero-quant/`, playbook **§3.17 (new)** -- ⛔ **NEITHER quant has had its §4.4 serve test** — GPU1 has 19.9 GB free against 19.5 GB of v2 - weights, so it needs a live seat displaced. Operator's call; *"vllm servable"* unverified until then. +- ⛔ **The v2 DENSE has still not had a serve test** — GPU1 has ~19 GB free against 19.5 GB of its + weights, so it needs a live seat displaced. Operator's call; *"vllm servable"* unverified for the + dense tree. (The A4B's serve test is DONE and green, above.) +- ⚠ **A co-resident temp port could not be made to fit even for the 16 G A4B**, so §4.4's "temp port, + never the live seat" was substituted with reversibility: named `.env` backup, prove the seat on its + real port while no alias routes to it, move the alias last. Measured refusals: `gpu-memory-util 0.20` + → admission refused (18.26 GiB free vs 18.99 requested); `0.185` → past admission and past the KV + reservation, then OOM in **multimodal encoder-cache profiling** (3 video items at max feature size), + which is easy to forget when budgeting a vision model. - ⚠ **The A4B is 30 layers / kv 8 — Pfish-6's geometry**, so it fits 262k in the existing KV budget. The dense 31B is 60/16, ~4x KV per token, and does NOT fit 262k on GPU1 beside the other seats. -- Pfish-6 remains the standing seat on ana-ml2 `:8021`. Nothing repointed, no `rp-fast` alias exists. +- ⛔ **`Pfish-6` IS RETIRED from the gateway** (2026-09-10). Its seat now serves the A4B; a caller + asking for `Pfish-6` gets an explicit 400 "Invalid model name", not a substitution. Audited first: + **0 of 17 LiteLLM keys** scoped it, so nothing was orphaned. The artifact stays on disk at + `/tank/aimodels/erp-tune-v6-nvfp4a16` and is the rollback target — `cp + .env.pfish6.bak-20260910 .env && docker compose up -d` in `/opt/docker/compose/erp-seat` restores + it in ~4 min. +- The `char-rp` family is now **`char-rp` (stock Gemma-4 26B-A4B, :8016) / `char-rp-fast` (MeroMero + A4B, :8021) / `char-rp-reasoning` (Dark-Scarlett-27B, :8019)**. All three verified working after + the change. ⚠ The `char-rp` comment block in `stacks/litellm/conf/config.yaml` still describes its + seat as MeroMero-v2; that has been stale since the 2026-08-24 swap to stock Gemma-4. Not fixed. ### Fleet @@ -181,6 +208,7 @@ _As of 2026-09-10 10:25 PT._ - `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md` - `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`. +- `[2026-09-10]` **MeroMero A4B swapped onto the `erp-seat` seat as `char-rp-fast`; `Pfish-6` alias removed.** The A4B's FIRST quant used the dense recipe and 4-bit-quantized all 30 MoE routers — it passed its healthcheck and answered every request with the full token count decoding to the empty string, NaN logits the only tell. Re-quantized with the MoE recipe; live and verified (prose, vision, tool call, finite logprobs). Durable lesson: **a positive control must match the ARCHITECTURE CLASS** — the broken A4B was diffed against a good *dense* quant, which has no routers, so the clean result was meaningless. → playbook §3.15, §4.4 - `[2026-09-10]` **MeroMero: BOTH quants landed in-house at W4A16 — A4B first try, v2 dense on attempt 5.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* The durable lesson is **§3.17**: `pip install llmcompressor` silently pins transformers down a version, so attempt 4's error was a moved toolchain, not the malformed upload it looked like — a known-good positive control is what told them apart. Serve test still owed. → `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md` - `[2026-09-10]` **althing 3.6.2 deployed — post office + both heralds — and the fleet has TWO herald nodes, not seven.** Ask the post office's `nodes` table, not the box inventory. Cost a self-inflicted ~12 min bus outage. → `persistent-memory.d/2026-09-10-althing-362-rollout.md` - `[2026-09-10]` **A grep over a log that records your greps counts itself.** I reported forseti's drop defect as reproducing here with 3 drops in 21 s; the session had **zero**. Searching transcripts writes the search term into them. Filter by `"type":"system"` provenance, never content. Generalises to any instrument that can see itself. Auto-memory `feedback_grep_over_a_log_that_records_your_greps`. diff --git a/services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md b/services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md new file mode 100644 index 0000000..c8be5f0 --- /dev/null +++ b/services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md @@ -0,0 +1,113 @@ +# char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (2026-09-10) + +Operator: *"replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an +alias for char-rp-fast."* + +Result: `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` is live on ana-ml2 `:8021` +behind gateway alias `char-rp-fast`. `Pfish-6` is gone from the gateway. It took two +attempts, because the first quant was broken in a way that looks exactly like a healthy seat. + +## ⚠ The failure worth remembering: a 4-bit MoE router serves NaN and passes its healthcheck + +The A4B built that morning used `services/meromero-quant/quant_a16_datafree.py` — the **dense** +v2-31B recipe. Its IGNORE list has no `re:.*router.*` entry, so all 30 MoE routers were quantized +to NVFP4. A 4-bit router does not degrade expert selection, it *changes which experts run* +(playbook §3.15). + +What that looked like on the seat, in order of how convincing each signal was: + +| signal | what it said | +|---|---| +| quant exit code | `rc=0`, 16 G, no warning | +| `docker` healthcheck | healthy in 210 s | +| engine log | KV pool 534,649 tokens, 2.04x — exactly right | +| `/v1/models` | correct served name, 262,144 context | +| every completion | `finish_reason: "length"`, **full** `completion_tokens` (120/120, 600/600) | +| `content` | `null`. Every time. | +| raw `/v1/completions` | `text: ''` — so it was not the chat template or the reasoning parser | +| **`logprobs: 1`** | **HTTP 400 `Out of range float values are not JSON compliant: nan`** | + +The model was generating a full budget of tokens that decoded to the empty string, and the only +thing that named the fault was asking for logprobs. `seat_verify.py` now carries that as check 6. + +**What actually found it** was not the CPU forward (started, then abandoned as too slow): it was +diffing `quantization_config.ignore` against **Pfish-6** — a known-good NVFP4A16 quant of the +*same architecture class*. 222 entries against 252, and the 30 missing were precisely +`model.language_model.layers.N.router.proj`. + +⚠⚠ **The broken tree HAD been structurally diffed before it shipped — against a verified-good +DENSE 31B quant of the same Gemma-4 family, which came back clean.** A dense model has no +routers, so the one thing that was wrong was the one thing that control could not see. **A +positive control is only worth what it can distinguish; "same family" is not "same architecture +class."** + +Fix: re-quantize with `quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts +`layers × experts × 3 = 11,520` expert Linears and refuses if a router lands in the quantize set, +both before any GPU time. 90 seconds end to end. The broken tree is parked on ana-ml2 as +`...-NVFP4A16.BROKEN-routers-quantized-20260910`. **Do not serve it.** + +## Why the seat went dark for ~16 minutes instead of not at all + +Playbook §4.4 wants a temp port. It was not reachable, twice, and the numbers are worth keeping: + +- `--gpu-memory-utilization 0.20` → **admission refused**: `Free memory on device cuda:0 + (18.26/94.97 GiB) on startup is less than desired GPU memory utilization (0.2, 18.99 GiB)`. +- `0.185` + `--kv-cache-memory 1.5 GB` + `--max-model-len 8192` + `--enforce-eager` → past + admission, past the KV reservation, then `torch.OutOfMemoryError` during **multimodal + encoder-cache profiling** (`profiled with 3 video items of the maximum feature size`). That + profiling cost is easy to forget when budgeting a vision model. + +15.9 GiB of weights plus a KV pool plus vision profiling does not fit in the ~19 GiB free beside +the other six GPU1 tenants. So the substitute was **reversibility and ordering**: + +1. back the host `.env` up to a *named* file first (`.env.pfish6.bak-20260910`); +2. swap `.env`, `up -d`, and prove the seat on its real port **while no gateway alias points at + it**; +3. move the gateway alias **last**. + +That ordering is why the NaN-serving seat never reached a consumer — `char-rp-fast` did not exist +yet and `Pfish-6` still resolved to nothing else. The cost was ~16 minutes of that one seat being +down, twice, and nothing downstream saw a broken alias. + +## The swap, as steps + +```bash +# on ana-ml2, /opt/docker/compose/erp-seat +cp -n .env .env.pfish6.bak-20260910 # ROLLBACK LIVES HERE +# point ERP_MODEL / ERP_SERVED_NAME / ERP_CHAT_TEMPLATE at the new tree +sudo docker compose up -d # ~210 s to healthy + +# verify BEFORE touching the gateway +python3 seat_verify.py http://127.0.0.1:8021/v1 + +# gateway (canonical: stacks/litellm/conf/config.yaml) +scripts/deploy-stack.sh ana-docker litellm --conf +ssh ana-docker 'cd /opt/docker/compose/litellm && sudo docker compose restart litellm' +``` + +**Rollback to Pfish-6** is `cp .env.pfish6.bak-20260910 .env && sudo docker compose up -d`, +~4 minutes. `/tank/aimodels/erp-tune-v6-nvfp4a16` is untouched. + +## What was checked, and what was not + +Verified on the live seat (`raw/char-rp-fast-seat-verification-2026-09-10.txt`): served name and +262,144 context; KV 534,649 tokens / 2.04x; clean prose with no `<|channel>thought` leak and no +reasoning field; **a solid-colour image read correctly**, so vision is tested rather than inferred +from a tensor count; an auto `tool_choice` call parsed with correct arguments; finite logprobs. +Through the gateway with the shared `all-agents-local` key: `char-rp-fast` answers, `Pfish-6` +returns an explicit `400 Invalid model name` rather than a substitution, and `char-rp` / +`char-rp-reasoning` are both unaffected. + +Audited before removing the alias: **0 of 17 LiteLLM keys** named `Pfish-6` in their model +allowlist, so nothing was orphaned (1 of 17 is unrestricted and reaches whatever the gateway +serves). ⚠ The first attempt at that audit passed `size=200` and got a silent `422`, which the +script reported as "scanned 0 keys" — an empty result and a rejected query look identical if you +do not check. + +**Not established:** anything about quality. No RP eval, no long-context check, no A/B against +Pfish-6 or `char-rp`. The samplers are the author's card values (Temp 0.8–1.0, MinP 0.05), not +tuned here. n=1 smoke output is not evidence about writing. + +⚠ Pre-existing doc rot noticed and **not** fixed: the `char-rp` comment block in +`stacks/litellm/conf/config.yaml` still describes its `:8016` seat as MeroMero-v2. That has been +stale since the 2026-08-24 swap to stock Gemma-4. diff --git a/services/erp-seat-quant/raw/char-rp-fast-seat-verification-2026-09-10.txt b/services/erp-seat-quant/raw/char-rp-fast-seat-verification-2026-09-10.txt new file mode 100644 index 0000000..36c898a --- /dev/null +++ b/services/erp-seat-quant/raw/char-rp-fast-seat-verification-2026-09-10.txt @@ -0,0 +1,39 @@ +### char-rp-fast seat verification — ana-ml2 :8021, 2026-09-10 +### model: G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 (MoE-recipe re-quant) + +$ docker logs vllm-erp-seat | grep 'GPU KV cache size' +(EngineCore pid=663) INFO 09-10 18:30:39 [kv_cache_utils.py:1869] GPU KV cache size: 534,649 tokens, Maximum concurrency for 262,144 tokens per request: 2.04x + +$ python3 seat_verify.py http://127.0.0.1:8021/v1 +== 1. served name + context + served: ['G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16'] + max_model_len: {'G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16': 262144} + OK 'G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16' is served + +== 3. prose, non-thinking (the <|channel>thought leak) + content (277 chars): 'Oil-slicked puddles mirror the fractured glow of a flickering neon sign, casting distorted crimson light across the uneven cobblestones. The sharp, metallic tang of wet iron clings to the air as water cascades rhythmical' + reasoning_content: None + OK clean prose in content, no reasoning, no channel prefix + +== 4. vision (towers preserved, tested not inferred) + answer: 'Blue' (image was solid RGB(30,60,200) = blue) + OK image was decoded and read correctly + +== 5. tool call (auto) + tool_calls: [{"id": "chatcmpl-tool-ba6a1874968381f1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Anaheim\"}"}}] + content: '' + OK parsed a get_weather call, arguments='{"city": "Anaheim"}' + +== 6. logprobs (NaN logits, the router-quant tell) + text: '' + token_logprobs: [-1.3935617208480835, -0.1289057433605194, -0.006735478527843952, -0.006430173758417368, -0.005962086841464043] + OK finite logprobs, non-empty raw text + +============================================================ +ALL CHECKS PASSED + +### ignore-list diff vs Pfish-6 (the known-good MoE quant of the SAME architecture class) +Pfish-6 (known good) : 252 ignore entries +A4B re-quant (live) : 252 ignore entries identical to Pfish-6: True +A4B FIRST quant (bad) : 222 ignore entries missing vs good: 30 + the missing ones : ['model.language_model.layers.0.router.proj', 'model.language_model.layers.1.router.proj', 'model.language_model.layers.10.router.proj'] ... (all 30 are layers.N.router.proj) diff --git a/services/erp-seat-quant/seat_verify.py b/services/erp-seat-quant/seat_verify.py new file mode 100644 index 0000000..9072d94 --- /dev/null +++ b/services/erp-seat-quant/seat_verify.py @@ -0,0 +1,210 @@ +"""Verify the swapped char-rp-fast seat before the gateway alias points at it. + +The order matters: the seat is proven on its direct port FIRST, and only then does +`char-rp-fast` start resolving. That way no consumer ever sees a half-working alias +-- which is the reason playbook §4.4 wants a temp port. A temp port was not +reachable here (18.26 GiB free against 15.9 GiB of weights plus a 8.5 GiB KV pool), +so the substitute is: prove it on :8021 while nothing routes to it, and keep the +one-flip rollback to Pfish-6 intact until it passes. + +Five checks, and each one exists because this seat family has broken in that exact +way before: + + 1. served name + context -- a stale served-name is a silent substitution + 2. KV pool -- Pfish-6's 9.114 GB pinning should transfer, because + the architecture is identical field for field; if the + token count moved, that assumption was wrong + 3. prose, non-thinking -- the `<|channel>thought` leak into content, which + stacks/gemma4-charrp/README.md warns about and which + was measured 3/3 on this recipe without the parser pin + 4. vision -- the "vision towers intact" claim, tested rather than + inferred from a tensor count + 5. tool call (auto) -- the seat advertises gemma4 tool parsing +""" +import base64 +import json +import struct +import sys +import urllib.error +import urllib.request +import zlib + +BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8021/v1" +MODEL = sys.argv[2] if len(sys.argv) > 2 else None +KEY = sys.argv[3] if len(sys.argv) > 3 else None + +fails = [] + + +def post(path, body, timeout=180): + req = urllib.request.Request( + BASE + path, data=json.dumps(body).encode(), + headers={"Content-Type": "application/json", + **({"Authorization": f"Bearer {KEY}"} if KEY else {})}) + with urllib.request.urlopen(req, timeout=timeout) as r: + return json.load(r) + + +def get(path, timeout=30): + req = urllib.request.Request( + BASE + path, + headers={**({"Authorization": f"Bearer {KEY}"} if KEY else {})}) + with urllib.request.urlopen(req, timeout=timeout) as r: + return json.load(r) + + +def png(rgb, w=64, h=64): + """Minimal solid-colour PNG, built here so the test needs no asset on disk.""" + raw = b"".join(b"\x00" + bytes(rgb) * w for _ in range(h)) + + def chunk(tag, data): + c = tag + data + return struct.pack(">I", len(data)) + c + struct.pack(">I", zlib.crc32(c)) + + return (b"\x89PNG\r\n\x1a\n" + + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)) + + chunk(b"IDAT", zlib.compress(raw)) + + chunk(b"IEND", b"")) + + +# ---- 1. served name + context ------------------------------------------------- +print("== 1. served name + context") +models = get("/models") +ids = [m["id"] for m in models["data"]] +mlen = {m["id"]: m.get("max_model_len") for m in models["data"]} +print(f" served: {ids}") +print(f" max_model_len: {mlen}") +if MODEL: + if MODEL in ids: + print(f" OK '{MODEL}' is served") + else: + fails.append(f"'{MODEL}' not in served names {ids}") + print(f" *** '{MODEL}' NOT SERVED") +target = MODEL if MODEL in ids else ids[0] +if "Pfish-6" in ids: + fails.append("Pfish-6 is STILL served -- the swap did not take") + print(" *** Pfish-6 still served") + +# ---- 3. prose, non-thinking --------------------------------------------------- +print("\n== 3. prose, non-thinking (the <|channel>thought leak)") +r = post("/chat/completions", { + "model": target, + "messages": [{"role": "user", "content": + "Describe a rain-slicked alley at night in two sentences."}], + "max_tokens": 120, +}) +msg = r["choices"][0]["message"] +content = msg.get("content") or "" +reasoning = msg.get("reasoning_content") or msg.get("reasoning") +print(f" content ({len(content)} chars): {content[:220]!r}") +print(f" reasoning_content: {reasoning!r}") +if not content.strip(): + fails.append("prose: content is empty") + print(" *** content EMPTY") +elif "<|channel" in content or "channel>thought" in content: + fails.append("prose: <|channel>thought prefix leaked into content") + print(" *** CHANNEL PREFIX LEAKED into content") +elif reasoning: + fails.append(f"prose: reasoning_content populated with enable_thinking=false ({len(reasoning)} chars)") + print(" *** reasoning_content populated despite enable_thinking=false") +else: + print(" OK clean prose in content, no reasoning, no channel prefix") + +# ---- 4. vision ---------------------------------------------------------------- +print("\n== 4. vision (towers preserved, tested not inferred)") +blue = base64.b64encode(png((30, 60, 200))).decode() +try: + r = post("/chat/completions", { + "model": target, + "messages": [{"role": "user", "content": [ + {"type": "text", "text": + "This image is one flat colour. Name that colour in one word."}, + {"type": "image_url", + "image_url": {"url": f"data:image/png;base64,{blue}"}}, + ]}], + "max_tokens": 24, + "temperature": 0, + }) + v = (r["choices"][0]["message"].get("content") or "").strip() + print(f" answer: {v!r} (image was solid RGB(30,60,200) = blue)") + if "blue" in v.lower(): + print(" OK image was decoded and read correctly") + elif v: + fails.append(f"vision: answered {v!r} for a solid blue image") + print(" *** answered, but not blue -- vision path suspect") + else: + fails.append("vision: empty answer") + print(" *** empty answer") +except urllib.error.HTTPError as e: + body = e.read().decode()[:300] + fails.append(f"vision: HTTP {e.code} {body}") + print(f" *** HTTP {e.code}: {body}") + +# ---- 5. tool call ------------------------------------------------------------- +print("\n== 5. tool call (auto)") +try: + r = post("/chat/completions", { + "model": target, + "messages": [{"role": "user", "content": "What is the weather in Anaheim?"}], + "tools": [{"type": "function", "function": { + "name": "get_weather", + "description": "Get the current weather for a city.", + "parameters": {"type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"]}}}], + "tool_choice": "auto", + "max_tokens": 120, + }) + m = r["choices"][0]["message"] + tc = m.get("tool_calls") + print(f" tool_calls: {json.dumps(tc)[:240] if tc else None}") + print(f" content: {(m.get('content') or '')[:120]!r}") + if tc and tc[0]["function"]["name"] == "get_weather": + args = tc[0]["function"].get("arguments") + print(f" OK parsed a get_weather call, arguments={args!r}") + else: + fails.append("tool call: no parsed get_weather tool_call") + print(" *** no parsed tool call (auto tool_choice)") +except urllib.error.HTTPError as e: + body = e.read().decode()[:300] + fails.append(f"tool call: HTTP {e.code} {body}") + print(f" *** HTTP {e.code}: {body}") + +# ---- 6. NaN logits ----------------------------------------------------------- +print("\n== 6. logprobs (NaN logits, the router-quant tell)") +try: + r = post("/completions", { + "model": target, "prompt": "Rain on asphalt at midnight.", + "max_tokens": 8, "temperature": 0, "logprobs": 1, + }) + txt = r["choices"][0].get("text") + lp = r["choices"][0].get("logprobs") or {} + vals = lp.get("token_logprobs") or [] + print(f" text: {txt!r}") + print(f" token_logprobs: {vals[:5]}") + if not (txt or "").strip(): + fails.append("logprobs: raw completion decoded to the empty string -- generating, but no text") + print(" *** EMPTY raw completion: tokens generated that decode to nothing") + elif any(v is None or v != v for v in vals): + fails.append("logprobs: NaN/None in token_logprobs") + print(" *** NaN in token_logprobs") + else: + print(" OK finite logprobs, non-empty raw text") +except urllib.error.HTTPError as e: + body = e.read().decode()[:300] + # vLLM cannot serialize NaN, so the 400 IS the positive finding here. + if "nan" in body.lower(): + fails.append("logprobs: NaN logits -- vLLM refused to serialize them. " + "On a MoE this is the router-quantized signature (playbook §3.15)") + print(f" *** NaN LOGITS: {body}") + else: + fails.append(f"logprobs: HTTP {e.code} {body}") + print(f" *** HTTP {e.code}: {body}") + +print("\n" + "=" * 60) +if fails: + print(f"FAILED ({len(fails)}):") + for f in fails: + print(f" - {f}") + sys.exit(1) +print("ALL CHECKS PASSED") diff --git a/stacks/erp-seat/.env.example b/stacks/erp-seat/.env.example index 3f463aa..759ea03 100644 --- a/stacks/erp-seat/.env.example +++ b/stacks/erp-seat/.env.example @@ -1,12 +1,15 @@ # erp-seat — ana-ml2 GPU1. Real .env lives on the host at /opt/docker/compose/erp-seat/.env. ERP_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0 -ERP_MODEL=/tank/aimodels/erp-tune-v6-nvfp4a16 -ERP_SERVED_NAME=erp-tune-v6-nvfp4a16 -ERP_CHAT_TEMPLATE=/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja +ERP_MODEL=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 +ERP_SERVED_NAME=G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 +ERP_CHAT_TEMPLATE=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja ERP_PORT=8021 ERP_GPU_ID=1 # 0.35 x 97.9 GiB = 34 GiB. GPU1 had ~47 GiB free on 2026-09-08 (scriberr/embed/rerank/coder/reward resident). -ERP_GPU_MEM_UTIL=0.35 -ERP_MAX_MODEL_LEN=32768 -ERP_MAX_NUM_SEQS=8 +ERP_GPU_MEM_UTIL=0.30 +ERP_MAX_MODEL_LEN=262144 +ERP_MAX_NUM_SEQS=32 API_KEY= +# 8.49 GiB -> 534,649 KV tokens -> 2.04x a 262,144 context (operator's KV = 2x rule). +ERP_KV_CACHE_MEMORY=9114000000 +ERP_MOE_BACKEND=auto diff --git a/stacks/erp-seat/compose.yaml b/stacks/erp-seat/compose.yaml index 83c6088..57ce76b 100644 --- a/stacks/erp-seat/compose.yaml +++ b/stacks/erp-seat/compose.yaml @@ -1,17 +1,38 @@ -# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of **Pfish-6**, the run-6 LoRA -# merge on the jenerallee78 ARA abliteration, served under that name. +# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant +# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`. # -# ⚠ RUN 7 IS RETIRED (operator ruling 2026-09-09): "we're gonna stay on 6 for now". Run 7's -# gate failure turned out to be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed -# cc42d76 in brokkr-smithy) — but run 7 was independently a poor run (primary FLAT +2, both -# diversity families reduced, long-context coherence 1.0 -> 0.875). Run 6 is the standing seat. -# Routing aliases (e.g. LiteLLM `trial`) are the operator's call and live in the gateway, not here. +# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until +# 2026-09-10, when the operator swapped the occupant: "replace that a4b moe over pfish-6 -- +# remove the pfish-6 alias and create an alias for char-rp-fast." The compose PROJECT name is +# deliberately NOT renamed: asset-engine derives seat liveness from it, so a rename reads as +# OFFLINE. Pfish-6 remains on disk at /tank/aimodels/erp-tune-v6-nvfp4a16 and the pre-swap host +# env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- one cp plus `up -d` back. +# +# ⚠ THE FIRST A4B QUANT SERVED NaN AND LOOKED HEALTHY DOING IT. It was built with the DENSE +# recipe, whose ignore list carries no router regex, so all 30 MoE routers were quantized to +# 4 bits and expert selection was destroyed (playbook §3.15). The seat passed its healthcheck, +# returned finish_reason=length with the full token count, and every response decoded to the +# empty string; the give-away was NaN logprobs. Re-quantized with +# services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose target guard refuses exactly +# that. Use the MoE recipe for anything in this family; the dense one is for the v2-31B. +# +# PFISH-6 PROVENANCE, kept because it is still the rollback target. Run 7 was retired by +# operator ruling 2026-09-09 ("we're gonna stay on 6 for now"): its gate failure turned out to +# be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed cc42d76 in brokkr-smithy), but +# run 7 was independently a poor run (primary FLAT +2, both diversity families reduced, +# long-context coherence 1.0 -> 0.875). Run 6 was the standing seat here until the 2026-09-10 +# swap above. Routing aliases live in the gateway, not here. # # Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this # box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template. -# GPU1 is SHARED (charrp-MoE moved? no — scriberr, embed, rerank, coder, reward live there): -# ~47 GiB was free on 2026-09-08; 0.35 x 97.9 GiB = 34 GiB keeps ~13 GiB of real margin. -# Quant pipeline: services/erp-seat-quant/. Tunables in .env. +# It carries over to MeroMero A4B unchanged -- verified 2026-09-10 end to end: clean prose with +# no channel-prefix leak, a solid-colour image read correctly (vision towers intact), and an +# auto tool_choice call parsed. +# GPU1 is SHARED (scriberr, embed, rerank, coder, reward, charrp live there): the live .env runs +# ERP_GPU_MEM_UTIL=0.30, and the KV pool is pinned in bytes below regardless, so the ratio only +# has to clear admission. +# Quant pipeline: services/erp-seat-quant/ (MoE recipe -- NOT services/meromero-quant/, which is +# the dense one). Tunables in .env. name: erp-seat @@ -28,11 +49,11 @@ services: environment: - VLLM_API_KEY=${API_KEY:-} command: - - ${ERP_MODEL:-/tank/aimodels/erp-tune-v6-nvfp4a16} + - ${ERP_MODEL:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16} - --quantization - compressed-tensors - --served-model-name - - ${ERP_SERVED_NAME:-Pfish-6} + - ${ERP_SERVED_NAME:-G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16} - --tool-call-parser - gemma4 - --enable-auto-tool-choice @@ -52,7 +73,7 @@ services: # an empty turn. The flag drops the tools from the prompt so the model answers in prose. - --exclude-tools-when-tool-choice-none - --chat-template - - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja} + - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja} - --max-model-len - "${ERP_MAX_MODEL_LEN:-262144}" # KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is @@ -71,6 +92,12 @@ services: # # 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is # the operator's sizing rule (KV = 2x max context, 2026-09-09). + # + # This figure SURVIVED the 2026-09-10 Pfish-6 -> MeroMero-A4B swap unchanged, and that + # is not luck: the two are the same architecture field for field (30 layers, kv 8, + # head_dim 256, sliding_window 1024, 25 sliding / 5 full, 128 experts top-8), so the + # KV-per-token is the same number. Confirmed by reading 534,649 tokens / 2.04x back out + # of the new engine's log rather than assuming the pinning carried. - --kv-cache-memory - "${ERP_KV_CACHE_MEMORY:-9114000000}" - --max-num-seqs @@ -122,9 +149,9 @@ services: - tnet labels: - homepage.group=AI - Inference - - homepage.name=Pfish-6 (Gemma-4 26B-A4B ARA, NVFP4A16) + - homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE) - homepage.icon=mdi-fire - - homepage.description=Pfish-6 — the run-6 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1) + - homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1) - homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs networks: diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 93cdfc1..a1437d5 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -872,9 +872,10 @@ model_list: # base profile ~0% on 30/35 axes per brokkr) -- this seat is for the operator's ear; treat # it as unrated on every safety axis. # - # ROLLBACK: /tank/aimodels/erp-tune-v6-nvfp4a16 is still on disk, and the previous host env - # is at /tmp/erp-seat-env.v6.bak on ana-ml2 -- flip ERP_MODEL/ERP_SERVED_NAME/ERP_CHAT_TEMPLATE - # in /opt/docker/compose/erp-seat/.env back to v6 and `docker compose up -d`. + # ROLLBACK (updated 2026-09-10, the seat now serves MeroMero A4B): Pfish-6 itself is + # the rollback target. /tank/aimodels/erp-tune-v6-nvfp4a16 is still on disk and the + # pre-swap host env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- + # `cp .env.pfish6.bak-20260910 .env && docker compose up -d` restores Pfish-6 in ~4 min. # # THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here, # in every session and project. Removing this alias does not remove the operator's @@ -882,21 +883,50 @@ model_list: # # Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing). - # Pfish-6 — the STANDING seat as of 2026-09-09. NVFP4A16 quant of the run-6 - # LoRA merge on the jenerallee78 ARA abliteration, on ana-ml2 GPU1 (:8021). - # Operator ruling: "declare run 6 as Pfish-6 ... we're gonna stay on 6 for now." + # char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10: + # "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias + # for char-rp-fast." REPLACES the `Pfish-6` alias, which is removed with this change. # - # REPLACES the `trial` alias, which is retired with run 7. Run 7's CSAM gate - # failure turned out to be a DETECTOR BUG (the adjective "minor" matching a - # HARD rule — fixed cc42d76 in brokkr-smithy), but run 7 was independently a - # poor run and is not coming back. + # The seat itself is unchanged in every dimension that matters to a caller: the A4B is + # 30 layers / kv 8 / sliding_window 1024 / 128 experts top-8 — field for field the same + # geometry as Pfish-6 — so the 9.114 GB KV pinning transfers exactly and the seat still + # reports 534,649 tokens and 2.04x concurrency at 262,144. That was verified from the + # engine log, not assumed, because KV-per-token is normally NOT transferable. # - # Served under its TRUE name. This is now a named seat, not a trial. - - model_name: Pfish-6 + # ⚠ Pfish-6 IS GONE from this gateway and its seat no longer serves that name. A caller + # still asking for `Pfish-6` gets a clean 404 rather than a silent substitution, which is + # the intended behaviour. The artifact is still on disk (see the ROLLBACK note above). + # + # SAMPLERS, and they are the author's, not inherited: the model card states Temp 0.8-1.0 + # and MinP 0.05. Temperature/top_p/top_k already come from the tree's own + # generation_config (1.0 / 0.95 / 64) which vLLM applies server-side, and 1.0 sits at the + # top of the card's stated range, so the only value that needs stating here is min_p. + # Deliberately NOT copying char-rp's temp 1.1 / min_p 0.10 — those were A/B-tuned against + # a retired Mistral seat and are inherited, not canonical, as that block's own note says. + # + # enable_thinking:false is ALSO pinned process-level on the seat + # (--default-chat-template-kwargs). Stated here as well so the intent is visible at the + # routing layer: without it the gemma4 reasoning parser pre-initialises to REASONING and + # plain prose comes back with a null `content`. + # + # ⚠ ITS FIRST QUANT WAS BROKEN AND SERVED NaN. The 2026-09-10 08:15 build was made with + # the DENSE recipe, whose ignore list has no router regex, so all 30 MoE routers were + # quantized to 4 bits — a 4-bit router picks different experts (playbook §3.15). The seat + # came up healthy, answered every request with 120 tokens, and decoded to the empty string; + # logprobs were NaN. Re-quantized with services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, + # whose target guard refuses that exact mistake. The broken tree is parked on ana-ml2 as + # ...-NVFP4A16.BROKEN-routers-quantized-20260910. Do not serve it. + - model_name: char-rp-fast litellm_params: - model: hosted_vllm/Pfish-6 + model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 api_base: http://10.250.50.54:8021/v1 api_key: os.environ/VLLM_API_KEY + extra_body: + min_p: 0.05 + chat_template_kwargs: + enable_thinking: false + model_info: + mode: chat general_settings: master_key: os.environ/LITELLM_MASTER_KEY