Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
This commit is contained in:
@@ -291,6 +291,43 @@ hit it exactly, the recipe is wrong and the failure is silent.
|
||||
⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
|
||||
error does not average out downstream, it changes which weights run at all.
|
||||
|
||||
**And here is what that actually looks like when it ships — measured 2026-09-10 on the MeroMero
|
||||
26B-A4B, which reached a live seat before anyone noticed.** The A4B was quantized with the *dense*
|
||||
recipe (`services/meromero-quant/quant_a16_datafree.py`), whose IGNORE list has no `re:.*router.*`
|
||||
entry. All 30 routers went to NVFP4. Then:
|
||||
|
||||
- the quant **completed cleanly**, 16 G, no warning;
|
||||
- the tensor table looked **plausible** — 11,755 quantized modules against 11,725 in the
|
||||
known-good build, a 0.26% difference nobody eyeballs;
|
||||
- vLLM **started, passed its healthcheck, and reported the correct KV pool**;
|
||||
- every request returned `finish_reason: "length"` with the **full completion_tokens count** —
|
||||
120 of 120, 600 of 600 — and `content: null`. The model was generating, and every token decoded
|
||||
to the empty string;
|
||||
- the give-away was **`logprobs` coming back NaN**, which surfaced only because a diagnostic asked
|
||||
for them and vLLM refused to serialize the response: `Out of range float values are not JSON
|
||||
compliant: nan`.
|
||||
|
||||
⚠⚠ **So the router mistake has no symptom you would catch by watching a seat come up.** It has
|
||||
exactly one cheap tell, and §4.4 now carries it: ask for `logprobs` once.
|
||||
|
||||
**Two guards, both cheap, both would have caught this before the seat:**
|
||||
|
||||
1. **Use the architecture-class-correct recipe and let its guard fire.**
|
||||
`services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` refuses outright — *"⚠ REFUSING: a
|
||||
router/vision/audio Linear is in the quantize set"* — and asserts `layers × experts × 3` before
|
||||
any GPU time. Its `--dry-run` does the whole check with no GPU and no save. The dense recipe has
|
||||
neither guard and will happily eat a MoE.
|
||||
2. **Diff `quantization_config.ignore` against a known-good quant of the SAME ARCHITECTURE CLASS.**
|
||||
The broken build had **222** ignore entries against the good build's **252**; the 30 missing
|
||||
were exactly `model.language_model.layers.N.router.proj`. That diff is a two-line script and it
|
||||
names the defect precisely.
|
||||
|
||||
⚠ **The control has to match the architecture class, not just the family.** The broken A4B *was*
|
||||
structurally diffed before it shipped — against a verified-good **dense** 31B quant of the same
|
||||
Gemma-4 family. A dense model has no routers, so the one thing that was wrong was the one thing
|
||||
that control could not see, and the comparison came back clean. A positive control is only worth
|
||||
what it can distinguish; "same family" is not "same architecture class".
|
||||
|
||||
### 3.4 Toolchain version deadlocks
|
||||
|
||||
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
|
||||
@@ -629,6 +666,25 @@ a `--check` mode; point it at a tree you already trust before you trust its verd
|
||||
Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
|
||||
only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.
|
||||
|
||||
**Ask for `logprobs` once, on the temp port, before the alias moves.** A seat can pass its
|
||||
healthcheck, report the right KV pool, and answer every request with the full token count while
|
||||
every token decodes to the empty string — that is what a router-quantized MoE does (§3.15). NaN
|
||||
logits are invisible to `/health`, invisible to the token counts, and invisible to a tensor-table
|
||||
diff against the wrong control; a single `logprobs: 1` request surfaces them immediately, because
|
||||
vLLM cannot even serialize the response (`Out of range float values are not JSON compliant: nan`).
|
||||
Add it to the smoke set: **served name, one prose completion, one image if the model is
|
||||
multimodal, one tool call, and one `logprobs` request.**
|
||||
|
||||
⚠ **A co-resident temp port is not always reachable, and the fallback is reversibility, not
|
||||
skipping the test.** Measured 2026-09-10: with 19 GiB free on a shared card, a 16 G A4B refused
|
||||
admission at `gpu-memory-utilization 0.20` (18.26 free vs 18.99 requested), and at 0.185 it got
|
||||
past admission and past the KV reservation only to OOM in **multimodal encoder-cache profiling**
|
||||
(`profiled with 3 video items of the maximum feature size`) — a cost easy to forget when budgeting
|
||||
a vision model. When the card genuinely cannot hold both, the substitute is: back up the host
|
||||
`.env` to a named file first, prove the new seat on its real port **while no gateway alias points
|
||||
at it**, and move the alias last. That ordering is what kept a NaN-serving seat away from every
|
||||
consumer; the seat itself was down ~16 minutes and nothing downstream saw a broken alias.
|
||||
|
||||
---
|
||||
|
||||
## 5. The acceptance gate — and how measurement lies to you
|
||||
|
||||
@@ -119,4 +119,45 @@ GPU1 beside the current tenants regardless.
|
||||
Instruments and the full write-up: `services/meromero-quant/`. General lessons:
|
||||
`docs/pfi/model-quantization-playbook.md` §3.16, **§3.17 (new)**, §4.3.
|
||||
|
||||
|
||||
## The A4B reached a live seat while broken — and looked healthy doing it
|
||||
|
||||
Operator, later the same day: *"replace that a4b moe over pfish-6 — remove the pfish-6 alias and
|
||||
create an alias for char-rp-fast."* The A4B went onto the `erp-seat` seat (ana-ml2 `:8021`) and
|
||||
**served NaN**.
|
||||
|
||||
Cause: the morning's batch used the **dense** recipe for a **MoE** model. Its IGNORE list has no
|
||||
`re:.*router.*`, so all 30 MoE routers were quantized to NVFP4, and a 4-bit router does not degrade
|
||||
expert selection — it changes which experts run.
|
||||
|
||||
**Nothing in the normal startup path showed it.** Quant `rc=0`. Healthcheck green in 210 s. Engine
|
||||
log reported the correct KV pool. `/v1/models` correct. Every completion came back
|
||||
`finish_reason=length` with the **full** token count and `content: null` — the model was generating
|
||||
a full budget of tokens that decoded to the empty string. Raw `/v1/completions` was empty too, which
|
||||
ruled out the chat template and the reasoning parser. The one signal that named it: `logprobs: 1`
|
||||
→ HTTP 400 `Out of range float values are not JSON compliant: nan`.
|
||||
|
||||
⚠⚠ **The durable lesson is about the CONTROL, not the router.** That broken tree HAD been
|
||||
structurally diffed before it shipped — and passed — against a verified-good **dense** 31B quant of
|
||||
the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the
|
||||
single thing that control could not distinguish. **A positive control is only worth what it can
|
||||
distinguish; "same family" is not "same architecture class."** Diffing instead against **Pfish-6**
|
||||
— a known-good NVFP4A16 quant of the same 26B-A4B MoE — gave the answer in one line: 222 ignore
|
||||
entries against 252, the 30 missing being exactly `model.language_model.layers.N.router.proj`.
|
||||
|
||||
Re-quantized with `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts
|
||||
11,520 expert Linears and refuses a router in the quantize set, both before any GPU time. 90 s.
|
||||
Live seat then passed prose / vision / tool-call / logprobs. Broken tree parked as
|
||||
`...-NVFP4A16.BROKEN-routers-quantized-20260910`.
|
||||
|
||||
**§4.4's temp port was not reachable, and the fallback mattered.** 15.9 GiB of weights + KV +
|
||||
multimodal encoder-cache profiling does not fit in ~19 GiB free beside GPU1's six other tenants:
|
||||
`gpu-memory-utilization 0.20` refused admission (18.26 free vs 18.99 wanted) and `0.185` OOM'd in
|
||||
encoder profiling. Substitute: named `.env` backup, prove the seat on its real port **while no
|
||||
gateway alias points at it**, move the alias last. That ordering is the only reason a NaN-serving
|
||||
seat never reached a consumer. Cost: ~16 min of seat downtime, twice; zero broken aliases.
|
||||
|
||||
Runbook: `services/erp-seat-quant/RUNBOOK-char-rp-fast-swap.md`. Playbook §3.15 (failure signature +
|
||||
the control-class rule), §4.4 (ask for logprobs once).
|
||||
|
||||
Related: [[2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot]]
|
||||
|
||||
+35
-7
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-10 11:00 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; **MeroMero BOTH quants landed** — v2 dense on attempt 5, serve test still owed; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_
|
||||
_Last updated: 2026-09-10 11:35 PT (R49 1-epoch pilot COMPLETE + all 3 arms cut, awaiting adjudication; **MeroMero BOTH quants landed and the A4B is LIVE on :8021 as `char-rp-fast`, `Pfish-6` alias removed** — its first quant served NaN and looked healthy; althing 3.6.2 on post office + both heralds; ~574 GB reclaimed)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -141,9 +141,20 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
### MeroMero seats
|
||||
|
||||
- ✅ `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` — 16 G, W4A16, §4.3 post-steps DONE.
|
||||
⚠ It had the §3.14 **truncation cap baked into `tokenizer.json`** (`max_length: 8192`, because it
|
||||
was quantized *with* the corpus). Fixed; backup `tokenizer.json.bak-pre-truncfix`.
|
||||
- ✅ **LIVE on ana-ml2 `:8021` as gateway alias `char-rp-fast`** (operator, 2026-09-10: *"replace
|
||||
that a4b moe over pfish-6 — remove the pfish-6 alias and create an alias for char-rp-fast"*).
|
||||
`G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`, 16 G, served under its own true name on the
|
||||
**`erp-seat` stack** (name kept: asset-engine derives liveness from the compose project name).
|
||||
Verified end to end — prose, **vision (reads a solid-colour image)**, auto tool call, finite
|
||||
logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged.
|
||||
⚠⚠ **ITS FIRST QUANT SERVED NaN AND PASSED ITS HEALTHCHECK DOING IT.** Built with the DENSE
|
||||
recipe (no `re:.*router.*` in IGNORE) → all 30 MoE routers quantized to 4 bits → expert selection
|
||||
destroyed. Every request returned `finish_reason=length` with the FULL token count and
|
||||
`content: null`; the only tell was **NaN logprobs**. Re-quantized with
|
||||
`services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (its guard refuses exactly that). Broken
|
||||
tree parked at `...-NVFP4A16.BROKEN-routers-quantized-20260910` — **do not serve it**.
|
||||
⚠ Also had the §3.14 truncation cap baked in (`max_length: 8192`, quantized *with* the corpus);
|
||||
the MoE recipe's own post-step resets it.
|
||||
- ✅ `G4-MeroMero-v2-31B-heretic-NVFP4A16` — **19 G, landed on attempt 5.** Tensor table identical
|
||||
family-for-family to the 2026-08-21 canonical quant; **356 BF16 vision tensors preserved**;
|
||||
`input_activations=None` (genuinely A16). CPU load+generate coherent, 0 tensors on meta.
|
||||
@@ -152,11 +163,27 @@ _As of 2026-09-10 10:25 PT._
|
||||
the redundant `per_layer_config`, not to force global access.
|
||||
→ `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md`,
|
||||
`services/meromero-quant/`, playbook **§3.17 (new)**
|
||||
- ⛔ **NEITHER quant has had its §4.4 serve test** — GPU1 has 19.9 GB free against 19.5 GB of v2
|
||||
weights, so it needs a live seat displaced. Operator's call; *"vllm servable"* unverified until then.
|
||||
- ⛔ **The v2 DENSE has still not had a serve test** — GPU1 has ~19 GB free against 19.5 GB of its
|
||||
weights, so it needs a live seat displaced. Operator's call; *"vllm servable"* unverified for the
|
||||
dense tree. (The A4B's serve test is DONE and green, above.)
|
||||
- ⚠ **A co-resident temp port could not be made to fit even for the 16 G A4B**, so §4.4's "temp port,
|
||||
never the live seat" was substituted with reversibility: named `.env` backup, prove the seat on its
|
||||
real port while no alias routes to it, move the alias last. Measured refusals: `gpu-memory-util 0.20`
|
||||
→ admission refused (18.26 GiB free vs 18.99 requested); `0.185` → past admission and past the KV
|
||||
reservation, then OOM in **multimodal encoder-cache profiling** (3 video items at max feature size),
|
||||
which is easy to forget when budgeting a vision model.
|
||||
- ⚠ **The A4B is 30 layers / kv 8 — Pfish-6's geometry**, so it fits 262k in the existing KV budget.
|
||||
The dense 31B is 60/16, ~4x KV per token, and does NOT fit 262k on GPU1 beside the other seats.
|
||||
- Pfish-6 remains the standing seat on ana-ml2 `:8021`. Nothing repointed, no `rp-fast` alias exists.
|
||||
- ⛔ **`Pfish-6` IS RETIRED from the gateway** (2026-09-10). Its seat now serves the A4B; a caller
|
||||
asking for `Pfish-6` gets an explicit 400 "Invalid model name", not a substitution. Audited first:
|
||||
**0 of 17 LiteLLM keys** scoped it, so nothing was orphaned. The artifact stays on disk at
|
||||
`/tank/aimodels/erp-tune-v6-nvfp4a16` and is the rollback target — `cp
|
||||
.env.pfish6.bak-20260910 .env && docker compose up -d` in `/opt/docker/compose/erp-seat` restores
|
||||
it in ~4 min.
|
||||
- The `char-rp` family is now **`char-rp` (stock Gemma-4 26B-A4B, :8016) / `char-rp-fast` (MeroMero
|
||||
A4B, :8021) / `char-rp-reasoning` (Dark-Scarlett-27B, :8019)**. All three verified working after
|
||||
the change. ⚠ The `char-rp` comment block in `stacks/litellm/conf/config.yaml` still describes its
|
||||
seat as MeroMero-v2; that has been stale since the 2026-08-24 swap to stock Gemma-4. Not fixed.
|
||||
|
||||
### Fleet
|
||||
|
||||
@@ -181,6 +208,7 @@ _As of 2026-09-10 10:25 PT._
|
||||
|
||||
- `[2026-09-10]` **R49 carrier SETTLED on dense `Qwen3-{0.6,1.7,4}B-Base`, overriding H02's own pin — the newest carrier was the SLOW one.** Dense 4.089 B trains 33% faster than hybrid 0.765 B; no fused SSM kernel installed. D1–D3 built, 1-epoch pilot beats the 3-epoch by 0.21 nats held-out. → `persistent-memory.d/2026-09-10-r49-babybronte-d1-d3-and-the-1-epoch-pilot.md`
|
||||
- `[2026-09-10]` **R49 adjudication routed to infra-ops entirely** (operator, relayed by brokkr: *"leave babybronte to infra — concentrate on r50 and the memory mechanism"*). brokkr handed over the Delta instrument and stepped off. ⚠ I now grade my own run; brokkr's decision rule is **ratified verbatim and frozen before any adapted text existed** and must not be amended after seeing numbers. Their controls: real Charlotte 1.65–2.17, **Anne at 2.374** — so the absolute band decides, never `nearest`.
|
||||
- `[2026-09-10]` **MeroMero A4B swapped onto the `erp-seat` seat as `char-rp-fast`; `Pfish-6` alias removed.** The A4B's FIRST quant used the dense recipe and 4-bit-quantized all 30 MoE routers — it passed its healthcheck and answered every request with the full token count decoding to the empty string, NaN logits the only tell. Re-quantized with the MoE recipe; live and verified (prose, vision, tool call, finite logprobs). Durable lesson: **a positive control must match the ARCHITECTURE CLASS** — the broken A4B was diffed against a good *dense* quant, which has no routers, so the clean result was meaningless. → playbook §3.15, §4.4
|
||||
- `[2026-09-10]` **MeroMero: BOTH quants landed in-house at W4A16 — A4B first try, v2 dense on attempt 5.** Published quants are all W4A4 (our measured long-context collapse) or nonexistent for v2. Operator: *"pull both ablits bf16, run our own quant."* The durable lesson is **§3.17**: `pip install llmcompressor` silently pins transformers down a version, so attempt 4's error was a moved toolchain, not the malformed upload it looked like — a known-good positive control is what told them apart. Serve test still owed. → `persistent-memory.d/2026-09-10-meromero-quants-and-the-pinned-transformers-trap.md`
|
||||
- `[2026-09-10]` **althing 3.6.2 deployed — post office + both heralds — and the fleet has TWO herald nodes, not seven.** Ask the post office's `nodes` table, not the box inventory. Cost a self-inflicted ~12 min bus outage. → `persistent-memory.d/2026-09-10-althing-362-rollout.md`
|
||||
- `[2026-09-10]` **A grep over a log that records your greps counts itself.** I reported forseti's drop defect as reproducing here with 3 drops in 21 s; the session had **zero**. Searching transcripts writes the search term into them. Filter by `"type":"system"` provenance, never content. Generalises to any instrument that can see itself. Auto-memory `feedback_grep_over_a_log_that_records_your_greps`.
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# char-rp-fast — swapping the MeroMero A4B onto the erp-seat seat (2026-09-10)
|
||||
|
||||
Operator: *"replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an
|
||||
alias for char-rp-fast."*
|
||||
|
||||
Result: `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` is live on ana-ml2 `:8021`
|
||||
behind gateway alias `char-rp-fast`. `Pfish-6` is gone from the gateway. It took two
|
||||
attempts, because the first quant was broken in a way that looks exactly like a healthy seat.
|
||||
|
||||
## ⚠ The failure worth remembering: a 4-bit MoE router serves NaN and passes its healthcheck
|
||||
|
||||
The A4B built that morning used `services/meromero-quant/quant_a16_datafree.py` — the **dense**
|
||||
v2-31B recipe. Its IGNORE list has no `re:.*router.*` entry, so all 30 MoE routers were quantized
|
||||
to NVFP4. A 4-bit router does not degrade expert selection, it *changes which experts run*
|
||||
(playbook §3.15).
|
||||
|
||||
What that looked like on the seat, in order of how convincing each signal was:
|
||||
|
||||
| signal | what it said |
|
||||
|---|---|
|
||||
| quant exit code | `rc=0`, 16 G, no warning |
|
||||
| `docker` healthcheck | healthy in 210 s |
|
||||
| engine log | KV pool 534,649 tokens, 2.04x — exactly right |
|
||||
| `/v1/models` | correct served name, 262,144 context |
|
||||
| every completion | `finish_reason: "length"`, **full** `completion_tokens` (120/120, 600/600) |
|
||||
| `content` | `null`. Every time. |
|
||||
| raw `/v1/completions` | `text: ''` — so it was not the chat template or the reasoning parser |
|
||||
| **`logprobs: 1`** | **HTTP 400 `Out of range float values are not JSON compliant: nan`** |
|
||||
|
||||
The model was generating a full budget of tokens that decoded to the empty string, and the only
|
||||
thing that named the fault was asking for logprobs. `seat_verify.py` now carries that as check 6.
|
||||
|
||||
**What actually found it** was not the CPU forward (started, then abandoned as too slow): it was
|
||||
diffing `quantization_config.ignore` against **Pfish-6** — a known-good NVFP4A16 quant of the
|
||||
*same architecture class*. 222 entries against 252, and the 30 missing were precisely
|
||||
`model.language_model.layers.N.router.proj`.
|
||||
|
||||
⚠⚠ **The broken tree HAD been structurally diffed before it shipped — against a verified-good
|
||||
DENSE 31B quant of the same Gemma-4 family, which came back clean.** A dense model has no
|
||||
routers, so the one thing that was wrong was the one thing that control could not see. **A
|
||||
positive control is only worth what it can distinguish; "same family" is not "same architecture
|
||||
class."**
|
||||
|
||||
Fix: re-quantize with `quant_nvfp4a16_gemma4_moe.py`, whose `--dry-run` asserts
|
||||
`layers × experts × 3 = 11,520` expert Linears and refuses if a router lands in the quantize set,
|
||||
both before any GPU time. 90 seconds end to end. The broken tree is parked on ana-ml2 as
|
||||
`...-NVFP4A16.BROKEN-routers-quantized-20260910`. **Do not serve it.**
|
||||
|
||||
## Why the seat went dark for ~16 minutes instead of not at all
|
||||
|
||||
Playbook §4.4 wants a temp port. It was not reachable, twice, and the numbers are worth keeping:
|
||||
|
||||
- `--gpu-memory-utilization 0.20` → **admission refused**: `Free memory on device cuda:0
|
||||
(18.26/94.97 GiB) on startup is less than desired GPU memory utilization (0.2, 18.99 GiB)`.
|
||||
- `0.185` + `--kv-cache-memory 1.5 GB` + `--max-model-len 8192` + `--enforce-eager` → past
|
||||
admission, past the KV reservation, then `torch.OutOfMemoryError` during **multimodal
|
||||
encoder-cache profiling** (`profiled with 3 video items of the maximum feature size`). That
|
||||
profiling cost is easy to forget when budgeting a vision model.
|
||||
|
||||
15.9 GiB of weights plus a KV pool plus vision profiling does not fit in the ~19 GiB free beside
|
||||
the other six GPU1 tenants. So the substitute was **reversibility and ordering**:
|
||||
|
||||
1. back the host `.env` up to a *named* file first (`.env.pfish6.bak-20260910`);
|
||||
2. swap `.env`, `up -d`, and prove the seat on its real port **while no gateway alias points at
|
||||
it**;
|
||||
3. move the gateway alias **last**.
|
||||
|
||||
That ordering is why the NaN-serving seat never reached a consumer — `char-rp-fast` did not exist
|
||||
yet and `Pfish-6` still resolved to nothing else. The cost was ~16 minutes of that one seat being
|
||||
down, twice, and nothing downstream saw a broken alias.
|
||||
|
||||
## The swap, as steps
|
||||
|
||||
```bash
|
||||
# on ana-ml2, /opt/docker/compose/erp-seat
|
||||
cp -n .env .env.pfish6.bak-20260910 # ROLLBACK LIVES HERE
|
||||
# point ERP_MODEL / ERP_SERVED_NAME / ERP_CHAT_TEMPLATE at the new tree
|
||||
sudo docker compose up -d # ~210 s to healthy
|
||||
|
||||
# verify BEFORE touching the gateway
|
||||
python3 seat_verify.py http://127.0.0.1:8021/v1 <served-name>
|
||||
|
||||
# gateway (canonical: stacks/litellm/conf/config.yaml)
|
||||
scripts/deploy-stack.sh ana-docker litellm --conf
|
||||
ssh ana-docker 'cd /opt/docker/compose/litellm && sudo docker compose restart litellm'
|
||||
```
|
||||
|
||||
**Rollback to Pfish-6** is `cp .env.pfish6.bak-20260910 .env && sudo docker compose up -d`,
|
||||
~4 minutes. `/tank/aimodels/erp-tune-v6-nvfp4a16` is untouched.
|
||||
|
||||
## What was checked, and what was not
|
||||
|
||||
Verified on the live seat (`raw/char-rp-fast-seat-verification-2026-09-10.txt`): served name and
|
||||
262,144 context; KV 534,649 tokens / 2.04x; clean prose with no `<|channel>thought` leak and no
|
||||
reasoning field; **a solid-colour image read correctly**, so vision is tested rather than inferred
|
||||
from a tensor count; an auto `tool_choice` call parsed with correct arguments; finite logprobs.
|
||||
Through the gateway with the shared `all-agents-local` key: `char-rp-fast` answers, `Pfish-6`
|
||||
returns an explicit `400 Invalid model name` rather than a substitution, and `char-rp` /
|
||||
`char-rp-reasoning` are both unaffected.
|
||||
|
||||
Audited before removing the alias: **0 of 17 LiteLLM keys** named `Pfish-6` in their model
|
||||
allowlist, so nothing was orphaned (1 of 17 is unrestricted and reaches whatever the gateway
|
||||
serves). ⚠ The first attempt at that audit passed `size=200` and got a silent `422`, which the
|
||||
script reported as "scanned 0 keys" — an empty result and a rejected query look identical if you
|
||||
do not check.
|
||||
|
||||
**Not established:** anything about quality. No RP eval, no long-context check, no A/B against
|
||||
Pfish-6 or `char-rp`. The samplers are the author's card values (Temp 0.8–1.0, MinP 0.05), not
|
||||
tuned here. n=1 smoke output is not evidence about writing.
|
||||
|
||||
⚠ Pre-existing doc rot noticed and **not** fixed: the `char-rp` comment block in
|
||||
`stacks/litellm/conf/config.yaml` still describes its `:8016` seat as MeroMero-v2. That has been
|
||||
stale since the 2026-08-24 swap to stock Gemma-4.
|
||||
@@ -0,0 +1,39 @@
|
||||
### char-rp-fast seat verification — ana-ml2 :8021, 2026-09-10
|
||||
### model: G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 (MoE-recipe re-quant)
|
||||
|
||||
$ docker logs vllm-erp-seat | grep 'GPU KV cache size'
|
||||
(EngineCore pid=663) INFO 09-10 18:30:39 [kv_cache_utils.py:1869] GPU KV cache size: 534,649 tokens, Maximum concurrency for 262,144 tokens per request: 2.04x
|
||||
|
||||
$ python3 seat_verify.py http://127.0.0.1:8021/v1 <served-name>
|
||||
== 1. served name + context
|
||||
served: ['G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16']
|
||||
max_model_len: {'G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16': 262144}
|
||||
OK 'G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16' is served
|
||||
|
||||
== 3. prose, non-thinking (the <|channel>thought leak)
|
||||
content (277 chars): 'Oil-slicked puddles mirror the fractured glow of a flickering neon sign, casting distorted crimson light across the uneven cobblestones. The sharp, metallic tang of wet iron clings to the air as water cascades rhythmical'
|
||||
reasoning_content: None
|
||||
OK clean prose in content, no reasoning, no channel prefix
|
||||
|
||||
== 4. vision (towers preserved, tested not inferred)
|
||||
answer: 'Blue' (image was solid RGB(30,60,200) = blue)
|
||||
OK image was decoded and read correctly
|
||||
|
||||
== 5. tool call (auto)
|
||||
tool_calls: [{"id": "chatcmpl-tool-ba6a1874968381f1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Anaheim\"}"}}]
|
||||
content: ''
|
||||
OK parsed a get_weather call, arguments='{"city": "Anaheim"}'
|
||||
|
||||
== 6. logprobs (NaN logits, the router-quant tell)
|
||||
text: '</b></b></b></b></b></b></b><b>'
|
||||
token_logprobs: [-1.3935617208480835, -0.1289057433605194, -0.006735478527843952, -0.006430173758417368, -0.005962086841464043]
|
||||
OK finite logprobs, non-empty raw text
|
||||
|
||||
============================================================
|
||||
ALL CHECKS PASSED
|
||||
|
||||
### ignore-list diff vs Pfish-6 (the known-good MoE quant of the SAME architecture class)
|
||||
Pfish-6 (known good) : 252 ignore entries
|
||||
A4B re-quant (live) : 252 ignore entries identical to Pfish-6: True
|
||||
A4B FIRST quant (bad) : 222 ignore entries missing vs good: 30
|
||||
the missing ones : ['model.language_model.layers.0.router.proj', 'model.language_model.layers.1.router.proj', 'model.language_model.layers.10.router.proj'] ... (all 30 are layers.N.router.proj)
|
||||
@@ -0,0 +1,210 @@
|
||||
"""Verify the swapped char-rp-fast seat before the gateway alias points at it.
|
||||
|
||||
The order matters: the seat is proven on its direct port FIRST, and only then does
|
||||
`char-rp-fast` start resolving. That way no consumer ever sees a half-working alias
|
||||
-- which is the reason playbook §4.4 wants a temp port. A temp port was not
|
||||
reachable here (18.26 GiB free against 15.9 GiB of weights plus a 8.5 GiB KV pool),
|
||||
so the substitute is: prove it on :8021 while nothing routes to it, and keep the
|
||||
one-flip rollback to Pfish-6 intact until it passes.
|
||||
|
||||
Five checks, and each one exists because this seat family has broken in that exact
|
||||
way before:
|
||||
|
||||
1. served name + context -- a stale served-name is a silent substitution
|
||||
2. KV pool -- Pfish-6's 9.114 GB pinning should transfer, because
|
||||
the architecture is identical field for field; if the
|
||||
token count moved, that assumption was wrong
|
||||
3. prose, non-thinking -- the `<|channel>thought` leak into content, which
|
||||
stacks/gemma4-charrp/README.md warns about and which
|
||||
was measured 3/3 on this recipe without the parser pin
|
||||
4. vision -- the "vision towers intact" claim, tested rather than
|
||||
inferred from a tensor count
|
||||
5. tool call (auto) -- the seat advertises gemma4 tool parsing
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import zlib
|
||||
|
||||
BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8021/v1"
|
||||
MODEL = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
KEY = sys.argv[3] if len(sys.argv) > 3 else None
|
||||
|
||||
fails = []
|
||||
|
||||
|
||||
def post(path, body, timeout=180):
|
||||
req = urllib.request.Request(
|
||||
BASE + path, data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json",
|
||||
**({"Authorization": f"Bearer {KEY}"} if KEY else {})})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def get(path, timeout=30):
|
||||
req = urllib.request.Request(
|
||||
BASE + path,
|
||||
headers={**({"Authorization": f"Bearer {KEY}"} if KEY else {})})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.load(r)
|
||||
|
||||
|
||||
def png(rgb, w=64, h=64):
|
||||
"""Minimal solid-colour PNG, built here so the test needs no asset on disk."""
|
||||
raw = b"".join(b"\x00" + bytes(rgb) * w for _ in range(h))
|
||||
|
||||
def chunk(tag, data):
|
||||
c = tag + data
|
||||
return struct.pack(">I", len(data)) + c + struct.pack(">I", zlib.crc32(c))
|
||||
|
||||
return (b"\x89PNG\r\n\x1a\n"
|
||||
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
||||
+ chunk(b"IDAT", zlib.compress(raw))
|
||||
+ chunk(b"IEND", b""))
|
||||
|
||||
|
||||
# ---- 1. served name + context -------------------------------------------------
|
||||
print("== 1. served name + context")
|
||||
models = get("/models")
|
||||
ids = [m["id"] for m in models["data"]]
|
||||
mlen = {m["id"]: m.get("max_model_len") for m in models["data"]}
|
||||
print(f" served: {ids}")
|
||||
print(f" max_model_len: {mlen}")
|
||||
if MODEL:
|
||||
if MODEL in ids:
|
||||
print(f" OK '{MODEL}' is served")
|
||||
else:
|
||||
fails.append(f"'{MODEL}' not in served names {ids}")
|
||||
print(f" *** '{MODEL}' NOT SERVED")
|
||||
target = MODEL if MODEL in ids else ids[0]
|
||||
if "Pfish-6" in ids:
|
||||
fails.append("Pfish-6 is STILL served -- the swap did not take")
|
||||
print(" *** Pfish-6 still served")
|
||||
|
||||
# ---- 3. prose, non-thinking ---------------------------------------------------
|
||||
print("\n== 3. prose, non-thinking (the <|channel>thought leak)")
|
||||
r = post("/chat/completions", {
|
||||
"model": target,
|
||||
"messages": [{"role": "user", "content":
|
||||
"Describe a rain-slicked alley at night in two sentences."}],
|
||||
"max_tokens": 120,
|
||||
})
|
||||
msg = r["choices"][0]["message"]
|
||||
content = msg.get("content") or ""
|
||||
reasoning = msg.get("reasoning_content") or msg.get("reasoning")
|
||||
print(f" content ({len(content)} chars): {content[:220]!r}")
|
||||
print(f" reasoning_content: {reasoning!r}")
|
||||
if not content.strip():
|
||||
fails.append("prose: content is empty")
|
||||
print(" *** content EMPTY")
|
||||
elif "<|channel" in content or "channel>thought" in content:
|
||||
fails.append("prose: <|channel>thought prefix leaked into content")
|
||||
print(" *** CHANNEL PREFIX LEAKED into content")
|
||||
elif reasoning:
|
||||
fails.append(f"prose: reasoning_content populated with enable_thinking=false ({len(reasoning)} chars)")
|
||||
print(" *** reasoning_content populated despite enable_thinking=false")
|
||||
else:
|
||||
print(" OK clean prose in content, no reasoning, no channel prefix")
|
||||
|
||||
# ---- 4. vision ----------------------------------------------------------------
|
||||
print("\n== 4. vision (towers preserved, tested not inferred)")
|
||||
blue = base64.b64encode(png((30, 60, 200))).decode()
|
||||
try:
|
||||
r = post("/chat/completions", {
|
||||
"model": target,
|
||||
"messages": [{"role": "user", "content": [
|
||||
{"type": "text", "text":
|
||||
"This image is one flat colour. Name that colour in one word."},
|
||||
{"type": "image_url",
|
||||
"image_url": {"url": f"data:image/png;base64,{blue}"}},
|
||||
]}],
|
||||
"max_tokens": 24,
|
||||
"temperature": 0,
|
||||
})
|
||||
v = (r["choices"][0]["message"].get("content") or "").strip()
|
||||
print(f" answer: {v!r} (image was solid RGB(30,60,200) = blue)")
|
||||
if "blue" in v.lower():
|
||||
print(" OK image was decoded and read correctly")
|
||||
elif v:
|
||||
fails.append(f"vision: answered {v!r} for a solid blue image")
|
||||
print(" *** answered, but not blue -- vision path suspect")
|
||||
else:
|
||||
fails.append("vision: empty answer")
|
||||
print(" *** empty answer")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:300]
|
||||
fails.append(f"vision: HTTP {e.code} {body}")
|
||||
print(f" *** HTTP {e.code}: {body}")
|
||||
|
||||
# ---- 5. tool call -------------------------------------------------------------
|
||||
print("\n== 5. tool call (auto)")
|
||||
try:
|
||||
r = post("/chat/completions", {
|
||||
"model": target,
|
||||
"messages": [{"role": "user", "content": "What is the weather in Anaheim?"}],
|
||||
"tools": [{"type": "function", "function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city.",
|
||||
"parameters": {"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"]}}}],
|
||||
"tool_choice": "auto",
|
||||
"max_tokens": 120,
|
||||
})
|
||||
m = r["choices"][0]["message"]
|
||||
tc = m.get("tool_calls")
|
||||
print(f" tool_calls: {json.dumps(tc)[:240] if tc else None}")
|
||||
print(f" content: {(m.get('content') or '')[:120]!r}")
|
||||
if tc and tc[0]["function"]["name"] == "get_weather":
|
||||
args = tc[0]["function"].get("arguments")
|
||||
print(f" OK parsed a get_weather call, arguments={args!r}")
|
||||
else:
|
||||
fails.append("tool call: no parsed get_weather tool_call")
|
||||
print(" *** no parsed tool call (auto tool_choice)")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:300]
|
||||
fails.append(f"tool call: HTTP {e.code} {body}")
|
||||
print(f" *** HTTP {e.code}: {body}")
|
||||
|
||||
# ---- 6. NaN logits -----------------------------------------------------------
|
||||
print("\n== 6. logprobs (NaN logits, the router-quant tell)")
|
||||
try:
|
||||
r = post("/completions", {
|
||||
"model": target, "prompt": "Rain on asphalt at midnight.",
|
||||
"max_tokens": 8, "temperature": 0, "logprobs": 1,
|
||||
})
|
||||
txt = r["choices"][0].get("text")
|
||||
lp = r["choices"][0].get("logprobs") or {}
|
||||
vals = lp.get("token_logprobs") or []
|
||||
print(f" text: {txt!r}")
|
||||
print(f" token_logprobs: {vals[:5]}")
|
||||
if not (txt or "").strip():
|
||||
fails.append("logprobs: raw completion decoded to the empty string -- generating, but no text")
|
||||
print(" *** EMPTY raw completion: tokens generated that decode to nothing")
|
||||
elif any(v is None or v != v for v in vals):
|
||||
fails.append("logprobs: NaN/None in token_logprobs")
|
||||
print(" *** NaN in token_logprobs")
|
||||
else:
|
||||
print(" OK finite logprobs, non-empty raw text")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()[:300]
|
||||
# vLLM cannot serialize NaN, so the 400 IS the positive finding here.
|
||||
if "nan" in body.lower():
|
||||
fails.append("logprobs: NaN logits -- vLLM refused to serialize them. "
|
||||
"On a MoE this is the router-quantized signature (playbook §3.15)")
|
||||
print(f" *** NaN LOGITS: {body}")
|
||||
else:
|
||||
fails.append(f"logprobs: HTTP {e.code} {body}")
|
||||
print(f" *** HTTP {e.code}: {body}")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
if fails:
|
||||
print(f"FAILED ({len(fails)}):")
|
||||
for f in fails:
|
||||
print(f" - {f}")
|
||||
sys.exit(1)
|
||||
print("ALL CHECKS PASSED")
|
||||
@@ -1,12 +1,15 @@
|
||||
# erp-seat — ana-ml2 GPU1. Real .env lives on the host at /opt/docker/compose/erp-seat/.env.
|
||||
ERP_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0
|
||||
ERP_MODEL=/tank/aimodels/erp-tune-v6-nvfp4a16
|
||||
ERP_SERVED_NAME=erp-tune-v6-nvfp4a16
|
||||
ERP_CHAT_TEMPLATE=/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja
|
||||
ERP_MODEL=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
ERP_SERVED_NAME=G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
ERP_CHAT_TEMPLATE=/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja
|
||||
ERP_PORT=8021
|
||||
ERP_GPU_ID=1
|
||||
# 0.35 x 97.9 GiB = 34 GiB. GPU1 had ~47 GiB free on 2026-09-08 (scriberr/embed/rerank/coder/reward resident).
|
||||
ERP_GPU_MEM_UTIL=0.35
|
||||
ERP_MAX_MODEL_LEN=32768
|
||||
ERP_MAX_NUM_SEQS=8
|
||||
ERP_GPU_MEM_UTIL=0.30
|
||||
ERP_MAX_MODEL_LEN=262144
|
||||
ERP_MAX_NUM_SEQS=32
|
||||
API_KEY=
|
||||
# 8.49 GiB -> 534,649 KV tokens -> 2.04x a 262,144 context (operator's KV = 2x rule).
|
||||
ERP_KV_CACHE_MEMORY=9114000000
|
||||
ERP_MOE_BACKEND=auto
|
||||
|
||||
@@ -1,17 +1,38 @@
|
||||
# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of **Pfish-6**, the run-6 LoRA
|
||||
# merge on the jenerallee78 ARA abliteration, served under that name.
|
||||
# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
|
||||
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
|
||||
#
|
||||
# ⚠ RUN 7 IS RETIRED (operator ruling 2026-09-09): "we're gonna stay on 6 for now". Run 7's
|
||||
# gate failure turned out to be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed
|
||||
# cc42d76 in brokkr-smithy) — but run 7 was independently a poor run (primary FLAT +2, both
|
||||
# diversity families reduced, long-context coherence 1.0 -> 0.875). Run 6 is the standing seat.
|
||||
# Routing aliases (e.g. LiteLLM `trial`) are the operator's call and live in the gateway, not here.
|
||||
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
|
||||
# 2026-09-10, when the operator swapped the occupant: "replace that a4b moe over pfish-6 --
|
||||
# remove the pfish-6 alias and create an alias for char-rp-fast." The compose PROJECT name is
|
||||
# deliberately NOT renamed: asset-engine derives seat liveness from it, so a rename reads as
|
||||
# OFFLINE. Pfish-6 remains on disk at /tank/aimodels/erp-tune-v6-nvfp4a16 and the pre-swap host
|
||||
# env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- one cp plus `up -d` back.
|
||||
#
|
||||
# ⚠ THE FIRST A4B QUANT SERVED NaN AND LOOKED HEALTHY DOING IT. It was built with the DENSE
|
||||
# recipe, whose ignore list carries no router regex, so all 30 MoE routers were quantized to
|
||||
# 4 bits and expert selection was destroyed (playbook §3.15). The seat passed its healthcheck,
|
||||
# returned finish_reason=length with the full token count, and every response decoded to the
|
||||
# empty string; the give-away was NaN logprobs. Re-quantized with
|
||||
# services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose target guard refuses exactly
|
||||
# that. Use the MoE recipe for anything in this family; the dense one is for the v2-31B.
|
||||
#
|
||||
# PFISH-6 PROVENANCE, kept because it is still the rollback target. Run 7 was retired by
|
||||
# operator ruling 2026-09-09 ("we're gonna stay on 6 for now"): its gate failure turned out to
|
||||
# be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed cc42d76 in brokkr-smithy), but
|
||||
# run 7 was independently a poor run (primary FLAT +2, both diversity families reduced,
|
||||
# long-context coherence 1.0 -> 0.875). Run 6 was the standing seat here until the 2026-09-10
|
||||
# swap above. Routing aliases live in the gateway, not here.
|
||||
#
|
||||
# Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this
|
||||
# box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template.
|
||||
# GPU1 is SHARED (charrp-MoE moved? no — scriberr, embed, rerank, coder, reward live there):
|
||||
# ~47 GiB was free on 2026-09-08; 0.35 x 97.9 GiB = 34 GiB keeps ~13 GiB of real margin.
|
||||
# Quant pipeline: services/erp-seat-quant/. Tunables in .env.
|
||||
# It carries over to MeroMero A4B unchanged -- verified 2026-09-10 end to end: clean prose with
|
||||
# no channel-prefix leak, a solid-colour image read correctly (vision towers intact), and an
|
||||
# auto tool_choice call parsed.
|
||||
# GPU1 is SHARED (scriberr, embed, rerank, coder, reward, charrp live there): the live .env runs
|
||||
# ERP_GPU_MEM_UTIL=0.30, and the KV pool is pinned in bytes below regardless, so the ratio only
|
||||
# has to clear admission.
|
||||
# Quant pipeline: services/erp-seat-quant/ (MoE recipe -- NOT services/meromero-quant/, which is
|
||||
# the dense one). Tunables in .env.
|
||||
|
||||
name: erp-seat
|
||||
|
||||
@@ -28,11 +49,11 @@ services:
|
||||
environment:
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${ERP_MODEL:-/tank/aimodels/erp-tune-v6-nvfp4a16}
|
||||
- ${ERP_MODEL:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
|
||||
- --quantization
|
||||
- compressed-tensors
|
||||
- --served-model-name
|
||||
- ${ERP_SERVED_NAME:-Pfish-6}
|
||||
- ${ERP_SERVED_NAME:-G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
|
||||
- --tool-call-parser
|
||||
- gemma4
|
||||
- --enable-auto-tool-choice
|
||||
@@ -52,7 +73,7 @@ services:
|
||||
# an empty turn. The flag drops the tools from the prompt so the model answers in prose.
|
||||
- --exclude-tools-when-tool-choice-none
|
||||
- --chat-template
|
||||
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja}
|
||||
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja}
|
||||
- --max-model-len
|
||||
- "${ERP_MAX_MODEL_LEN:-262144}"
|
||||
# KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is
|
||||
@@ -71,6 +92,12 @@ services:
|
||||
#
|
||||
# 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is
|
||||
# the operator's sizing rule (KV = 2x max context, 2026-09-09).
|
||||
#
|
||||
# This figure SURVIVED the 2026-09-10 Pfish-6 -> MeroMero-A4B swap unchanged, and that
|
||||
# is not luck: the two are the same architecture field for field (30 layers, kv 8,
|
||||
# head_dim 256, sliding_window 1024, 25 sliding / 5 full, 128 experts top-8), so the
|
||||
# KV-per-token is the same number. Confirmed by reading 534,649 tokens / 2.04x back out
|
||||
# of the new engine's log rather than assuming the pinning carried.
|
||||
- --kv-cache-memory
|
||||
- "${ERP_KV_CACHE_MEMORY:-9114000000}"
|
||||
- --max-num-seqs
|
||||
@@ -122,9 +149,9 @@ services:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=Pfish-6 (Gemma-4 26B-A4B ARA, NVFP4A16)
|
||||
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
|
||||
- homepage.icon=mdi-fire
|
||||
- homepage.description=Pfish-6 — the run-6 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1)
|
||||
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1)
|
||||
- homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
|
||||
|
||||
networks:
|
||||
|
||||
@@ -872,9 +872,10 @@ model_list:
|
||||
# base profile ~0% on 30/35 axes per brokkr) -- this seat is for the operator's ear; treat
|
||||
# it as unrated on every safety axis.
|
||||
#
|
||||
# ROLLBACK: /tank/aimodels/erp-tune-v6-nvfp4a16 is still on disk, and the previous host env
|
||||
# is at /tmp/erp-seat-env.v6.bak on ana-ml2 -- flip ERP_MODEL/ERP_SERVED_NAME/ERP_CHAT_TEMPLATE
|
||||
# in /opt/docker/compose/erp-seat/.env back to v6 and `docker compose up -d`.
|
||||
# ROLLBACK (updated 2026-09-10, the seat now serves MeroMero A4B): Pfish-6 itself is
|
||||
# the rollback target. /tank/aimodels/erp-tune-v6-nvfp4a16 is still on disk and the
|
||||
# pre-swap host env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 --
|
||||
# `cp .env.pfish6.bak-20260910 .env && docker compose up -d` restores Pfish-6 in ~4 min.
|
||||
#
|
||||
# THIS GATEWAY IS THE SHARED-KEY SURFACE: `all-agents-local` reaches every model here,
|
||||
# in every session and project. Removing this alias does not remove the operator's
|
||||
@@ -882,21 +883,50 @@ model_list:
|
||||
#
|
||||
# Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing).
|
||||
|
||||
# Pfish-6 — the STANDING seat as of 2026-09-09. NVFP4A16 quant of the run-6
|
||||
# LoRA merge on the jenerallee78 ARA abliteration, on ana-ml2 GPU1 (:8021).
|
||||
# Operator ruling: "declare run 6 as Pfish-6 ... we're gonna stay on 6 for now."
|
||||
# char-rp-fast — the MeroMero A4B MoE, on ana-ml2 GPU1 (:8021). Operator, 2026-09-10:
|
||||
# "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias
|
||||
# for char-rp-fast." REPLACES the `Pfish-6` alias, which is removed with this change.
|
||||
#
|
||||
# REPLACES the `trial` alias, which is retired with run 7. Run 7's CSAM gate
|
||||
# failure turned out to be a DETECTOR BUG (the adjective "minor" matching a
|
||||
# HARD rule — fixed cc42d76 in brokkr-smithy), but run 7 was independently a
|
||||
# poor run and is not coming back.
|
||||
# The seat itself is unchanged in every dimension that matters to a caller: the A4B is
|
||||
# 30 layers / kv 8 / sliding_window 1024 / 128 experts top-8 — field for field the same
|
||||
# geometry as Pfish-6 — so the 9.114 GB KV pinning transfers exactly and the seat still
|
||||
# reports 534,649 tokens and 2.04x concurrency at 262,144. That was verified from the
|
||||
# engine log, not assumed, because KV-per-token is normally NOT transferable.
|
||||
#
|
||||
# Served under its TRUE name. This is now a named seat, not a trial.
|
||||
- model_name: Pfish-6
|
||||
# ⚠ Pfish-6 IS GONE from this gateway and its seat no longer serves that name. A caller
|
||||
# still asking for `Pfish-6` gets a clean 404 rather than a silent substitution, which is
|
||||
# the intended behaviour. The artifact is still on disk (see the ROLLBACK note above).
|
||||
#
|
||||
# SAMPLERS, and they are the author's, not inherited: the model card states Temp 0.8-1.0
|
||||
# and MinP 0.05. Temperature/top_p/top_k already come from the tree's own
|
||||
# generation_config (1.0 / 0.95 / 64) which vLLM applies server-side, and 1.0 sits at the
|
||||
# top of the card's stated range, so the only value that needs stating here is min_p.
|
||||
# Deliberately NOT copying char-rp's temp 1.1 / min_p 0.10 — those were A/B-tuned against
|
||||
# a retired Mistral seat and are inherited, not canonical, as that block's own note says.
|
||||
#
|
||||
# enable_thinking:false is ALSO pinned process-level on the seat
|
||||
# (--default-chat-template-kwargs). Stated here as well so the intent is visible at the
|
||||
# routing layer: without it the gemma4 reasoning parser pre-initialises to REASONING and
|
||||
# plain prose comes back with a null `content`.
|
||||
#
|
||||
# ⚠ ITS FIRST QUANT WAS BROKEN AND SERVED NaN. The 2026-09-10 08:15 build was made with
|
||||
# the DENSE recipe, whose ignore list has no router regex, so all 30 MoE routers were
|
||||
# quantized to 4 bits — a 4-bit router picks different experts (playbook §3.15). The seat
|
||||
# came up healthy, answered every request with 120 tokens, and decoded to the empty string;
|
||||
# logprobs were NaN. Re-quantized with services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py,
|
||||
# whose target guard refuses that exact mistake. The broken tree is parked on ana-ml2 as
|
||||
# ...-NVFP4A16.BROKEN-routers-quantized-20260910. Do not serve it.
|
||||
- model_name: char-rp-fast
|
||||
litellm_params:
|
||||
model: hosted_vllm/Pfish-6
|
||||
model: hosted_vllm/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16
|
||||
api_base: http://10.250.50.54:8021/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
extra_body:
|
||||
min_p: 0.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: false
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
|
||||
Reference in New Issue
Block a user