vh
9a916a759f
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and create an alias for char-rp-fast." G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone from the gateway and now returns an explicit 400 rather than a substitution; 0 of 17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays erp-seat because asset-engine derives seat liveness from it. The first quant of that A4B served NaN and passed its healthcheck doing it. It was built with the dense v2-31B recipe, whose ignore list has no router regex, so all 30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool, correct served name, and every completion returned finish_reason=length with the full token count and content: null. The model was emitting a full budget of tokens that decoded to the empty string. Raw /v1/completions was empty too, ruling out the chat template and the reasoning parser. The signal that named it was logprobs: vLLM refused to serialize the response, "Out of range float values are not JSON compliant: nan". The lesson is about the control rather than the router. That tree had already been structurally diffed and passed -- against a verified-good DENSE quant of the same Gemma-4 family. A dense model has no routers, so the single thing that was wrong was the single thing the control could not distinguish. Diffing instead against Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222 ignore entries against 252, the 30 missing being layers.N.router.proj. A positive control is only worth what it can distinguish, and "same family" is not "same architecture class". Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and refuses a router in the quantize set before any GPU time. The live seat then passed prose with no channel-prefix leak, a solid-colour image read correctly, an auto tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from Pfish-6 unchanged. The broken tree is parked on ana-ml2 as ...-NVFP4A16.BROKEN-routers-quantized-20260910. Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder profiling. The substitute was reversibility and ordering: named .env backup, prove the seat on its real port while no alias points at it, move the alias last. That is why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes across two attempts; no consumer saw a broken alias. Playbook gains the router-quant failure signature and the control-class rule in 3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6. Quality is NOT established: no RP eval, no long-context check, no A/B against Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
erp-seat — ERP-tune seat on ana-ml2 (GPU1, :8021)
Serves the latest gated ERP LoRA merge as an NVFP4A16 (weight-only) compressed-tensors checkpoint so the GX10 is free to train the next run.
Current occupant: run 7 — erp-tune-v7-nvfp4a16 = merged-run07 (jenerallee78
ARA-abliterated Gemma-4-26B-A4B-it, index 33c59654…, + R47 SFT r7 = r6 plus the opening-split
slot and its companion loss mask), quantized by services/erp-seat-quant/ on 2026-09-09.
49 GiB bf16 → 16 GiB NVFP4A16. Runbook docs/runbooks/gx10-run-07.md.
Previous: run 6 (erp-tune-v6-nvfp4a16, 2026-09-08). Its artifact is still on /tank/aimodels/
and the pre-swap host env is at /tmp/erp-seat-env.v6.bak on ana-ml2, so a rollback is an .env
flip plus docker compose up -d.
- True name only.
--served-model-name erp-tune-v7-nvfp4a16. Gateway aliases (trial) are set in LiteLLM on the operator's word, never here (no silent substitution — the bf16 arm on the GX10 and this NVFP4 arm are different artifacts). - Recipe =
stacks/gemma4-charrp(same arch + format, proven on this box):gemma4tool and reasoning parsers,enable_thinkingpinned false, the model's own stock template (ae53464b…, the one it trained through). Without the reasoning parser the post-tool turn leaks<|channel>markers; without the kwargs pin all prose lands inreasoning_content. tool_choice: "none"trap (measured 2026-09-08, fixed with--exclude-tools-when-tool-choice-none). Without the flag vLLM still renders the tools into the prompt, the model emits a tool call anyway, and because parsing is off fornonethe reply iscontent: null, tool_calls: null— an empty turn, 3/3 reproductions. With the flag the tools are dropped from the prompt and the model answers in prose (3/3). The rest of the matrix (auto / required / named / parallel / nested schema / emptytools: []/ streaming / tool-result round trip) was green before and after.stacks/gemma4-charrphas the same exposure and does NOT carry the flag yet.- Forced tool_choice (named /
required) is prompt-driven on EVERY Gemma-4 seat, not grammar-enforced — by vLLM design.vllm/tool_parsers/gemma4_engine_tool_parser.pysetssupports_required_and_named = Falseand itsadjust_requestdeliberately skips the structured-output JSON for required/named so the model can emit its native<|tool_call>call:…syntax. A tune that weakened that syntax (this ERP tune) therefore honours forced calls only sometimes. Measured 2026-09-08, 3 conversations × 3 turns, real system prompt: v0.26.0 1/9; nightly311b3513(v0.27.2rc1, thegenseat's image) 6/9 and the tool-result round trip stays clean 3/3 — so the seat runs the nightly. tts-dev measured 0/18 on v0.26.0 withgen18/18 as the positive control, andresponse_format: json_schema(guided decoding) 18/18 on this seat — that is the deterministic path for a forced call;tool_choice: autoworks normally. A parser plugin that re-enables guided JSON would also need JSON extraction in the engine-parser path; not attempted. - GPU1 is shared — check real usage (
nvidia-smi --query-compute-apps=pid,used_memory) before raisingERP_GPU_MEM_UTIL; the flag sizes KV, not CUDA context. - Rollback / next run: point
ERP_MODEL+ERP_SERVED_NAMEat the next quant dir, keep the previous on disk. Deploy withscripts/deploy-stack.sh ana-ml2 erp-seat.