vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
19aeaedeba21b3d9f36de8b726d9717358a01883
esh-pfi-infrastructure/stacks/erp-seat/compose.yaml
T
vh 9a916a759f Swap the MeroMero A4B onto the erp-seat seat as char-rp-fast, retire the Pfish-6 alias 
Operator: "replace that a4b moe over pfish-6 -- remove the pfish-6 alias and
create an alias for char-rp-fast."

G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16 is live on ana-ml2 :8021 under
its own served name, behind the new gateway alias char-rp-fast. Pfish-6 is gone
from the gateway and now returns an explicit 400 rather than a substitution; 0 of
17 LiteLLM keys scoped it, so nothing was orphaned. The compose project name stays
erp-seat because asset-engine derives seat liveness from it.

The first quant of that A4B served NaN and passed its healthcheck doing it. It was
built with the dense v2-31B recipe, whose ignore list has no router regex, so all
30 MoE routers were quantized to 4 bits -- and a 4-bit router changes which experts
run rather than degrading them. Quant rc=0, healthcheck green, correct KV pool,
correct served name, and every completion returned finish_reason=length with the
full token count and content: null. The model was emitting a full budget of tokens
that decoded to the empty string. Raw /v1/completions was empty too, ruling out the
chat template and the reasoning parser. The signal that named it was logprobs:
vLLM refused to serialize the response, "Out of range float values are not JSON
compliant: nan".

The lesson is about the control rather than the router. That tree had already been
structurally diffed and passed -- against a verified-good DENSE quant of the same
Gemma-4 family. A dense model has no routers, so the single thing that was wrong
was the single thing the control could not distinguish. Diffing instead against
Pfish-6, a known-good quant of the same 26B-A4B MoE, gave it in one line: 222
ignore entries against 252, the 30 missing being layers.N.router.proj. A positive
control is only worth what it can distinguish, and "same family" is not "same
architecture class".

Re-quantized with the MoE recipe, whose dry-run asserts 11,520 expert Linears and
refuses a router in the quantize set before any GPU time. The live seat then passed
prose with no channel-prefix leak, a solid-colour image read correctly, an auto
tool call parsed, finite logprobs, and KV 534,649 tokens / 2.04x carried over from
Pfish-6 unchanged. The broken tree is parked on ana-ml2 as
...-NVFP4A16.BROKEN-routers-quantized-20260910.

Section 4.4's temp port was not reachable: 15.9 GiB of weights plus KV plus
multimodal encoder-cache profiling does not fit in the ~19 GiB free beside GPU1's
six other tenants -- 0.20 utilization refused admission, 0.185 OOM'd in encoder
profiling. The substitute was reversibility and ordering: named .env backup, prove
the seat on its real port while no alias points at it, move the alias last. That is
why a NaN-serving seat never reached a consumer. The seat was down about 16 minutes
across two attempts; no consumer saw a broken alias.

Playbook gains the router-quant failure signature and the control-class rule in
3.15, and a logprobs check in 4.4. seat_verify.py carries that check as check 6.

Quality is NOT established: no RP eval, no long-context check, no A/B against
Pfish-6 or char-rp. Samplers are the author's card values, untuned here.
2026-09-10 11:34:05 -07:00

161 lines
8.7 KiB
YAML
Raw Blame History

# erp-seat — the RP seat on ana-ml2 GPU1. Serves the **MeroMero A4B MoE** NVFP4A16 quant
# (G4-MeroMero-26B-A4B-it-uncensored-heretic) behind the gateway alias `char-rp-fast`.
#
# ⚠ THE STACK NAME IS HISTORICAL. It served Pfish-6 (the run-6 ERP-tune LoRA merge) until
# 2026-09-10, when the operator swapped the occupant: "replace that a4b moe over pfish-6 --
# remove the pfish-6 alias and create an alias for char-rp-fast." The compose PROJECT name is
# deliberately NOT renamed: asset-engine derives seat liveness from it, so a rename reads as
# OFFLINE. Pfish-6 remains on disk at /tank/aimodels/erp-tune-v6-nvfp4a16 and the pre-swap host
# env is at /opt/docker/compose/erp-seat/.env.pfish6.bak-20260910 -- one cp plus `up -d` back.
#
# ⚠ THE FIRST A4B QUANT SERVED NaN AND LOOKED HEALTHY DOING IT. It was built with the DENSE
# recipe, whose ignore list carries no router regex, so all 30 MoE routers were quantized to
# 4 bits and expert selection was destroyed (playbook §3.15). The seat passed its healthcheck,
# returned finish_reason=length with the full token count, and every response decoded to the
# empty string; the give-away was NaN logprobs. Re-quantized with
# services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py, whose target guard refuses exactly
# that. Use the MoE recipe for anything in this family; the dense one is for the v2-31B.
#
# PFISH-6 PROVENANCE, kept because it is still the rollback target. Run 7 was retired by
# operator ruling 2026-09-09 ("we're gonna stay on 6 for now"): its gate failure turned out to
# be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed cc42d76 in brokkr-smithy), but
# run 7 was independently a poor run (primary FLAT +2, both diversity families reduced,
# long-context coherence 1.0 -> 0.875). Run 6 was the standing seat here until the 2026-09-10
# swap above. Routing aliases live in the gateway, not here.
#
# Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this
# box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template.
# It carries over to MeroMero A4B unchanged -- verified 2026-09-10 end to end: clean prose with
# no channel-prefix leak, a solid-colour image read correctly (vision towers intact), and an
# auto tool_choice call parsed.
# GPU1 is SHARED (scriberr, embed, rerank, coder, reward, charrp live there): the live .env runs
# ERP_GPU_MEM_UTIL=0.30, and the KV pool is pinned in bytes below regardless, so the ratio only
# has to clear admission.
# Quant pipeline: services/erp-seat-quant/ (MoE recipe -- NOT services/meromero-quant/, which is
# the dense one). Tunables in .env.
name: erp-seat
services:
vllm-erp-seat:
image: ${ERP_IMAGE:-vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0}
container_name: ${ERP_CONTAINER:-vllm-erp-seat}
restart: unless-stopped
ipc: host
ports:
- "${ERP_PORT:-8021}:8000"
volumes:
- /tank/aimodels:/tank/aimodels
environment:
- VLLM_API_KEY=${API_KEY:-}
command:
- ${ERP_MODEL:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
- --quantization
- compressed-tensors
- --served-model-name
- ${ERP_SERVED_NAME:-G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16}
- --tool-call-parser
- gemma4
- --enable-auto-tool-choice
# --reasoning-parser gemma4 + the enable_thinking pin are BOTH required (measured 2026-09-08):
# without them the post-tool-response turn leaks a literal "<|channel>thought\n<channel|>"
# prefix into content (3/3), exactly as stacks/gemma4-charrp/README.md warns. They do NOT
# fix forced (named) tool_choice — that failure is independent of the reasoning parser
# (measured 1/9 with it, 3/9 without): the model free-writes Gemma native call syntax
# instead of grammar-constrained JSON. See README for the forced-tool_choice status.
- --reasoning-parser
- gemma4
- --default-chat-template-kwargs
- '{"enable_thinking": false}'
# tool_choice:"none" trap (measured 2026-09-08): without this flag vLLM still renders the
# tools into the prompt, the model emits a tool call anyway, and because tool parsing is
# off for tool_choice=none the reply comes back with content=null AND tool_calls=null —
# an empty turn. The flag drops the tools from the prompt so the model answers in prose.
- --exclude-tools-when-tool-choice-none
- --chat-template
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16/chat_template.jinja}
- --max-model-len
- "${ERP_MAX_MODEL_LEN:-262144}"
# KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is
# SHARED with seven other containers (charrp, scriberr, embed, rerank, coder,
# reward), so a ratio silently yields a different cache depending on what else
# is resident at start time; an explicit figure is reproducible and is what the
# concurrency ceiling is actually computed from.
#
# ⚠ KV-per-token is NOT a constant you may carry between context settings.
# Measured on THIS build (2026-09-09): 63.2 KiB/token at max-model-len 32768,
# 16.6 KiB/token at 262144 — a 3.8x difference, because Gemma-4 is hybrid
# attention (sliding_window 1024 on most layers) and the sliding layers stop
# scaling with the context while the full-attention layers keep going.
# Extrapolating the 32k figure to 256k under-counted the achievable cache by
# 3.8x and would have sized this seat wrong. Re-measure per configuration.
#
# 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is
# the operator's sizing rule (KV = 2x max context, 2026-09-09).
#
# This figure SURVIVED the 2026-09-10 Pfish-6 -> MeroMero-A4B swap unchanged, and that
# is not luck: the two are the same architecture field for field (30 layers, kv 8,
# head_dim 256, sliding_window 1024, 25 sliding / 5 full, 128 experts top-8), so the
# KV-per-token is the same number. Confirmed by reading 534,649 tokens / 2.04x back out
# of the new engine's log rather than assuming the pinning carried.
- --kv-cache-memory
- "${ERP_KV_CACHE_MEMORY:-9114000000}"
- --max-num-seqs
- "${ERP_MAX_NUM_SEQS:-32}"
- --gpu-memory-utilization
- "${ERP_GPU_MEM_UTIL:-0.35}"
- --kv-cache-dtype
- fp8
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
# do not apply — vLLM's "your GPU does not have native support for FP4"
# warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
# already settled against for this seat. Marlin is the correct kernel here.
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
# init — "does not support the deployment configuration since kernel does not
# support current device cuda". Marlin is the only viable NVFP4 MoE kernel
# here, so `auto` is already optimal and this knob exists to document that
# rather than to be turned.
#
# THE REAL LEVER WAS CONCURRENCY. max-num-seqs was 8, which capped the seat
# far below its KV pool (534,649 tokens). Measured on this build 2026-09-09,
# 3 rounds each, 300-token generations:
# n=1 219 tok/s (unchanged by the seq limit)
# n=4 676 tok/s aggregate
# n=8 1269 tok/s aggregate <- was NOT REACHABLE at max-num-seqs 8
# n=16 2170 tok/s aggregate <- 3.2x the old ceiling
# Single-stream latency is identical, so the raise costs nothing and lifts
# the ceiling ~3.2x. CUDA-graph capture sizes followed automatically to 64.
- --kernel-config
- '{"moe_backend": "${ERP_MOE_BACKEND:-auto}"}'
- --trust-remote-code
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${ERP_GPU_ID:-1}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 600s
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=char-rp-fast (MeroMero 26B-A4B, NVFP4A16 MoE)
- homepage.icon=mdi-fire
- homepage.description=MeroMero A4B abliterated RP seat, NVFP4A16 weight-only, vision intact (ana-ml2 GPU1)
- homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
networks:
tnet:
name: traefik-net
external: true
Reference in New Issue View Git Blame Copy Permalink