feat(erp-seat): Pfish-6 replaces run 7 — run-6 NVFP4 quant on both boxes, max-num-seqs 8->32
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is retired (its gate failure was the detector bug fixed in cc42d76, but the run was independently poor). - served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6 - ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact - SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails, 'kernel does not support current device'); Marlin is correct for a weight-only scheme, and vLLM's 'no native FP4' warning is about the scheme not the card - the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s, 3.2x the old ceiling, with single-stream latency unchanged - head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
This commit is contained in:
@@ -1,5 +1,10 @@
|
||||
# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of the latest gated ERP LoRA merge
|
||||
# (run 7 = jenerallee78 ARA-abliterated Gemma-4-26B-A4B + R47 SFT + the opening-split slot), served under its TRUE name.
|
||||
# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of **Pfish-6**, the run-6 LoRA
|
||||
# merge on the jenerallee78 ARA abliteration, served under that name.
|
||||
#
|
||||
# ⚠ RUN 7 IS RETIRED (operator ruling 2026-09-09): "we're gonna stay on 6 for now". Run 7's
|
||||
# gate failure turned out to be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed
|
||||
# cc42d76 in brokkr-smithy) — but run 7 was independently a poor run (primary FLAT +2, both
|
||||
# diversity families reduced, long-context coherence 1.0 -> 0.875). Run 6 is the standing seat.
|
||||
# Routing aliases (e.g. LiteLLM `trial`) are the operator's call and live in the gateway, not here.
|
||||
#
|
||||
# Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this
|
||||
@@ -23,11 +28,11 @@ services:
|
||||
environment:
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${ERP_MODEL:-/tank/aimodels/erp-tune-v7-nvfp4a16}
|
||||
- ${ERP_MODEL:-/tank/aimodels/erp-tune-v6-nvfp4a16}
|
||||
- --quantization
|
||||
- compressed-tensors
|
||||
- --served-model-name
|
||||
- ${ERP_SERVED_NAME:-erp-tune-v7-nvfp4a16}
|
||||
- ${ERP_SERVED_NAME:-Pfish-6}
|
||||
- --tool-call-parser
|
||||
- gemma4
|
||||
- --enable-auto-tool-choice
|
||||
@@ -47,15 +52,56 @@ services:
|
||||
# an empty turn. The flag drops the tools from the prompt so the model answers in prose.
|
||||
- --exclude-tools-when-tool-choice-none
|
||||
- --chat-template
|
||||
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v7-nvfp4a16/chat_template.jinja}
|
||||
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja}
|
||||
- --max-model-len
|
||||
- "${ERP_MAX_MODEL_LEN:-32768}"
|
||||
- "${ERP_MAX_MODEL_LEN:-262144}"
|
||||
# KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is
|
||||
# SHARED with seven other containers (charrp, scriberr, embed, rerank, coder,
|
||||
# reward), so a ratio silently yields a different cache depending on what else
|
||||
# is resident at start time; an explicit figure is reproducible and is what the
|
||||
# concurrency ceiling is actually computed from.
|
||||
#
|
||||
# ⚠ KV-per-token is NOT a constant you may carry between context settings.
|
||||
# Measured on THIS build (2026-09-09): 63.2 KiB/token at max-model-len 32768,
|
||||
# 16.6 KiB/token at 262144 — a 3.8x difference, because Gemma-4 is hybrid
|
||||
# attention (sliding_window 1024 on most layers) and the sliding layers stop
|
||||
# scaling with the context while the full-attention layers keep going.
|
||||
# Extrapolating the 32k figure to 256k under-counted the achievable cache by
|
||||
# 3.8x and would have sized this seat wrong. Re-measure per configuration.
|
||||
#
|
||||
# 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is
|
||||
# the operator's sizing rule (KV = 2x max context, 2026-09-09).
|
||||
- --kv-cache-memory
|
||||
- "${ERP_KV_CACHE_MEMORY:-9114000000}"
|
||||
- --max-num-seqs
|
||||
- "${ERP_MAX_NUM_SEQS:-8}"
|
||||
- "${ERP_MAX_NUM_SEQS:-32}"
|
||||
- --gpu-memory-utilization
|
||||
- "${ERP_GPU_MEM_UTIL:-0.35}"
|
||||
- --kv-cache-dtype
|
||||
- fp8
|
||||
# SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so
|
||||
# there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels
|
||||
# do not apply — vLLM's "your GPU does not have native support for FP4"
|
||||
# warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have
|
||||
# native FP4; a W4A4 build would use it, at a fidelity cost the operator
|
||||
# already settled against for this seat. Marlin is the correct kernel here.
|
||||
# PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine
|
||||
# init — "does not support the deployment configuration since kernel does not
|
||||
# support current device cuda". Marlin is the only viable NVFP4 MoE kernel
|
||||
# here, so `auto` is already optimal and this knob exists to document that
|
||||
# rather than to be turned.
|
||||
#
|
||||
# THE REAL LEVER WAS CONCURRENCY. max-num-seqs was 8, which capped the seat
|
||||
# far below its KV pool (534,649 tokens). Measured on this build 2026-09-09,
|
||||
# 3 rounds each, 300-token generations:
|
||||
# n=1 219 tok/s (unchanged by the seq limit)
|
||||
# n=4 676 tok/s aggregate
|
||||
# n=8 1269 tok/s aggregate <- was NOT REACHABLE at max-num-seqs 8
|
||||
# n=16 2170 tok/s aggregate <- 3.2x the old ceiling
|
||||
# Single-stream latency is identical, so the raise costs nothing and lifts
|
||||
# the ceiling ~3.2x. CUDA-graph capture sizes followed automatically to 64.
|
||||
- --kernel-config
|
||||
- '{"moe_backend": "${ERP_MOE_BACKEND:-auto}"}'
|
||||
- --trust-remote-code
|
||||
deploy:
|
||||
resources:
|
||||
@@ -76,9 +122,9 @@ services:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=erp-tune-v7 (Gemma-4 26B-A4B ARA, NVFP4A16)
|
||||
- homepage.name=Pfish-6 (Gemma-4 26B-A4B ARA, NVFP4A16)
|
||||
- homepage.icon=mdi-fire
|
||||
- homepage.description=ERP-seat run-7 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1)
|
||||
- homepage.description=Pfish-6 — the run-6 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1)
|
||||
- homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
|
||||
|
||||
networks:
|
||||
|
||||
@@ -882,30 +882,21 @@ model_list:
|
||||
#
|
||||
# Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing).
|
||||
|
||||
# ⚠ REMOVED 2026-09-09 17:5x PT — run 7 FAILED brokkr's CSAM safety gate
|
||||
# (detector fired on the tuned arm, fail-closed). The ana-ml2:8021 seat serving
|
||||
# the NVFP4 quant of that tune was stopped at 17:43; this alias is commented
|
||||
# out rather than left pointing at a dead port because ALIAS-PRESENT-WITH-
|
||||
# BACKEND-DOWN IS A DIFFERENT AND WORSE STATE THAN ALIAS-REMOVED: it re-arms
|
||||
# silently the moment anything is served on that port again, and the next
|
||||
# thing served there would inherit a fleet-reachable name without anyone
|
||||
# deciding it should. (brokkr-smithy-dev flagged the state; the reasoning is
|
||||
# the same writer/reader-contract family as ops-lessons §11.)
|
||||
# Pfish-6 — the STANDING seat as of 2026-09-09. NVFP4A16 quant of the run-6
|
||||
# LoRA merge on the jenerallee78 ARA abliteration, on ana-ml2 GPU1 (:8021).
|
||||
# Operator ruling: "declare run 6 as Pfish-6 ... we're gonna stay on 6 for now."
|
||||
#
|
||||
# The quant is rejected BY PROVENANCE — it descends from an artifact that
|
||||
# failed. Its OWN behaviour on the flagged axis is UNMEASURED and nothing here
|
||||
# claims otherwise.
|
||||
# REPLACES the `trial` alias, which is retired with run 7. Run 7's CSAM gate
|
||||
# failure turned out to be a DETECTOR BUG (the adjective "minor" matching a
|
||||
# HARD rule — fixed cc42d76 in brokkr-smithy), but run 7 was independently a
|
||||
# poor run and is not coming back.
|
||||
#
|
||||
# Commented, not deleted, so restoring is uncommenting rather than
|
||||
# reconstructing. Disposition of the artifact and of this alias is the
|
||||
# OPERATOR'S. If it returns, erp-tune-v6-nvfp4a16 is still on disk and passed
|
||||
# by his own adjudication — the obvious rollback target.
|
||||
#
|
||||
# - model_name: trial
|
||||
# litellm_params:
|
||||
# model: hosted_vllm/erp-tune-v7-nvfp4a16
|
||||
# api_base: http://10.250.50.54:8021/v1
|
||||
# api_key: os.environ/VLLM_API_KEY
|
||||
# Served under its TRUE name. This is now a named seat, not a trial.
|
||||
- model_name: Pfish-6
|
||||
litellm_params:
|
||||
model: hosted_vllm/Pfish-6
|
||||
api_base: http://10.250.50.54:8021/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
|
||||
Reference in New Issue
Block a user