diff --git a/docs/runbooks/gx10-run-07.md b/docs/runbooks/gx10-run-07.md index 42acdce..018c4ea 100644 --- a/docs/runbooks/gx10-run-07.md +++ b/docs/runbooks/gx10-run-07.md @@ -230,3 +230,49 @@ result; it makes a bad one moot.** Re-testing the opening-split idea is a fresh run on a clean base, not a re-read of this one — and it is the operator's call, not a default. + +## Postscript — run 7 retired, Pfish-6 is the standing seat (2026-09-09) + +Operator ruling: *"declare run 6 as Pfish-6 … we're gonna stay on 6 for now."* +Run 7's gate failure was a **detector bug** (the adjective "minor" in a HARD rule, +fixed `cc42d76`), but run 7 was independently a poor run and is not returning. + +`Pfish-6` = `erp-tune-v6-nvfp4a16`, served under that name on **both** boxes at +262,144 context for a head-to-head. + +### Head-to-head, same artifact, same context, 3 rounds each + +| | ana-ml2 (RTX PRO 6000 Blackwell) | pfi-gx10 (GB10) | +|---|---|---| +| n=1 | **219 tok/s** | 53.6 tok/s | +| n=4 aggregate | **676 tok/s** | 175 tok/s | +| KV pool | 534,649 tok (2.04× ctx) | 4,608,942 tok (17.6× ctx) | + +⚠ **ana-ml2 is ~4.1× FASTER, not slower.** The expectation going in was the +reverse. Decode is memory-bandwidth-bound and the discrete card's GDDR7 has +several times the bandwidth of the GX10's unified LPDDR5X; the GX10's advantage is +capacity (a 4.6M-token KV pool) and that is not what single-stream decode uses. + +### Kernel: Marlin is correct here, and this was PROVEN not assumed + +vLLM warns *"your GPU does not have native support for FP4 computation"* on +ana-ml2, which is **misleading — sm_120 does have native FP4**. NVFP4A16 is +weight-only: activations stay 16-bit, so no FP4 GEMM is issued. Forcing +`moe_backend=flashinfer_cutedsl` fails at engine init with *"does not support the +deployment configuration since kernel does not support current device cuda"*. +`auto` → MARLIN is already optimal for this scheme. A W4A4 build would use the +FP4 units, at the fidelity cost already settled against for this seat. + +### The real lever was concurrency + +`max-num-seqs` was **8**, capping the seat far below its KV pool. Raised to 32 +(CUDA-graph capture sizes followed to 64). Measured on ana-ml2: + +| streams | aggregate tok/s | +|---|---| +| 1 | 219 | +| 4 | 676 | +| 8 | **1,269** — unreachable at the old cap | +| 16 | **2,170** — 3.2× the old ceiling | + +Single-stream latency is **unchanged**, so the raise costs nothing. diff --git a/stacks/erp-seat/compose.yaml b/stacks/erp-seat/compose.yaml index bf41c67..83c6088 100644 --- a/stacks/erp-seat/compose.yaml +++ b/stacks/erp-seat/compose.yaml @@ -1,5 +1,10 @@ -# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of the latest gated ERP LoRA merge -# (run 7 = jenerallee78 ARA-abliterated Gemma-4-26B-A4B + R47 SFT + the opening-split slot), served under its TRUE name. +# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of **Pfish-6**, the run-6 LoRA +# merge on the jenerallee78 ARA abliteration, served under that name. +# +# ⚠ RUN 7 IS RETIRED (operator ruling 2026-09-09): "we're gonna stay on 6 for now". Run 7's +# gate failure turned out to be a DETECTOR BUG (the adjective "minor" in a HARD rule, fixed +# cc42d76 in brokkr-smithy) — but run 7 was independently a poor run (primary FLAT +2, both +# diversity families reduced, long-context coherence 1.0 -> 0.875). Run 6 is the standing seat. # Routing aliases (e.g. LiteLLM `trial`) are the operator's call and live in the gateway, not here. # # Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this @@ -23,11 +28,11 @@ services: environment: - VLLM_API_KEY=${API_KEY:-} command: - - ${ERP_MODEL:-/tank/aimodels/erp-tune-v7-nvfp4a16} + - ${ERP_MODEL:-/tank/aimodels/erp-tune-v6-nvfp4a16} - --quantization - compressed-tensors - --served-model-name - - ${ERP_SERVED_NAME:-erp-tune-v7-nvfp4a16} + - ${ERP_SERVED_NAME:-Pfish-6} - --tool-call-parser - gemma4 - --enable-auto-tool-choice @@ -47,15 +52,56 @@ services: # an empty turn. The flag drops the tools from the prompt so the model answers in prose. - --exclude-tools-when-tool-choice-none - --chat-template - - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v7-nvfp4a16/chat_template.jinja} + - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja} - --max-model-len - - "${ERP_MAX_MODEL_LEN:-32768}" + - "${ERP_MAX_MODEL_LEN:-262144}" + # KV pool pinned in BYTES, not inferred from the utilization ratio. GPU1 is + # SHARED with seven other containers (charrp, scriberr, embed, rerank, coder, + # reward), so a ratio silently yields a different cache depending on what else + # is resident at start time; an explicit figure is reproducible and is what the + # concurrency ceiling is actually computed from. + # + # ⚠ KV-per-token is NOT a constant you may carry between context settings. + # Measured on THIS build (2026-09-09): 63.2 KiB/token at max-model-len 32768, + # 16.6 KiB/token at 262144 — a 3.8x difference, because Gemma-4 is hybrid + # attention (sliding_window 1024 on most layers) and the sliding layers stop + # scaling with the context while the full-attention layers keep going. + # Extrapolating the 32k figure to 256k under-counted the achievable cache by + # 3.8x and would have sized this seat wrong. Re-measure per configuration. + # + # 8.49 GiB -> 534,649 tokens -> 2.04x a full 262,144-token context, which is + # the operator's sizing rule (KV = 2x max context, 2026-09-09). + - --kv-cache-memory + - "${ERP_KV_CACHE_MEMORY:-9114000000}" - --max-num-seqs - - "${ERP_MAX_NUM_SEQS:-8}" + - "${ERP_MAX_NUM_SEQS:-32}" - --gpu-memory-utilization - "${ERP_GPU_MEM_UTIL:-0.35}" - --kv-cache-dtype - fp8 + # SPEED (2026-09-09). ⚠ NVFP4A16 is WEIGHT-ONLY: activations stay 16-bit, so + # there is no FP4 GEMM to issue and the FlashInfer/CUTLASS NVFP4 MoE kernels + # do not apply — vLLM's "your GPU does not have native support for FP4" + # warning is about the SCHEME, not the card. ana-ml2 is sm_120 and DOES have + # native FP4; a W4A4 build would use it, at a fidelity cost the operator + # already settled against for this seat. Marlin is the correct kernel here. + # PROVEN, not assumed: forcing moe_backend=flashinfer_cutedsl fails at engine + # init — "does not support the deployment configuration since kernel does not + # support current device cuda". Marlin is the only viable NVFP4 MoE kernel + # here, so `auto` is already optimal and this knob exists to document that + # rather than to be turned. + # + # THE REAL LEVER WAS CONCURRENCY. max-num-seqs was 8, which capped the seat + # far below its KV pool (534,649 tokens). Measured on this build 2026-09-09, + # 3 rounds each, 300-token generations: + # n=1 219 tok/s (unchanged by the seq limit) + # n=4 676 tok/s aggregate + # n=8 1269 tok/s aggregate <- was NOT REACHABLE at max-num-seqs 8 + # n=16 2170 tok/s aggregate <- 3.2x the old ceiling + # Single-stream latency is identical, so the raise costs nothing and lifts + # the ceiling ~3.2x. CUDA-graph capture sizes followed automatically to 64. + - --kernel-config + - '{"moe_backend": "${ERP_MOE_BACKEND:-auto}"}' - --trust-remote-code deploy: resources: @@ -76,9 +122,9 @@ services: - tnet labels: - homepage.group=AI - Inference - - homepage.name=erp-tune-v7 (Gemma-4 26B-A4B ARA, NVFP4A16) + - homepage.name=Pfish-6 (Gemma-4 26B-A4B ARA, NVFP4A16) - homepage.icon=mdi-fire - - homepage.description=ERP-seat run-7 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1) + - homepage.description=Pfish-6 — the run-6 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1) - homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs networks: diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 4294b8f..93cdfc1 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -882,30 +882,21 @@ model_list: # # Same-site: seat and gateway are both at Anaheim (local hop, no mesh crossing). - # ⚠ REMOVED 2026-09-09 17:5x PT — run 7 FAILED brokkr's CSAM safety gate - # (detector fired on the tuned arm, fail-closed). The ana-ml2:8021 seat serving - # the NVFP4 quant of that tune was stopped at 17:43; this alias is commented - # out rather than left pointing at a dead port because ALIAS-PRESENT-WITH- - # BACKEND-DOWN IS A DIFFERENT AND WORSE STATE THAN ALIAS-REMOVED: it re-arms - # silently the moment anything is served on that port again, and the next - # thing served there would inherit a fleet-reachable name without anyone - # deciding it should. (brokkr-smithy-dev flagged the state; the reasoning is - # the same writer/reader-contract family as ops-lessons §11.) + # Pfish-6 — the STANDING seat as of 2026-09-09. NVFP4A16 quant of the run-6 + # LoRA merge on the jenerallee78 ARA abliteration, on ana-ml2 GPU1 (:8021). + # Operator ruling: "declare run 6 as Pfish-6 ... we're gonna stay on 6 for now." # - # The quant is rejected BY PROVENANCE — it descends from an artifact that - # failed. Its OWN behaviour on the flagged axis is UNMEASURED and nothing here - # claims otherwise. + # REPLACES the `trial` alias, which is retired with run 7. Run 7's CSAM gate + # failure turned out to be a DETECTOR BUG (the adjective "minor" matching a + # HARD rule — fixed cc42d76 in brokkr-smithy), but run 7 was independently a + # poor run and is not coming back. # - # Commented, not deleted, so restoring is uncommenting rather than - # reconstructing. Disposition of the artifact and of this alias is the - # OPERATOR'S. If it returns, erp-tune-v6-nvfp4a16 is still on disk and passed - # by his own adjudication — the obvious rollback target. - # - # - model_name: trial - # litellm_params: - # model: hosted_vllm/erp-tune-v7-nvfp4a16 - # api_base: http://10.250.50.54:8021/v1 - # api_key: os.environ/VLLM_API_KEY + # Served under its TRUE name. This is now a named seat, not a trial. + - model_name: Pfish-6 + litellm_params: + model: hosted_vllm/Pfish-6 + api_base: http://10.250.50.54:8021/v1 + api_key: os.environ/VLLM_API_KEY general_settings: master_key: os.environ/LITELLM_MASTER_KEY