feat(erp-seat): Pfish-6 replaces run 7 — run-6 NVFP4 quant on both boxes, max-num-seqs 8->32
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is retired (its gate failure was the detector bug fixed in cc42d76, but the run was independently poor). - served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6 - ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact - SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails, 'kernel does not support current device'); Marlin is correct for a weight-only scheme, and vLLM's 'no native FP4' warning is about the scheme not the card - the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s, 3.2x the old ceiling, with single-stream latency unchanged - head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
This commit is contained in:
@@ -230,3 +230,49 @@ result; it makes a bad one moot.**
|
||||
|
||||
Re-testing the opening-split idea is a fresh run on a clean base, not a re-read of
|
||||
this one — and it is the operator's call, not a default.
|
||||
|
||||
## Postscript — run 7 retired, Pfish-6 is the standing seat (2026-09-09)
|
||||
|
||||
Operator ruling: *"declare run 6 as Pfish-6 … we're gonna stay on 6 for now."*
|
||||
Run 7's gate failure was a **detector bug** (the adjective "minor" in a HARD rule,
|
||||
fixed `cc42d76`), but run 7 was independently a poor run and is not returning.
|
||||
|
||||
`Pfish-6` = `erp-tune-v6-nvfp4a16`, served under that name on **both** boxes at
|
||||
262,144 context for a head-to-head.
|
||||
|
||||
### Head-to-head, same artifact, same context, 3 rounds each
|
||||
|
||||
| | ana-ml2 (RTX PRO 6000 Blackwell) | pfi-gx10 (GB10) |
|
||||
|---|---|---|
|
||||
| n=1 | **219 tok/s** | 53.6 tok/s |
|
||||
| n=4 aggregate | **676 tok/s** | 175 tok/s |
|
||||
| KV pool | 534,649 tok (2.04× ctx) | 4,608,942 tok (17.6× ctx) |
|
||||
|
||||
⚠ **ana-ml2 is ~4.1× FASTER, not slower.** The expectation going in was the
|
||||
reverse. Decode is memory-bandwidth-bound and the discrete card's GDDR7 has
|
||||
several times the bandwidth of the GX10's unified LPDDR5X; the GX10's advantage is
|
||||
capacity (a 4.6M-token KV pool) and that is not what single-stream decode uses.
|
||||
|
||||
### Kernel: Marlin is correct here, and this was PROVEN not assumed
|
||||
|
||||
vLLM warns *"your GPU does not have native support for FP4 computation"* on
|
||||
ana-ml2, which is **misleading — sm_120 does have native FP4**. NVFP4A16 is
|
||||
weight-only: activations stay 16-bit, so no FP4 GEMM is issued. Forcing
|
||||
`moe_backend=flashinfer_cutedsl` fails at engine init with *"does not support the
|
||||
deployment configuration since kernel does not support current device cuda"*.
|
||||
`auto` → MARLIN is already optimal for this scheme. A W4A4 build would use the
|
||||
FP4 units, at the fidelity cost already settled against for this seat.
|
||||
|
||||
### The real lever was concurrency
|
||||
|
||||
`max-num-seqs` was **8**, capping the seat far below its KV pool. Raised to 32
|
||||
(CUDA-graph capture sizes followed to 64). Measured on ana-ml2:
|
||||
|
||||
| streams | aggregate tok/s |
|
||||
|---|---|
|
||||
| 1 | 219 |
|
||||
| 4 | 676 |
|
||||
| 8 | **1,269** — unreachable at the old cap |
|
||||
| 16 | **2,170** — 3.2× the old ceiling |
|
||||
|
||||
Single-stream latency is **unchanged**, so the raise costs nothing.
|
||||
|
||||
Reference in New Issue
Block a user