vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b8dbe71a1ca26f32992ce01c868877dbad9f52ad
esh-pfi-infrastructure/stacks/erp-seat
T
History
vh b8956b58fa feat(erp-seat): Pfish-6 replaces run 7 — run-6 NVFP4 quant on both boxes, max-num-seqs 8->32 
Operator ruling: run 6 is declared Pfish-6 and is the standing seat; run 7 is
retired (its gate failure was the detector bug fixed in cc42d76, but the run was
independently poor).

- served under its TRUE name Pfish-6; LiteLLM alias trial -> Pfish-6
- ana-ml2 :8021 and pfi-gx10 :8098, both at 262,144 ctx, same artifact
- SPEED: moe_backend=flashinfer_cutedsl PROVEN unusable here (engine init fails,
  'kernel does not support current device'); Marlin is correct for a weight-only
  scheme, and vLLM's 'no native FP4' warning is about the scheme not the card
- the real lever was max-num-seqs 8 -> 32: n=8 1269 tok/s and n=16 2170 tok/s,
  3.2x the old ceiling, with single-stream latency unchanged
- head-to-head measured: ana-ml2 is 4.1x FASTER than the GX10, not slower
2026-09-09 19:16:36 -07:00
..
.env.example
feat(erp-tune): run 7 launched on pfi-gx10 (opening-split slot + mask union) — config, launcher, builder, runbook; erp-seat: pin vLLM nightly 311b3513 (forced tool_choice 1/9 -> 6/9, round trip clean), README on the gemma4-parser design limit
2026-09-08 23:31:09 -07:00
compose.yaml
feat(erp-seat): Pfish-6 replaces run 7 — run-6 NVFP4 quant on both boxes, max-num-seqs 8->32
2026-09-09 19:16:36 -07:00
README.md
feat(erp-seat): run 7 quantized to NVFP4A16 and serving as the trial seat on ana-ml2
2026-09-09 15:30:15 -07:00

README.md

erp-seat — ERP-tune seat on ana-ml2 (GPU1, :8021)

Serves the latest gated ERP LoRA merge as an NVFP4A16 (weight-only) compressed-tensors checkpoint so the GX10 is free to train the next run.

Current occupant: run 7 — erp-tune-v7-nvfp4a16 = merged-run07 (jenerallee78 ARA-abliterated Gemma-4-26B-A4B-it, index 33c59654…, + R47 SFT r7 = r6 plus the opening-split slot and its companion loss mask), quantized by services/erp-seat-quant/ on 2026-09-09. 49 GiB bf16 → 16 GiB NVFP4A16. Runbook docs/runbooks/gx10-run-07.md.

Previous: run 6 (erp-tune-v6-nvfp4a16, 2026-09-08). Its artifact is still on /tank/aimodels/ and the pre-swap host env is at /tmp/erp-seat-env.v6.bak on ana-ml2, so a rollback is an .env flip plus docker compose up -d.

  • True name only. --served-model-name erp-tune-v7-nvfp4a16. Gateway aliases (trial) are set in LiteLLM on the operator's word, never here (no silent substitution — the bf16 arm on the GX10 and this NVFP4 arm are different artifacts).
  • Recipe = stacks/gemma4-charrp (same arch + format, proven on this box): gemma4 tool and reasoning parsers, enable_thinking pinned false, the model's own stock template (ae53464b…, the one it trained through). Without the reasoning parser the post-tool turn leaks <|channel> markers; without the kwargs pin all prose lands in reasoning_content.
  • tool_choice: "none" trap (measured 2026-09-08, fixed with --exclude-tools-when-tool-choice-none). Without the flag vLLM still renders the tools into the prompt, the model emits a tool call anyway, and because parsing is off for none the reply is content: null, tool_calls: null — an empty turn, 3/3 reproductions. With the flag the tools are dropped from the prompt and the model answers in prose (3/3). The rest of the matrix (auto / required / named / parallel / nested schema / empty tools: [] / streaming / tool-result round trip) was green before and after. stacks/gemma4-charrp has the same exposure and does NOT carry the flag yet.
  • Forced tool_choice (named / required) is prompt-driven on EVERY Gemma-4 seat, not grammar-enforced — by vLLM design. vllm/tool_parsers/gemma4_engine_tool_parser.py sets supports_required_and_named = False and its adjust_request deliberately skips the structured-output JSON for required/named so the model can emit its native <|tool_call>call:… syntax. A tune that weakened that syntax (this ERP tune) therefore honours forced calls only sometimes. Measured 2026-09-08, 3 conversations × 3 turns, real system prompt: v0.26.0 1/9; nightly 311b3513 (v0.27.2rc1, the gen seat's image) 6/9 and the tool-result round trip stays clean 3/3 — so the seat runs the nightly. tts-dev measured 0/18 on v0.26.0 with gen 18/18 as the positive control, and response_format: json_schema (guided decoding) 18/18 on this seat — that is the deterministic path for a forced call; tool_choice: auto works normally. A parser plugin that re-enables guided JSON would also need JSON extraction in the engine-parser path; not attempted.
  • GPU1 is shared — check real usage (nvidia-smi --query-compute-apps=pid,used_memory) before raising ERP_GPU_MEM_UTIL; the flag sizes KV, not CUDA context.
  • Rollback / next run: point ERP_MODEL + ERP_SERVED_NAME at the next quant dir, keep the previous on disk. Deploy with scripts/deploy-stack.sh ana-ml2 erp-seat.
Reference in New Issue View Git Blame Copy Permalink