vh
c35f6510bf
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768, the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision. Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against 7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's 32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the Qwen cards warn static YaRN degrades short prompts. gen-small gets --override-generation-config max_new_tokens 32768. It applies only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9 runaway ~260k-token replies in 7 days; the longest normal reply was 6,357. Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and health/KV verification per seat); all nine seats healthy, gateway smoke tests pass.
45 lines
3.5 KiB
Markdown
45 lines
3.5 KiB
Markdown
# erp-seat — ERP-tune seat on fv-ml1 (ex ana-ml2; GPU0 since 2026-10-05, `:8021`)
|
||
|
||
Serves the latest gated ERP LoRA merge as an **NVFP4A16** (weight-only) compressed-tensors
|
||
checkpoint so the GX10 is free to train the next run.
|
||
|
||
**Current occupant: run 7** — `erp-tune-v7-nvfp4a16` = merged-run07 (jenerallee78
|
||
ARA-abliterated Gemma-4-26B-A4B-it, index `33c59654…`, + R47 SFT r7 = r6 plus the opening-split
|
||
slot and its companion loss mask), quantized by `services/erp-seat-quant/` on 2026-09-09.
|
||
49 GiB bf16 → 16 GiB NVFP4A16. Runbook `docs/runbooks/gx10-run-07.md`.
|
||
|
||
*Previous: run 6 (`erp-tune-v6-nvfp4a16`, 2026-09-08). Its artifact is still on `/tank/aimodels/`
|
||
and the pre-swap host env is at `/tmp/erp-seat-env.v6.bak` on ana-ml2, so a rollback is an `.env`
|
||
flip plus `docker compose up -d`.*
|
||
|
||
- **True name only.** `--served-model-name erp-tune-v7-nvfp4a16`. Gateway aliases (`trial`) are
|
||
set in LiteLLM on the operator's word, never here (no silent substitution — the bf16 arm on the
|
||
GX10 and this NVFP4 arm are different artifacts).
|
||
- **Recipe** = `stacks/gemma4-charrp` (same arch + format, proven on this box): `gemma4` tool
|
||
and reasoning parsers, `enable_thinking` pinned false, the model's own stock template
|
||
(`ae53464b…`, the one it trained through). Without the reasoning parser the post-tool turn leaks
|
||
`<|channel>` markers; without the kwargs pin all prose lands in `reasoning_content`.
|
||
- **`tool_choice: "none"` trap (measured 2026-09-08, fixed with `--exclude-tools-when-tool-choice-none`).**
|
||
Without the flag vLLM still renders the tools into the prompt, the model emits a tool call
|
||
anyway, and because parsing is off for `none` the reply is `content: null, tool_calls: null`
|
||
— an empty turn, 3/3 reproductions. With the flag the tools are dropped from the prompt and
|
||
the model answers in prose (3/3). The rest of the matrix (auto / required / named / parallel /
|
||
nested schema / empty `tools: []` / streaming / tool-result round trip) was green before and
|
||
after. `stacks/gemma4-charrp` has the same exposure and does NOT carry the flag yet.
|
||
- **Forced tool_choice (named / `required`) is prompt-driven on EVERY Gemma-4 seat, not
|
||
grammar-enforced — by vLLM design.** `vllm/tool_parsers/gemma4_engine_tool_parser.py` sets
|
||
`supports_required_and_named = False` and its `adjust_request` deliberately skips the
|
||
structured-output JSON for required/named so the model can emit its native
|
||
`<|tool_call>call:…` syntax. A tune that weakened that syntax (this ERP tune) therefore
|
||
honours forced calls only sometimes. Measured 2026-09-08, 3 conversations × 3 turns, real
|
||
system prompt: v0.26.0 **1/9**; nightly `311b3513` (v0.27.2rc1, the `gen` seat's image) **6/9**
|
||
and the tool-result round trip stays clean 3/3 — so the seat runs the nightly. tts-dev measured
|
||
0/18 on v0.26.0 with `gen` 18/18 as the positive control, and **`response_format:
|
||
json_schema` (guided decoding) 18/18 on this seat** — that is the deterministic path for a
|
||
forced call; `tool_choice: auto` works normally. A parser plugin that re-enables guided JSON
|
||
would also need JSON extraction in the engine-parser path; not attempted.
|
||
- **GPU0 is shared** (since 2026-10-05: cyberprev, voices, coder, parakeet-nemo; KV pinned 5.5 GiB = 346,436 tokens, 1.32×) — check real usage (`nvidia-smi --query-compute-apps=pid,used_memory`) before
|
||
raising `ERP_GPU_MEM_UTIL`; the flag sizes KV, not CUDA context.
|
||
- **Rollback / next run:** point `ERP_MODEL` + `ERP_SERVED_NAME` at the next quant dir, keep the
|
||
previous on disk. Deploy with `scripts/deploy-stack.sh ana-ml2 erp-seat`.
|