vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a84f6b00df5d99236ce339e9f60f7d06e396d745
esh-pfi-infrastructure/stacks/erp-seat/README.md
T
vh c35f6510bf feat(fv-ml1): rebalance GPUs 0/1 by usage; contexts to published maximums; gen-small default output cap 
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from
GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native
maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768,
the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from
GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision.

Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against
7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps
spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's
32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the
Qwen cards warn static YaRN degrades short prompts.

gen-small gets --override-generation-config max_new_tokens 32768. It applies
only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9
runaway ~260k-token replies in 7 days; the longest normal reply was 6,357.

Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and
health/KV verification per seat); all nine seats healthy, gateway smoke
tests pass.
2026-10-05 22:51:17 -07:00

3.5 KiB
Raw Blame History

erp-seat — ERP-tune seat on fv-ml1 (ex ana-ml2; GPU0 since 2026-10-05, :8021)

Serves the latest gated ERP LoRA merge as an NVFP4A16 (weight-only) compressed-tensors checkpoint so the GX10 is free to train the next run.

Current occupant: run 7 — erp-tune-v7-nvfp4a16 = merged-run07 (jenerallee78 ARA-abliterated Gemma-4-26B-A4B-it, index 33c59654…, + R47 SFT r7 = r6 plus the opening-split slot and its companion loss mask), quantized by services/erp-seat-quant/ on 2026-09-09. 49 GiB bf16 → 16 GiB NVFP4A16. Runbook docs/runbooks/gx10-run-07.md.

Previous: run 6 (erp-tune-v6-nvfp4a16, 2026-09-08). Its artifact is still on /tank/aimodels/ and the pre-swap host env is at /tmp/erp-seat-env.v6.bak on ana-ml2, so a rollback is an .env flip plus docker compose up -d.

  • True name only. --served-model-name erp-tune-v7-nvfp4a16. Gateway aliases (trial) are set in LiteLLM on the operator's word, never here (no silent substitution — the bf16 arm on the GX10 and this NVFP4 arm are different artifacts).
  • Recipe = stacks/gemma4-charrp (same arch + format, proven on this box): gemma4 tool and reasoning parsers, enable_thinking pinned false, the model's own stock template (ae53464b…, the one it trained through). Without the reasoning parser the post-tool turn leaks <|channel> markers; without the kwargs pin all prose lands in reasoning_content.
  • tool_choice: "none" trap (measured 2026-09-08, fixed with --exclude-tools-when-tool-choice-none). Without the flag vLLM still renders the tools into the prompt, the model emits a tool call anyway, and because parsing is off for none the reply is content: null, tool_calls: null — an empty turn, 3/3 reproductions. With the flag the tools are dropped from the prompt and the model answers in prose (3/3). The rest of the matrix (auto / required / named / parallel / nested schema / empty tools: [] / streaming / tool-result round trip) was green before and after. stacks/gemma4-charrp has the same exposure and does NOT carry the flag yet.
  • Forced tool_choice (named / required) is prompt-driven on EVERY Gemma-4 seat, not grammar-enforced — by vLLM design. vllm/tool_parsers/gemma4_engine_tool_parser.py sets supports_required_and_named = False and its adjust_request deliberately skips the structured-output JSON for required/named so the model can emit its native <|tool_call>call:… syntax. A tune that weakened that syntax (this ERP tune) therefore honours forced calls only sometimes. Measured 2026-09-08, 3 conversations × 3 turns, real system prompt: v0.26.0 1/9; nightly 311b3513 (v0.27.2rc1, the gen seat's image) 6/9 and the tool-result round trip stays clean 3/3 — so the seat runs the nightly. tts-dev measured 0/18 on v0.26.0 with gen 18/18 as the positive control, and response_format: json_schema (guided decoding) 18/18 on this seat — that is the deterministic path for a forced call; tool_choice: auto works normally. A parser plugin that re-enables guided JSON would also need JSON extraction in the engine-parser path; not attempted.
  • GPU0 is shared (since 2026-10-05: cyberprev, voices, coder, parakeet-nemo; KV pinned 5.5 GiB = 346,436 tokens, 1.32×) — check real usage (nvidia-smi --query-compute-apps=pid,used_memory) before raising ERP_GPU_MEM_UTIL; the flag sizes KV, not CUDA context.
  • Rollback / next run: point ERP_MODEL + ERP_SERVED_NAME at the next quant dir, keep the previous on disk. Deploy with scripts/deploy-stack.sh ana-ml2 erp-seat.
Reference in New Issue View Git Blame Copy Permalink