feat(erp-seat): NVFP4A16 quant pipeline for the Gemma-4 26B-A4B MoE ERP tune + ana-ml2 GPU1 serve stack

- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15), asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP long-session fidelity, post-steps restore processor configs + template and reset the tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time - services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor) - stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
2026-09-08 21:55:28 -07:00
parent 1f95728728
commit 911ff20356
5 changed files with 269 additions and 0 deletions
@@ -0,0 +1,12 @@
+# erp-seat — ana-ml2 GPU1. Real .env lives on the host at /opt/docker/compose/erp-seat/.env.
+ERP_IMAGE=vllm/vllm-openai:v0.26.0
+ERP_MODEL=/tank/aimodels/erp-tune-v6-nvfp4a16
+ERP_SERVED_NAME=erp-tune-v6-nvfp4a16
+ERP_CHAT_TEMPLATE=/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja
+ERP_PORT=8021
+ERP_GPU_ID=1
+# 0.35 x 97.9 GiB = 34 GiB. GPU1 had ~47 GiB free on 2026-09-08 (scriberr/embed/rerank/coder/reward resident).
+ERP_GPU_MEM_UTIL=0.35
+ERP_MAX_MODEL_LEN=32768
+ERP_MAX_NUM_SEQS=8
+API_KEY=
@@ -0,0 +1,18 @@
+# erp-seat — ERP-tune seat on ana-ml2 (GPU1, `:8021`)
+
+Serves the latest gated ERP LoRA merge as an **NVFP4A16** (weight-only) compressed-tensors
+checkpoint so the GX10 is free to train the next run. First occupant: **run 6** —
+`erp-tune-v6-nvfp4a16` = merged-run06 (jenerallee78 ARA-abliterated Gemma-4-26B-A4B-it, index
+`33c59654…`, + R47 SFT r6) quantized by `services/erp-seat-quant/`.
+
+- **True name only.** `--served-model-name erp-tune-v6-nvfp4a16`. Gateway aliases (`trial`) are
+  set in LiteLLM on the operator's word, never here (no silent substitution — the bf16 arm on the
+  GX10 and this NVFP4 arm are different artifacts).
+- **Recipe** = `stacks/gemma4-charrp` (same arch + format, proven on this box): `gemma4` tool
+  and reasoning parsers, `enable_thinking` pinned false, the model's own stock template
+  (`ae53464b…`, the one it trained through). Without the reasoning parser the post-tool turn leaks
+  `<|channel>` markers; without the kwargs pin all prose lands in `reasoning_content`.
+- **GPU1 is shared** — check real usage (`nvidia-smi --query-compute-apps=pid,used_memory`) before
+  raising `ERP_GPU_MEM_UTIL`; the flag sizes KV, not CUDA context.
+- **Rollback / next run:** point `ERP_MODEL` + `ERP_SERVED_NAME` at the next quant dir, keep the
+  previous on disk. Deploy with `scripts/deploy-stack.sh ana-ml2 erp-seat`.
@@ -0,0 +1,76 @@
+# erp-seat — the ERP-tune seat on ana-ml2 GPU1: NVFP4A16 quant of the latest gated ERP LoRA merge
+# (run 6 = jenerallee78 ARA-abliterated Gemma-4-26B-A4B + R47 SFT), served under its TRUE name.
+# Routing aliases (e.g. LiteLLM `trial`) are the operator's call and live in the gateway, not here.
+#
+# Serve recipe copied from stacks/gemma4-charrp (same architecture + quant format, proven on this
+# box): gemma4 tool + reasoning parsers, enable_thinking pinned false, model's own stock template.
+# GPU1 is SHARED (charrp-MoE moved? no — scriberr, embed, rerank, coder, reward live there):
+# ~47 GiB was free on 2026-09-08; 0.35 x 97.9 GiB = 34 GiB keeps ~13 GiB of real margin.
+# Quant pipeline: services/erp-seat-quant/. Tunables in .env.
+
+name: erp-seat
+
+services:
+  vllm-erp-seat:
+    image: ${ERP_IMAGE:-vllm/vllm-openai:v0.26.0}
+    container_name: ${ERP_CONTAINER:-vllm-erp-seat}
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${ERP_PORT:-8021}:8000"
+    volumes:
+      - /tank/aimodels:/tank/aimodels
+    environment:
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${ERP_MODEL:-/tank/aimodels/erp-tune-v6-nvfp4a16}
+      - --quantization
+      - compressed-tensors
+      - --served-model-name
+      - ${ERP_SERVED_NAME:-erp-tune-v6-nvfp4a16}
+      - --tool-call-parser
+      - gemma4
+      - --enable-auto-tool-choice
+      - --reasoning-parser
+      - gemma4
+      - --default-chat-template-kwargs
+      - '{"enable_thinking": false}'
+      - --chat-template
+      - ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja}
+      - --max-model-len
+      - "${ERP_MAX_MODEL_LEN:-32768}"
+      - --max-num-seqs
+      - "${ERP_MAX_NUM_SEQS:-8}"
+      - --gpu-memory-utilization
+      - "${ERP_GPU_MEM_UTIL:-0.35}"
+      - --kv-cache-dtype
+      - fp8
+      - --trust-remote-code
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids:
+                - "${ERP_GPU_ID:-1}"
+              capabilities:
+                - gpu
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 600s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Inference
+      - homepage.name=erp-tune-v6 (Gemma-4 26B-A4B ARA, NVFP4A16)
+      - homepage.icon=mdi-fire
+      - homepage.description=ERP-seat run-6 LoRA merge on the jenerallee78 abliteration, NVFP4A16 MoE (ana-ml2 GPU1)
+      - homepage.href=http://10.250.50.54:${ERP_PORT:-8021}/docs
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true