feat(erp-seat): NVFP4A16 quant pipeline for the Gemma-4 26B-A4B MoE ERP tune + ana-ml2 GPU1 serve stack
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15), asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP long-session fidelity, post-steps restore processor configs + template and reset the tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time - services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor) - stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
This commit is contained in:
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# NVFP4A16 quant of the ERP run-6 merged model on ana-ml2 GPU1 (co-resident with the GPU1 seats;
|
||||
# CPU-resident load, per-layer onload). Detached container; watch with `docker logs -f erp-v6-quant`.
|
||||
# NOTE: no PYTORCH_CUDA_ALLOC_CONF=expandable_segments (playbook §3.10).
|
||||
set -euo pipefail
|
||||
WORK=/tank/aimodels/erp-tune-v6-quant-work
|
||||
SRC="${1:-/tank/aimodels/erp-tune-v6-bf16}"
|
||||
OUT="${2:-/tank/aimodels/erp-tune-v6-nvfp4a16}"
|
||||
MODE="${3:-full}" # full | dry-run
|
||||
EXTRA=""; [ "$MODE" = "dry-run" ] && EXTRA="--dry-run"
|
||||
NAME=erp-v6-quant; [ "$MODE" = "dry-run" ] && NAME=erp-v6-quant-dry
|
||||
docker rm -f "$NAME" 2>/dev/null || true
|
||||
docker run -d --name "$NAME" --gpus '"device=1"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels \
|
||||
--entrypoint python3 vllm-llmcompressor:latest \
|
||||
"$WORK/quant_nvfp4a16_gemma4_moe.py" --model "$SRC" --out "$OUT" \
|
||||
--num-samples "${NUM_SAMPLES:-256}" --seqlen "${SEQLEN:-8192}" $EXTRA
|
||||
echo "launched $NAME: $(docker ps --filter name=$NAME --format '{{.Status}}')"
|
||||
Reference in New Issue
Block a user