vh
911ff20356
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15), asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP long-session fidelity, post-steps restore processor configs + template and reset the tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time - services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor) - stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
19 lines
1.0 KiB
Bash
Executable File
19 lines
1.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# NVFP4A16 quant of the ERP run-6 merged model on ana-ml2 GPU1 (co-resident with the GPU1 seats;
|
|
# CPU-resident load, per-layer onload). Detached container; watch with `docker logs -f erp-v6-quant`.
|
|
# NOTE: no PYTORCH_CUDA_ALLOC_CONF=expandable_segments (playbook §3.10).
|
|
set -euo pipefail
|
|
WORK=/tank/aimodels/erp-tune-v6-quant-work
|
|
SRC="${1:-/tank/aimodels/erp-tune-v6-bf16}"
|
|
OUT="${2:-/tank/aimodels/erp-tune-v6-nvfp4a16}"
|
|
MODE="${3:-full}" # full | dry-run
|
|
EXTRA=""; [ "$MODE" = "dry-run" ] && EXTRA="--dry-run"
|
|
NAME=erp-v6-quant; [ "$MODE" = "dry-run" ] && NAME=erp-v6-quant-dry
|
|
docker rm -f "$NAME" 2>/dev/null || true
|
|
docker run -d --name "$NAME" --gpus '"device=1"' --ipc host \
|
|
-v /tank/aimodels:/tank/aimodels \
|
|
--entrypoint python3 vllm-llmcompressor:latest \
|
|
"$WORK/quant_nvfp4a16_gemma4_moe.py" --model "$SRC" --out "$OUT" \
|
|
--num-samples "${NUM_SAMPLES:-256}" --seqlen "${SEQLEN:-8192}" $EXTRA
|
|
echo "launched $NAME: $(docker ps --filter name=$NAME --format '{{.Status}}')"
|