feat(vllm): replace phi4-mini with Granite 4.1 8B summarizer + retune GPU 1

Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's
the new production summarizer/dreamer for nevermore.

- vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8,
  compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot.
- GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a
  5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18,
  freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and
  keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora).
- LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004);
  explicit entry shadows the '*' wildcard's llama-swap route.
- nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live.

Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok),
KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
This commit is contained in:
vh
2026-06-05 09:34:22 -07:00
parent 89a020a3cc
commit 34a43a0bc5
3 changed files with 67 additions and 52 deletions
+23 -17
View File
@@ -201,13 +201,13 @@ services:
# llama-swap granite-4-small pin. Generative chat model (OpenAI
# /v1/chat/completions), so NO --runner pooling. FP8 on RTX 6000 Ada
# (cc 8.9): near-lossless, ~1.2x, ~6 GB.
vllm-phi4:
vllm-granite:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-phi4
container_name: vllm-granite
restart: unless-stopped
ipc: host
ports:
- "${PHI4_PORT}:8000"
- "${GRANITE_PORT}:8000"
volumes:
- /tank/aimodels/huggingface:/hfcache
environment:
@@ -216,34 +216,40 @@ services:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
# Default = vLLM-native dynamic FP8 from the official BF16 weights (most
# robust on Ada). Alt = pre-quantized checkpoint: PHI4_MODEL=
# pytorch/Phi-4-mini-instruct-FP8 + PHI4_QUANT=torchao (see .env/README).
- ${PHI4_MODEL}
# Production summarizer (replaced phi4-mini 2026-06-05). Default = official
# IBM pre-quantized FP8 (compressed-tensors), loaded directly; FP8 is native
# on the RTX 6000 Ada (cc 8.9). Fallback to vLLM-native dynamic FP8 from
# BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
- ${GRANITE_MODEL}
- --served-model-name
- ${PHI4_SERVED_NAME}
- ${GRANITE_SERVED_NAME}
- --quantization
- ${PHI4_QUANT}
- ${GRANITE_QUANT}
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${PHI4_GPU_MEM_UTIL}
- ${GRANITE_GPU_MEM_UTIL}
- --max-model-len
- ${PHI4_MAX_MODEL_LEN}
- ${GRANITE_MAX_MODEL_LEN}
- --dtype
- auto
# FP8 KV cache — halves KV memory at 128K ctx on Ada (cc 8.9); near-lossless.
# CUDA graphs ENABLED (no --enforce-eager) for decode throughput. Made
# room 2026-06-05 by right-sizing the embed/rerank/reward trio's KV pools
# (they were over-provisioned at 5.9x/2.0x/3.9x concurrency); GPU 1 now has
# ~17 GB free after granite, so graph-capture buffers fit. If the trio
# ever grows back, granite may need --enforce-eager again on this card.
# FP8 KV cache — halves KV memory; near-lossless on Ada (cc 8.9).
- --kv-cache-dtype
- ${PHI4_KV_CACHE_DTYPE}
- ${GRANITE_KV_CACHE_DTYPE}
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${PHI4_GPU_ID}"
- "${GRANITE_GPU_ID}"
capabilities:
- gpu
healthcheck:
@@ -256,10 +262,10 @@ services:
- tnet
labels:
- homepage.group=AI Systems
- homepage.name=vLLM Phi-4-mini (summarizer/dreamer)
- homepage.name=vLLM Granite 4.1 8B (summarizer)
- homepage.icon=mdi-text-box-outline
- homepage.description=Phi-4-mini FP8 via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${PHI4_PORT}/docs
- homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2)
- homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs
networks:
tnet: