feat(vllm): replace phi4-mini with Granite 4.1 8B summarizer + retune GPU 1
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's the new production summarizer/dreamer for nevermore. - vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8, compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot. - GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a 5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18, freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora). - LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004); explicit entry shadows the '*' wildcard's llama-swap route. - nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live. Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok), KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
This commit is contained in:
+23
-17
@@ -201,13 +201,13 @@ services:
|
||||
# llama-swap granite-4-small pin. Generative chat model (OpenAI
|
||||
# /v1/chat/completions), so NO --runner pooling. FP8 on RTX 6000 Ada
|
||||
# (cc 8.9): near-lossless, ~1.2x, ~6 GB.
|
||||
vllm-phi4:
|
||||
vllm-granite:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-phi4
|
||||
container_name: vllm-granite
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${PHI4_PORT}:8000"
|
||||
- "${GRANITE_PORT}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
@@ -216,34 +216,40 @@ services:
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
# Default = vLLM-native dynamic FP8 from the official BF16 weights (most
|
||||
# robust on Ada). Alt = pre-quantized checkpoint: PHI4_MODEL=
|
||||
# pytorch/Phi-4-mini-instruct-FP8 + PHI4_QUANT=torchao (see .env/README).
|
||||
- ${PHI4_MODEL}
|
||||
# Production summarizer (replaced phi4-mini 2026-06-05). Default = official
|
||||
# IBM pre-quantized FP8 (compressed-tensors), loaded directly; FP8 is native
|
||||
# on the RTX 6000 Ada (cc 8.9). Fallback to vLLM-native dynamic FP8 from
|
||||
# BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
|
||||
- ${GRANITE_MODEL}
|
||||
- --served-model-name
|
||||
- ${PHI4_SERVED_NAME}
|
||||
- ${GRANITE_SERVED_NAME}
|
||||
- --quantization
|
||||
- ${PHI4_QUANT}
|
||||
- ${GRANITE_QUANT}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${PHI4_GPU_MEM_UTIL}
|
||||
- ${GRANITE_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${PHI4_MAX_MODEL_LEN}
|
||||
- ${GRANITE_MAX_MODEL_LEN}
|
||||
- --dtype
|
||||
- auto
|
||||
# FP8 KV cache — halves KV memory at 128K ctx on Ada (cc 8.9); near-lossless.
|
||||
# CUDA graphs ENABLED (no --enforce-eager) for decode throughput. Made
|
||||
# room 2026-06-05 by right-sizing the embed/rerank/reward trio's KV pools
|
||||
# (they were over-provisioned at 5.9x/2.0x/3.9x concurrency); GPU 1 now has
|
||||
# ~17 GB free after granite, so graph-capture buffers fit. If the trio
|
||||
# ever grows back, granite may need --enforce-eager again on this card.
|
||||
# FP8 KV cache — halves KV memory; near-lossless on Ada (cc 8.9).
|
||||
- --kv-cache-dtype
|
||||
- ${PHI4_KV_CACHE_DTYPE}
|
||||
- ${GRANITE_KV_CACHE_DTYPE}
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${PHI4_GPU_ID}"
|
||||
- "${GRANITE_GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
@@ -256,10 +262,10 @@ services:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI Systems
|
||||
- homepage.name=vLLM Phi-4-mini (summarizer/dreamer)
|
||||
- homepage.name=vLLM Granite 4.1 8B (summarizer)
|
||||
- homepage.icon=mdi-text-box-outline
|
||||
- homepage.description=Phi-4-mini FP8 via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${PHI4_PORT}/docs
|
||||
- homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
|
||||
Reference in New Issue
Block a user