vh
34a43a0bc5
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's the new production summarizer/dreamer for nevermore. - vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8, compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot. - GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a 5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18, freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora). - LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004); explicit entry shadows the '*' wildcard's llama-swap route. - nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live. Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok), KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
86 lines
4.0 KiB
Bash
86 lines
4.0 KiB
Bash
# vllm stack tunables. Copy this to `.env` on the server before deploying.
|
||
#
|
||
# cp .env.example .env
|
||
# # edit .env with real values
|
||
# docker compose up -d
|
||
|
||
# Image version — pin for reproducibility (`latest` for edge)
|
||
VLLM_VERSION=latest
|
||
|
||
# Host ports (container always listens on 8000 internally)
|
||
EMBED_PORT=8001
|
||
RERANK_PORT=8002
|
||
REWARD_PORT=8003
|
||
|
||
# GPU assignment — all services share this GPU
|
||
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
|
||
GPU_ID=1
|
||
|
||
# Models — reference by full repo name in API requests
|
||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
||
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
||
# reward container at /local-models — see compose.yaml. No env var here for
|
||
# the model path itself since it's hard-coded in the compose command.
|
||
|
||
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
||
# vLLM profiles each service independently, so each slice must be large enough
|
||
# to fit that service's model + KV cache with no awareness of the others.
|
||
# Setting any one too low causes that container to OOM on KV cache allocation
|
||
# with `Available KV cache memory: -X.XX GiB`.
|
||
#
|
||
# Layout on a 48 GB Ada, RIGHT-SIZED 2026-06-05 to free room for the granite
|
||
# summarizer's CUDA graphs + a LoRA hedge. The trio was wildly over-provisioned:
|
||
# embed ran a 5.89x KV pool, reward 3.90x — pure waste for utility models that
|
||
# need ~1.5x. Trimmed to free ~10 GB. (Utilization = fraction of TOTAL GPU mem;
|
||
# each service profiles independently. Too low → that container OOMs on KV with
|
||
# `Available KV cache memory: -X.XX GiB`.)
|
||
# EMBED 0.07 (~3.4 GB) — 0.6B Qwen3 embed @8k; 1.1GB weights + ~2GB KV (~3x)
|
||
# RERANK 0.07 (~3.4 GB) — 0.6B Qwen3 rerank @8k; comfortable
|
||
# REWARD 0.18 (~8.6 GB) — 8B Skywork AWQ @16k; 4.4GB weights + ~3.5GB KV (~1.5x)
|
||
EMBED_GPU_MEM_UTIL=0.07
|
||
RERANK_GPU_MEM_UTIL=0.07
|
||
REWARD_GPU_MEM_UTIL=0.18
|
||
|
||
# Context length caps — lower these if VRAM is tight.
|
||
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
||
# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
|
||
# enforces the cap at dispatch time per spec.
|
||
EMBED_MAX_MODEL_LEN=8192
|
||
RERANK_MAX_MODEL_LEN=8192
|
||
REWARD_MAX_MODEL_LEN=16384
|
||
|
||
# Optional API key — leave blank for no auth (fine on the internal network).
|
||
# If set, all three services require `Authorization: Bearer <key>`.
|
||
API_KEY=
|
||
|
||
# HuggingFace token — only needed for gated models in the HF-Hub-loaded
|
||
# services (embed/rerank). Reward is local-path, ignores this.
|
||
HF_TOKEN=
|
||
|
||
# === granite-4.1-8b (production summarizer / dreaming agent) ===
|
||
# Replaced phi4-mini 2026-06-05 (Granite 4.1 8B beat phi4 on precision in
|
||
# brokkr's R15 P03 model-fitness eval). Same GPU-1 slot, reusing phi4's port.
|
||
GRANITE_PORT=8004
|
||
# GPU 1 — co-located with the embed/rerank/reward trio. With phi4 retired, GPU 1
|
||
# has ~19 GB free; granite at 56K + FP8 KV needs ~15-16 GB → ~2-3 GB margin.
|
||
# NOTE: llama-swap also uses GPU 1 dynamically — a large swap-in could contend;
|
||
# pin llama-swap to GPU 0 for clean separation (follow-up).
|
||
GRANITE_GPU_ID=1
|
||
# Official IBM pre-quantized FP8 (compressed-tensors) — calibrated, ~9.6 GB,
|
||
# loaded directly (FP8 native on Ada cc 8.9). Fallback to vLLM-native dynamic FP8
|
||
# from BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
|
||
GRANITE_MODEL=ibm-granite/granite-4.1-8b-fp8
|
||
GRANITE_QUANT=compressed-tensors
|
||
GRANITE_SERVED_NAME=granite-4.1-8b
|
||
# 50K ctx (51200) — matches phi4's window. Deliberately NOT maxed: leaves ~3 GB
|
||
# on GPU 1 as a hedge for future Granite-LoRA serving (--enable-lora needs ~2-3 GB)
|
||
# / a holistic GPU-1 retune of the embed/rerank/reward trio. KV @50K fp8 ≈ 4.2 GB;
|
||
# +9.6 weights +~2.4 overhead ≈ 16.2 GB. Granite 4.1 supports up to 131072.
|
||
GRANITE_MAX_MODEL_LEN=51200
|
||
# FP8 KV cache (native on Ada cc 8.9). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
|
||
GRANITE_KV_CACHE_DTYPE=fp8
|
||
# Target ~16.2 GB of GPU 1's ~19 GB free: weights 9.6 + KV 4.2 + overhead ~2.4.
|
||
# 0.36×46≈16.6 GB budget (covers it with slack). Confirm vs vLLM's first-boot KV log.
|
||
GRANITE_GPU_MEM_UTIL=0.36
|