vh
edc9f42da1
vllm-lfm25 on ana-ml2 GPU1 :8021 (LiquidAI/LFM2.5-2.6B, util 0.09 into unreserved slack, max-len 16384, no reasoning-parser so content is non-empty). LiteLLM alias lfm2.5-2.6b with vendor sampling baked as default (temp 0.1; top_k 50 + repetition_penalty 1.1 via extra_body). Eval-only, not in any routing chain, pending operator ruling on LFM Open License production use.
160 lines
8.6 KiB
Bash
160 lines
8.6 KiB
Bash
# vllm stack tunables. Copy this to `.env` on the server before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# # edit .env with real values
|
|
# docker compose up -d
|
|
|
|
# Image version — pin for reproducibility (`latest` for edge)
|
|
VLLM_VERSION=latest
|
|
|
|
# Host ports (container always listens on 8000 internally)
|
|
EMBED_PORT=8001
|
|
RERANK_PORT=8002
|
|
REWARD_PORT=8003
|
|
|
|
# GPU assignment — all services share this GPU
|
|
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
|
|
GPU_ID=1
|
|
|
|
# Models — reference by full repo name in API requests
|
|
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
|
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
|
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
|
# reward container at /local-models — see compose.yaml. No env var here for
|
|
# the model path itself since it's hard-coded in the compose command.
|
|
|
|
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
|
# vLLM profiles each service independently, so each slice must be large enough
|
|
# to fit that service's model + KV cache with no awareness of the others.
|
|
# Setting any one too low causes that container to OOM on KV cache allocation
|
|
# with `Available KV cache memory: -X.XX GiB`.
|
|
#
|
|
# Layout on a 48 GB Ada, RIGHT-SIZED 2026-06-05 to free room for the granite
|
|
# summarizer's CUDA graphs + a LoRA hedge. The trio was wildly over-provisioned:
|
|
# embed ran a 5.89x KV pool, reward 3.90x — pure waste for utility models that
|
|
# need ~1.5x. Trimmed to free ~10 GB. (Utilization = fraction of TOTAL GPU mem;
|
|
# each service profiles independently. Too low → that container OOMs on KV with
|
|
# `Available KV cache memory: -X.XX GiB`.)
|
|
# EMBED 0.07 (~3.4 GB) — 0.6B Qwen3 embed @8k; 1.1GB weights + ~2GB KV (~3x)
|
|
# RERANK 0.07 (~3.4 GB) — 0.6B Qwen3 rerank @8k; comfortable
|
|
# REWARD 0.18 (~8.6 GB) — 8B Skywork AWQ @16k; 4.4GB weights + ~3.5GB KV (~1.5x)
|
|
#
|
|
# RETUNED 2026-06-12 for the Ada->Blackwell swap (cards are now 96 GB, so the
|
|
# Ada-era fractions reserved ~2x the bytes for the same models). Empirically
|
|
# re-floored: util climbed in 0.01 steps until each service was stable under
|
|
# 20x parallel inference.
|
|
# embed/rerank 0.05 (~4.8 GB ea) — the load-floor for the 0.6B models (0.04
|
|
# OOMs on engine init; their footprint is fixed weights+context, not KV).
|
|
# reward 0.10 (~9.6 GB) — the real over-provision was here (0.18 -> 0.10).
|
|
# Net: ~11 GB freed on GPU 1. (Fractions are of TOTAL card VRAM — re-floor if
|
|
# the cards change again.)
|
|
# 0.03 each — REBALANCED 2026-06-14 (was 0.05): the 0.6B models sat at ~5.5 GB
|
|
# each at 0.05 (mostly util-reservation waste); 0.03 (~3.6 GB) fits weights +
|
|
# CUDA context with room, freeing ~4 GB back to granite. Recreate them ONE AT A
|
|
# TIME — concurrent recreate races the memory-profiling assertion.
|
|
EMBED_GPU_MEM_UTIL=0.03
|
|
RERANK_GPU_MEM_UTIL=0.03
|
|
REWARD_GPU_MEM_UTIL=0.10
|
|
|
|
# Context length caps — lower these if VRAM is tight.
|
|
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
|
# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
|
|
# enforces the cap at dispatch time per spec.
|
|
EMBED_MAX_MODEL_LEN=8192
|
|
RERANK_MAX_MODEL_LEN=8192
|
|
REWARD_MAX_MODEL_LEN=16384
|
|
|
|
# Optional API key — leave blank for no auth (fine on the internal network).
|
|
# If set, all three services require `Authorization: Bearer <key>`.
|
|
API_KEY=
|
|
|
|
# HuggingFace token — only needed for gated models in the HF-Hub-loaded
|
|
# services (embed/rerank). Reward is local-path, ignores this.
|
|
HF_TOKEN=
|
|
|
|
# === granite-4.1-8b (production summarizer / dreaming agent) ===
|
|
# Replaced phi4-mini 2026-06-05 (Granite 4.1 8B beat phi4 on precision in
|
|
# brokkr's R15 P03 model-fitness eval). Same GPU-1 slot, reusing phi4's port.
|
|
GRANITE_PORT=8004
|
|
# GPU 1 — co-located with the embed/rerank/reward trio. With phi4 retired, GPU 1
|
|
# has ~19 GB free; granite at 56K + FP8 KV needs ~15-16 GB → ~2-3 GB margin.
|
|
# NOTE: llama-swap also uses GPU 1 dynamically — a large swap-in could contend;
|
|
# pin llama-swap to GPU 0 for clean separation (follow-up).
|
|
GRANITE_GPU_ID=1
|
|
# Official IBM pre-quantized FP8 (compressed-tensors) — calibrated, ~9.6 GB,
|
|
# loaded directly (FP8 native on Blackwell cc 12.0). Fallback to vLLM-native dynamic FP8
|
|
# from BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
|
|
GRANITE_MODEL=ibm-granite/granite-4.1-8b-fp8
|
|
GRANITE_QUANT=compressed-tensors
|
|
GRANITE_SERVED_NAME=granite-4.1-8b
|
|
# 131072 ctx — MAXED 2026-06-13 (was 51200/50K). GPU-1 rebalance: granite shares
|
|
# the card with the trio + qwen35-vl (vision). Qwen was over-provisioned on KV
|
|
# (20x conc @ 32k), so trimming it freed room for granite's ~305k-token pool here.
|
|
# PagedAttention allocates KV per ACTUAL token, so 131072 is only a CEILING — a 1k
|
|
# summarize turn uses ~1k tokens, so the pool holds ~300 concurrently; the "2.33x"
|
|
# headline is worst-case (every request maxing 131k). Granite 4.1 supports 131072.
|
|
# 65536 — REDUCED 2026-06-14 (was 131072) to free GPU-1 room for the FP8 vision
|
|
# model (Qwen3.6-35B-A3B, stacks/qwen36-vl, ~34 GB weights). Summarizer load is
|
|
# short parallel calls, so the 64K cap is ample.
|
|
# 131072 — RESTORED 2026-07-16 (native max) for full-chapter summarization; GPU-1
|
|
# freed by the image-bench evict + qwen36-vl gone, so the 128K ctx fits again.
|
|
# 16384 — SHRUNK 2026-07-27 (operator: granite is being phased out) to free GPU-1
|
|
# room for vllm-coder (the Zed FIM seat). Full-chapter ctx dropped; util 0.13 holds
|
|
# 16K cleanly (32768 @ util 0.12 crash-looped: KV est-max was only 19376 tokens).
|
|
GRANITE_MAX_MODEL_LEN=16384
|
|
# FP8 KV cache (native on Blackwell cc 12.0). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
|
|
GRANITE_KV_CACHE_DTYPE=fp8
|
|
# util 0.35 (~33.6 GB) — tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside
|
|
# the trio + qwen co-tenants. On this shared card vLLM needs free >= util*total at
|
|
# startup, and START ORDER matters: trim qwen FIRST, then grow granite, else granite
|
|
# OOMs against the full card. (0.37 overshot to 1.7 GB free; 0.35 lands ~3.7 GB.)
|
|
# 0.24 — REBALANCED 2026-06-14 (was 0.35) for the FP8 vision cutover. GPU-1 budget:
|
|
# qwen36-vl 0.46 + granite 0.24 + reward 0.10 + embed/rerank 0.03 ≈ 0.90 total,
|
|
# ~7.5 GB headroom (the OOM buffer; held under 20-concurrent load test). granite
|
|
# gets a 169K-token KV pool = 2.58x concurrency @ 64K. Bring qwen36 up LAST.
|
|
# 0.18 — RIGHT-SIZED 2026-07-16 (was 0.34 live; qwen36-vl no longer a GPU-1 tenant,
|
|
# image-bench evicted) to free ~10.5 GB for relocating a GPU0 model onto GPU1. KV
|
|
# 6.45 GiB = 84,528 tokens = 1.29x concurrency @ 65536 (summarizer = short parallel
|
|
# calls; ample). Effective slope on this shared card ≈ 950 MiB KV per 0.01 util, and
|
|
# KV must hold >= 1x max-model-len — util 0.15 undershot (crash: est max-len 47184 <
|
|
# 65536), 0.18 lands the target cleanly.
|
|
# 0.27 — RE-GROWN 2026-07-16 (same session) after char-rp moved onto GPU-1: spend the
|
|
# leftover room on full-chapter context (max-len 131072). KV 15.0 GiB = 196,560 tokens
|
|
# = 1.50x @ 131072; GPU-1 lands ~6.7 GB headroom (char-rp 30 + granite 27 + selene 17 + trio).
|
|
# 0.13 — SHRUNK 2026-07-27 (was 0.27) with the max-len drop; frees ~14 GB on GPU-1 for
|
|
# vllm-coder. granite phasing out, so no longer worth the big KV pool. (0.12 was too
|
|
# small for even 16K KV; 0.13 gives ~1.4x @ 16384.)
|
|
GRANITE_GPU_MEM_UTIL=0.13
|
|
# Concurrency cap. Was 1024 (fan-out summarizer); DROPPED to 256 on 2026-07-27 with the
|
|
# phasing-out shrink (256 is ample for the reduced summarizer load; smaller sched state).
|
|
GRANITE_MAX_NUM_SEQS=256
|
|
|
|
# Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed editor inline
|
|
# edit-predictions (deep-research pick 2026-07-27, Apache-2.0). GPU-1, alongside the
|
|
# small-model trio + (phasing-out) granite. Reached via LiteLLM `coder-fast` alias
|
|
# and the keyless zed-fim-proxy (stacks/zed-fim-proxy). util 0.06 (~5.7 GB) holds
|
|
# the 1.5B fp16 + fp8 KV; 8192 ctx is ample for FIM (KV = 13.75x concurrency).
|
|
CODER_PORT=8020
|
|
CODER_GPU_ID=1
|
|
CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B
|
|
CODER_SERVED_NAME=qwen2.5-coder-1.5b
|
|
CODER_MAX_MODEL_LEN=8192
|
|
CODER_KV_CACHE_DTYPE=fp8
|
|
CODER_GPU_MEM_UTIL=0.06
|
|
CODER_MAX_NUM_SEQS=32
|
|
|
|
# LFM2.5-2.6B — NON-PRODUCTION bake-off alias vs granite (brokkr R-target 2026-08-10).
|
|
# LiquidAI LFM Open License v1.0 (<USD 10M-rev commercial, not OSI) — eval-only pending
|
|
# an operator production ruling. Reasoning model (emits <think>); served raw (no vLLM
|
|
# reasoning-parser) so content is non-empty. util 0.09 (~8.6GB) fits GPU1's free slack
|
|
# without touching production reservations; max-len 16384 keeps KV small (bake-off
|
|
# doesn't need the 131k ceiling). Vendor sampling lives in the LiteLLM alias.
|
|
LFM25_PORT=8021
|
|
LFM25_GPU_ID=1
|
|
LFM25_MODEL=LiquidAI/LFM2.5-2.6B
|
|
LFM25_SERVED_NAME=lfm2.5-2.6b
|
|
LFM25_MAX_MODEL_LEN=16384
|
|
LFM25_KV_CACHE_DTYPE=auto
|
|
LFM25_GPU_MEM_UTIL=0.09
|
|
LFM25_MAX_NUM_SEQS=8
|