vh
1d3b80169a
nevermore pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired with the
granite seat on 2026-08-12. Every summarization call since then failed: 67
consecutive status=failure rows, 0 tokens, twice daily, entirely silently. The
briefing had been rendering with no LLM pass at all. Nothing alerts on
status=failure in the spend logs, so it took an unrelated question about
reranker VRAM to surface it.
It was also pinned to NEVERMORE_RERANK_MODEL=qwen3-reranker -- the incumbent
Brokkr R43 measured harming 80/90 fleet queries -- and was its ONLY caller,
while the production `reranker` alias sat at 0 calls for 4 days. The R43
cutover repointed the alias but never moved the consumer.
nevermore/.env LLAMA_SWAP_MODEL granite-4.1-8b -> summarizer
NEVERMORE_RERANK_MODEL qwen3-reranker -> reranker
(server-only; .env is excluded from the mirror both ways)
Verified against nevermore's exact call shape: summarizer returns clean content
with 0 reasoning chars at temperature 0.2 / max_tokens 4000; reranker scores
0.95 on-topic vs ~1e-5 off-topic; embedding returns dim-1024.
Retired alongside it:
vllm-rerank :8002 Qwen3-Reranker-0.6B + the qwen3-reranker alias
vllm-rerank-a4 :8014 gte-reranker-modernbert + its alias
vllm-granite :8004 Exited 8 days, dead service block
and vllm-rerank-a3 was promoted from a throwaway `docker run` into this stack
(the selection ledger's own open follow-up). Healthy in 55s. It keeps the
bake-off arm name so the ledger, memory and R43 record stay valid.
VLLM_VERSION is pinned latest -> v0.24.0. Every service in the stack shares that
one variable, so a bare `compose up -d` could have silently upgraded all of
them at once; both tags resolved to the same local image (4091d5593f77), so the
pin changed nothing at runtime.
GPU1 is down to 81,448 of 97,887 MiB -- 13.9 GB reclaimed tonight.
Correction: an earlier claim that A4 had no gateway alias was wrong. It did.
LiteLLM serves both config-defined and DB-defined models -- live showed 32
against config.yaml's 26 -- and grepping the file cannot see the difference.
/v1/models and /model/info (which flags db_model) are the ground truth. DB
models delete hot via POST /model/delete with no restart.
Left alone: reranker-a3-bge-v2-m3, a zero-call duplicate of `reranker` on the
same backend. It is Brokkr's cutover-verification handle -- redundant rather
than broken, and another agent's tooling is not mine to delete unilaterally.
111 lines
5.5 KiB
Bash
111 lines
5.5 KiB
Bash
# vllm stack tunables. Copy this to `.env` on the server before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# # edit .env with real values
|
|
# docker compose up -d
|
|
|
|
# Image version — PINNED. Do not put `latest` here: every service in this file
|
|
# shares this one variable, so a bare `docker compose up -d` would silently
|
|
# upgrade the whole stack's vLLM at once. Pinned 2026-08-20 to the version all
|
|
# seats were already running (`latest` and `v0.24.0` were the same local image,
|
|
# 4091d5593f77, so the pin changed nothing at runtime). Bump deliberately.
|
|
VLLM_VERSION=v0.24.0
|
|
|
|
# Host ports (container always listens on 8000 internally)
|
|
EMBED_PORT=8001
|
|
# 8013 — the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which
|
|
# had been running as a throwaway `docker run` on this port) was promoted into
|
|
# this stack and the Qwen incumbent on :8002 was retired.
|
|
RERANK_PORT=8013
|
|
REWARD_PORT=8003
|
|
|
|
# GPU assignment — all services share this GPU
|
|
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
|
|
GPU_ID=1
|
|
|
|
# Models — reference by full repo name in API requests
|
|
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
|
# bge-reranker-v2-m3 — the R43 bake-off winner, replacing Qwen3-Reranker-0.6B
|
|
# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no
|
|
# --hf-overrides, unlike the Qwen reranker it displaced.
|
|
RERANK_MODEL=BAAI/bge-reranker-v2-m3
|
|
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
|
# reward container at /local-models — see compose.yaml. No env var here for
|
|
# the model path itself since it's hard-coded in the compose command.
|
|
|
|
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
|
# vLLM profiles each service independently, so each slice must be large enough
|
|
# to fit that service's model + KV cache with no awareness of the others.
|
|
# Setting any one too low causes that container to OOM on KV cache allocation
|
|
# with `Available KV cache memory: -X.XX GiB`.
|
|
#
|
|
# Layout on a 48 GB Ada, RIGHT-SIZED 2026-06-05 to free room for the granite
|
|
# summarizer's CUDA graphs + a LoRA hedge. The trio was wildly over-provisioned:
|
|
# embed ran a 5.89x KV pool, reward 3.90x — pure waste for utility models that
|
|
# need ~1.5x. Trimmed to free ~10 GB. (Utilization = fraction of TOTAL GPU mem;
|
|
# each service profiles independently. Too low → that container OOMs on KV with
|
|
# `Available KV cache memory: -X.XX GiB`.)
|
|
# EMBED 0.07 (~3.4 GB) — 0.6B Qwen3 embed @8k; 1.1GB weights + ~2GB KV (~3x)
|
|
# RERANK 0.07 (~3.4 GB) — 0.6B Qwen3 rerank @8k; comfortable
|
|
# REWARD 0.18 (~8.6 GB) — 8B Skywork AWQ @16k; 4.4GB weights + ~3.5GB KV (~1.5x)
|
|
#
|
|
# RETUNED 2026-06-12 for the Ada->Blackwell swap (cards are now 96 GB, so the
|
|
# Ada-era fractions reserved ~2x the bytes for the same models). Empirically
|
|
# re-floored: util climbed in 0.01 steps until each service was stable under
|
|
# 20x parallel inference.
|
|
# embed/rerank 0.05 (~4.8 GB ea) — the load-floor for the 0.6B models (0.04
|
|
# OOMs on engine init; their footprint is fixed weights+context, not KV).
|
|
# reward 0.10 (~9.6 GB) — the real over-provision was here (0.18 -> 0.10).
|
|
# Net: ~11 GB freed on GPU 1. (Fractions are of TOTAL card VRAM — re-floor if
|
|
# the cards change again.)
|
|
# 0.03 each — REBALANCED 2026-06-14 (was 0.05): the 0.6B models sat at ~5.5 GB
|
|
# each at 0.05 (mostly util-reservation waste); 0.03 (~3.6 GB) fits weights +
|
|
# CUDA context with room, freeing ~4 GB back to granite. Recreate them ONE AT A
|
|
# TIME — concurrent recreate races the memory-profiling assertion.
|
|
EMBED_GPU_MEM_UTIL=0.03
|
|
RERANK_GPU_MEM_UTIL=0.03
|
|
REWARD_GPU_MEM_UTIL=0.10
|
|
|
|
# Context length caps — lower these if VRAM is tight.
|
|
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
|
# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
|
|
# enforces the cap at dispatch time per spec.
|
|
EMBED_MAX_MODEL_LEN=8192
|
|
RERANK_MAX_MODEL_LEN=8192
|
|
REWARD_MAX_MODEL_LEN=16384
|
|
|
|
# Optional API key — leave blank for no auth (fine on the internal network).
|
|
# If set, all three services require `Authorization: Bearer <key>`.
|
|
API_KEY=
|
|
|
|
# HuggingFace token — only needed for gated models in the HF-Hub-loaded
|
|
# services (embed/rerank). Reward is local-path, ignores this.
|
|
HF_TOKEN=
|
|
|
|
# === granite-4.1-8b — RETIRED 2026-08-12, vars removed 2026-08-20 ===
|
|
# Was the production summarizer. The `summarizer` / `classifier` gateway aliases
|
|
# were repointed at the gen seat and the container stopped; the service block and
|
|
# these tunables are now gone. See the tombstone in compose.yaml for the lesson
|
|
# this cost us (nevermore stayed pinned to the dead `granite-4.1-8b` alias and
|
|
# failed silently for 8 days).
|
|
|
|
# Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed editor inline
|
|
# edit-predictions (deep-research pick 2026-07-27, Apache-2.0). GPU-1, alongside the
|
|
# small-model trio + (phasing-out) granite. Reached via LiteLLM `coder-fast` alias
|
|
# and the keyless zed-fim-proxy (stacks/zed-fim-proxy). util 0.06 (~5.7 GB) holds
|
|
# the 1.5B fp16 + fp8 KV; 8192 ctx is ample for FIM (KV = 13.75x concurrency).
|
|
CODER_PORT=8020
|
|
CODER_GPU_ID=1
|
|
CODER_MODEL=Qwen/Qwen2.5-Coder-1.5B
|
|
CODER_SERVED_NAME=qwen2.5-coder-1.5b
|
|
CODER_MAX_MODEL_LEN=8192
|
|
CODER_KV_CACHE_DTYPE=fp8
|
|
CODER_GPU_MEM_UTIL=0.06
|
|
CODER_MAX_NUM_SEQS=32
|
|
|
|
# === LFM2.5-2.6B — RETIRED PERMANENTLY 2026-08-20 (operator directive) ===
|
|
# Eval-only bake-off seat vs granite-4.1-8b that never got its production ruling;
|
|
# its comparator was retired first, and it logged 0 calls in the 4 days before it
|
|
# came down. Container removed, service block and tunables deleted, gateway alias
|
|
# dropped. Weights remain in the shared HF cache.
|