vh
e124a2f233
Put GPU1's idle ~11 GB buffer to work on the two KV-bound models that gained live consumers from the worldtree migration (granite + the pooling models under-use their util, so growing them is wasted): - selene 0.13→0.17: KV 2.53→6.33 GiB, concurrency 1.27x→3.16x @32K (Domari judge) - qwen36 0.32→0.34: KV 7.73→9.63 GiB, concurrency 2.92x→3.64x @131K (arbo judge + worldtree actor/echo + gateway) GPU1 free now ~5.6 GB (safe floor for single-service recreates).
27 lines
1.1 KiB
Bash
27 lines
1.1 KiB
Bash
# Selene 1 Mini 8B judge (FP8) on ana-ml2 GPU 1 — copy to .env on the host.
|
|
# Real .env lives on ana-ml2 at /opt/docker/compose/selene/.env (gitignored).
|
|
#
|
|
# See compose.yaml header for the FP8-over-NVFP4 (judge-fidelity) rationale.
|
|
|
|
# 0.23.0 — Llama 3.1 + dynamic fp8 is rock-solid here (same digest as qwen36).
|
|
SELENE_IMAGE=vllm/vllm-openai@sha256:6d8429e38e3747723ca07ee1b17972e09bb9c51c4032b266f24fb1cc3b22ed8f
|
|
|
|
SELENE_CONTAINER_NAME=vllm-selene
|
|
SELENE_MODEL=AtlaAI/Selene-1-Mini-Llama-3.1-8B
|
|
SELENE_PORT=8011
|
|
|
|
# GPU 1 = shared with qwen36 (NVFP4) + granite + embed/rerank/reward.
|
|
SELENE_GPU_ID=1
|
|
|
|
# util 0.17 (~16.3 GB) — grown from 0.13 to give Domari's now-live judge role real
|
|
# concurrency headroom (KV ~6.3 GiB → ~3.2x at 32K, far more at short eval ctx).
|
|
# 8.51 GiB FP8 weights + graph + the rest is KV. (0.12 crashed — KV < one 32K req;
|
|
# 0.13 was the minimum-fit; 0.17 is the comfortable judge size.) ctx 32768.
|
|
SELENE_GPU_MEM_UTIL=0.17
|
|
SELENE_MAX_MODEL_LEN=32768
|
|
SELENE_MAX_NUM_SEQS=16
|
|
|
|
# Optional — checkpoint is ungated (Apache-2.0).
|
|
HF_TOKEN=
|
|
API_KEY=
|