feat(gpu): relocate char-rp to GPU1 + re-optimize both ana-ml2 cards for max context
Operator-directed 2026-07-16. Moved the char-rp prose seat (Magidonia-24B,
llama-charrp) from GPU0 to GPU1 (CHARRP_GPU_ID 0->1; recreate llama-charrp
only -- the var is shared with the retired GGUF reasoning service), then
re-optimized every context-relevant seat on both cards to native/max context
with acceptable headroom:
GPU0 (both seats now 256K native, ~14 GB reserve):
- char-rp-reasoning 150K -> 256K (heretic2 stack, util 0.38->0.46, 1.56x)
- gen 256K, max-num-seqs 16 -> 32 (qwen36-27b-aeon, util 0.30->0.42, 5.43x)
GPU1 (~6.7 GB headroom):
- granite 64K -> 128K full-chapter (vllm stack, util 0.18->0.27, 1.50x)
- char-rp 128K native (4 slots), selene/reward/embed/rerank unchanged
All seats gateway-verified healthy. Live .env changes on ana-ml2 with per-stack
backups (*-20260716). Templates updated to match; the qwen36-27b-aeon template
carries a NOTE that its served-name/model still lag the 2026-07-08 gen model swap
(35B-A3B-heretic) -- separate reconciliation. persistent-memory records the full
layout + the util-floor / per-model-KV-cost lessons.
Note: the heretic2-charrp-reasoning stack (char-rp-reasoning's live config) is
still untracked in git -- standing open-loop, its .env change lives server-side only.
This commit is contained in:
@@ -8,9 +8,11 @@
|
||||
# llama.cpp server image (bundles /app/llama-server; already present on ana-ml2).
|
||||
LLAMA_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda
|
||||
|
||||
# GPU 0 = the on-demand large-model card (co-resident with `gen`). GPU 1 is the
|
||||
# always-on vLLM trio (granite/embed/rerank/reward) — do NOT put this stack there.
|
||||
CHARRP_GPU_ID=0
|
||||
# GPU pin for the char-rp PROSE seat (llama-charrp). MOVED to GPU 1 on 2026-07-16 to
|
||||
# decongest GPU 0 (freed ~29 GB there for gen + char-rp-reasoning to grow to 256K).
|
||||
# NOTE: this var is ALSO read by the (retired, stopped) llama-charrp-reasoning service
|
||||
# below — recreate ONLY llama-charrp when changing it, or split into a dedicated var.
|
||||
CHARRP_GPU_ID=1
|
||||
|
||||
# GGUF store on the host (mounted read-only at /models). Candidate GGUFs live in
|
||||
# /tank/aimodels/llm/rp/ (pre-pulled).
|
||||
|
||||
@@ -19,9 +19,13 @@ AEON_GEN_PORT=8015
|
||||
AEON_GEN_SERVED_NAME=qwen3.6-27b-aeon
|
||||
AEON_GEN_SERVED_NAME_THINK=qwen3.6-27b-aeon-thinking
|
||||
AEON_GEN_MODEL=/tank/aimodels/qwen36-27b-aeon-nvfp4
|
||||
AEON_GEN_GPU_MEM_UTIL=0.45
|
||||
AEON_GEN_MAX_MODEL_LEN=131072
|
||||
AEON_GEN_MAX_NUM_SEQS=16
|
||||
# NOTE: live gen serves qwen3.6-35b-a3b-heretic since 2026-07-08 (aeon-27b displaced);
|
||||
# SERVED_NAME/MODEL above lag live — pending reconciliation with the model swap.
|
||||
# util/ctx/seqs RE-TUNED 2026-07-16 (char-rp left GPU-0 → room): util 0.30->0.42,
|
||||
# ctx 131072->262144 (256K native), seqs 16->32. gen MoE KV is cheap → 5.43x @ 256K.
|
||||
AEON_GEN_GPU_MEM_UTIL=0.42
|
||||
AEON_GEN_MAX_MODEL_LEN=262144
|
||||
AEON_GEN_MAX_NUM_SEQS=32
|
||||
AEON_GEN_KV_CACHE_DTYPE=fp8
|
||||
# Reasoning parser — KEEP qwen3. This checkpoint's chat_template.jinja injects the opening
|
||||
# <think> into the PROMPT (output has only </think>). qwen3 handles the non-thinking path
|
||||
|
||||
@@ -96,7 +96,9 @@ GRANITE_SERVED_NAME=granite-4.1-8b
|
||||
# 65536 — REDUCED 2026-06-14 (was 131072) to free GPU-1 room for the FP8 vision
|
||||
# model (Qwen3.6-35B-A3B, stacks/qwen36-vl, ~34 GB weights). Summarizer load is
|
||||
# short parallel calls, so the 64K cap is ample.
|
||||
GRANITE_MAX_MODEL_LEN=65536
|
||||
# 131072 — RESTORED 2026-07-16 (native max) for full-chapter summarization; GPU-1
|
||||
# freed by the image-bench evict + qwen36-vl gone, so the 128K ctx fits again.
|
||||
GRANITE_MAX_MODEL_LEN=131072
|
||||
# FP8 KV cache (native on Blackwell cc 12.0). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
|
||||
GRANITE_KV_CACHE_DTYPE=fp8
|
||||
# util 0.35 (~33.6 GB) — tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside
|
||||
@@ -113,4 +115,7 @@ GRANITE_KV_CACHE_DTYPE=fp8
|
||||
# calls; ample). Effective slope on this shared card ≈ 950 MiB KV per 0.01 util, and
|
||||
# KV must hold >= 1x max-model-len — util 0.15 undershot (crash: est max-len 47184 <
|
||||
# 65536), 0.18 lands the target cleanly.
|
||||
GRANITE_GPU_MEM_UTIL=0.18
|
||||
# 0.27 — RE-GROWN 2026-07-16 (same session) after char-rp moved onto GPU-1: spend the
|
||||
# leftover room on full-chapter context (max-len 131072). KV 15.0 GiB = 196,560 tokens
|
||||
# = 1.50x @ 131072; GPU-1 lands ~6.7 GB headroom (char-rp 30 + granite 27 + selene 17 + trio).
|
||||
GRANITE_GPU_MEM_UTIL=0.27
|
||||
|
||||
Reference in New Issue
Block a user