tune(gpu1): grow selene 0.13→0.17 + qwen36 0.32→0.34 into the buffer
Put GPU1's idle ~11 GB buffer to work on the two KV-bound models that gained live consumers from the worldtree migration (granite + the pooling models under-use their util, so growing them is wasted): - selene 0.13→0.17: KV 2.53→6.33 GiB, concurrency 1.27x→3.16x @32K (Domari judge) - qwen36 0.32→0.34: KV 7.73→9.63 GiB, concurrency 2.92x→3.64x @131K (arbo judge + worldtree actor/echo + gateway) GPU1 free now ~5.6 GB (safe floor for single-service recreates).
This commit is contained in:
@@ -13,11 +13,11 @@ SELENE_PORT=8011
|
||||
# GPU 1 = shared with qwen36 (NVFP4) + granite + embed/rerank/reward.
|
||||
SELENE_GPU_ID=1
|
||||
|
||||
# util 0.13 (~12.5 GB) — measured: 8.51 GiB FP8 weights + ~1.5 GiB graph +
|
||||
# 2.53 GiB fp8 KV (41,456-token pool, 1.27x concurrency at full 32K). util 0.12
|
||||
# was too thin (1.86 GiB KV < the 2.0 GiB a single 32K request needs → crash).
|
||||
# Keeps GPU 1 total ~0.95 → ~5 GB buffer; ctx 32768 mirrors the old judge config.
|
||||
SELENE_GPU_MEM_UTIL=0.13
|
||||
# util 0.17 (~16.3 GB) — grown from 0.13 to give Domari's now-live judge role real
|
||||
# concurrency headroom (KV ~6.3 GiB → ~3.2x at 32K, far more at short eval ctx).
|
||||
# 8.51 GiB FP8 weights + graph + the rest is KV. (0.12 crashed — KV < one 32K req;
|
||||
# 0.13 was the minimum-fit; 0.17 is the comfortable judge size.) ctx 32768.
|
||||
SELENE_GPU_MEM_UTIL=0.17
|
||||
SELENE_MAX_MODEL_LEN=32768
|
||||
SELENE_MAX_NUM_SEQS=16
|
||||
|
||||
|
||||
Reference in New Issue
Block a user