vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(intern-decision): cap 9.0 GiB with MAX_TOKENS 7168, the largest call measured to fit

Browse Source 
Both are required in compose because they are coupled: MAX_TOKENS is checked before the forward
pass, so an oversized call is a clear 422 instead of reaching the cap as a 503. Pre-deploy floor
is nvidia-smi Free >= 15,400 MiB on GPU 1 (card peak 9,876 + scriberr 5,496).
This commit is contained in:
vh
2026-09-30 09:40:08 -07:00
parent 66034cc69e
commit 750675e391
4 changed files with 38 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/intern-decision/.env.example
+8 -5
View File
@@ -6,10 +6,13 @@ HOST_IP=10.251.50.54
# fv-ml1 GPU 1 = the utility card (vllm-coder, erp, meromero, scriberr).
GPU_ID=1
# HARD torch-allocator cap: the single knob that holds the container's WHOLE nvidia-smi footprint
# (CUDA context included) <= 10,300 MiB, the GPU 1 budget next to scriberr (2026-09-30).
# footprint <= cap + non-allocator overhead = 9,472 MiB + 662 MiB = 10,134 MiB
# (overhead measured flat at 660-662 MiB with the single inference thread; README "VRAM").
# 9.25 still answers the largest request the API accepts (4 calls x 8,191 tokens) with a 200.
VRAM_CAP_GIB=9.25
# (CUDA context included) inside GPU 1's budget next to scriberr (infra-ops, 2026-09-30):
# GPU 1 nvidia-smi Free >= 15,400 MiB = our card peak 9,876 (cap 9.0 GiB + 660 MiB outside the
# allocator, measured) + scriberr's peak 5,496, rounded up. README "VRAM".
VRAM_CAP_GIB=9.0
# Tokens per CALL (state + up to 16 questions), checked BEFORE the forward pass: a longer call is a
# clear 422. 7,168 is the largest call measured to fit under VRAM_CAP_GIB=9.0. Change the two TOGETHER,
# and re-measure (README "VRAM"): a larger value would let a call reach the cap and return 503.
MAX_TOKENS=7168
# >= 32 characters; source of truth: secret get intern-decision/api-token
INTERN_DECISION_API_TOKEN=