vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

scriberr to fv-ml1 GPU 3 (on-demand, steps aside to irv-ml1 A6000); intern-decision 32k-token calls (cap 14.4 GiB)

Browse Source 
Prime 2026-09-30: move scriberr to GPU 3 and extend the Jev endpoint to 32k tokens.
Scriberr holds 0 VRAM idle; verified a 20-min job on GPU 3 at 5,496 MiB. With GPU 1
freed, intern-decision's measured card peak at MAX_TOKENS=32768 is 15,220 MiB against
a 15,437 MiB budget (n=3, 1 and 16 questions); 32,769 tokens is refused 422 up front.
JevBench v1.2.16 via /v1/systemone unchanged: 202/231, 0 diffs vs the bench.
This commit is contained in:
vh
2026-09-30 13:35:32 -07:00
parent 92501a29c1
commit 6b201e1d4a
8 changed files with 79 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/intern-decision/.env.example
+10 -9
View File
@@ -1,18 +1,19 @@
# intern-decision — copy to /opt/docker/compose/intern-decision/.env on fv-ml1 (mode 0600).
# Built on fv-ml1 from services/intern-decision-serve (see README "Building").
IMAGE=intern-decision-serve:0.1.0
IMAGE=intern-decision-serve:0.1.2
PORT=8033
HOST_IP=10.251.50.54
# fv-ml1 GPU 1 = the utility card (vllm-coder, erp, meromero, scriberr).
# fv-ml1 GPU 1 = the utility card (vllm-coder, erp, meromero). scriberr moved to GPU 3 (2026-09-30).
GPU_ID=1
# HARD torch-allocator cap: the single knob that holds the container's WHOLE nvidia-smi footprint
# (CUDA context included) inside GPU 1's budget next to scriberr (infra-ops, 2026-09-30):
# GPU 1 nvidia-smi Free >= 15,400 MiB = our card peak 9,876 (cap 9.0 GiB + 660 MiB outside the
# allocator, measured) + scriberr's peak 5,496, rounded up. README "VRAM".
VRAM_CAP_GIB=9.0
# (CUDA context included, ~660 MiB outside the allocator) inside GPU 1's free memory (infra-ops,
# 2026-09-30 1330): with the vLLM seats static, this container may use its rest (8,812) + GPU 1
# nvidia-smi Free (6,625) = 15,437 MiB. 14.4 GiB cap -> card ceiling ~15,408. README "VRAM".
VRAM_CAP_GIB=14.4
# Tokens per CALL (state + up to 16 questions), checked BEFORE the forward pass: a longer call is a
# clear 422. 7,168 is the largest call measured to fit under VRAM_CAP_GIB=9.0. Change the two TOGETHER,
# and re-measure (README "VRAM"): a larger value would let a call reach the cap and return 503.
MAX_TOKENS=7168
# clear 422. 32,768 = Jev's "32k for state plus the longest question"; measured to fit under 14.4 GiB
# at a card peak of 15,220 MiB (1 question and 16 questions, n=3 each). Change the two TOGETHER, and
# re-measure (README "VRAM"): a larger value would let a call reach the cap and return 503.
MAX_TOKENS=32768
# >= 32 characters; source of truth: secret get intern-decision/api-token
INTERN_DECISION_API_TOKEN=
stacks/intern-decision/README.md
+30 -1
View File
@@ -80,7 +80,36 @@ curl -s -H "Authorization: Bearer $T" http://intern-decision.fv.internal:8033/de
never truncated.
- A request with more decisions is split into more calls, and each call must fit.
## VRAM: fits beside scriberr's peak, whatever the request
## VRAM: 32k-token calls since 2026-09-30 1330 (scriberr moved to GPU 3)
**Current setting: `VRAM_CAP_GIB=14.4`, `MAX_TOKENS=32768`** (Prime: move scriberr to GPU 3 and "extend the jev
endpoint to hit 32k tokens if possible"). With scriberr gone, GPU 1 holds only the static vLLM seats and this
service. This container may therefore use its rest (8,812 MiB) plus GPU 1's nvidia-smi `Free` (6,625 MiB), which is
**15,437 MiB**. The 14.4 GiB cap plus the ~660 MiB outside the allocator puts the card ceiling at ~15,408 MiB.
Measured on the live service (per-process nvidia-smi every 0.1 s; single `noul` question unless noted; n=3, deterministic):
| call tokens | card peak MiB | wall (warm) |
|---|---|---|
| 3,187 | 9,306 | 0.15 s |
| 12,187 | 11,206 | 0.68 s |
| 24,187 | 13,552 | 1.49 s |
| 29,987 | 14,692 | 1.92 s |
| **32,768 (limit)** | **15,220** | 2.12 s |
| 32,765, 16 questions | 15,220 | 2.15 s |
| 32,769 | refused 422 before the forward | 0.30 s |
Spare at the limit: 217 MiB. JevBench v1.2.16 through `/v1/systemone` after the change: 202/231, with 0 answer and 0
probability diffs against the bench's r1..r4. `/decide` is unchanged.
⚠ **Cold-shape latency:** the first call in a new length bucket after a (re)start costs ~6.5 s extra
(3,001 and 4,000 words were slow; 3,002–3,500 and 4,097–5,000 were not). The fast kernels autotune per shape
bucket, and the result is cached in-process. Warm calls are as tabled. A startup warm-up across the buckets would
remove it; that is not done yet.
The table below is the ORIGINAL budget (9.0 GiB / 7,168 tokens, beside scriberr) and is kept for history.
## VRAM (history): fits beside scriberr's peak, whatever the request
**Budget (infra-ops, 2026-09-30):** GPU 1 needs nvidia-smi `Free` ≥ **15,400 MiB** before this
service starts. That is our card peak of 9,876 MiB plus scriberr's peak of 5,496 MiB (with its
stacks/intern-decision/compose.yaml
+4 -3
View File
@@ -1,5 +1,6 @@
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
# fv-ml1 GPU 1 (the utility card, beside vllm-coder, the erp/meromero seats and scriberr).
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
#
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
@@ -8,8 +9,8 @@
# fv-ml1 from that dir.
#
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
# container's WHOLE nvidia-smi footprint, CUDA context included, fits beside scriberr's peak
# (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
# gets 503 out_of_memory and the service stays up. See the README before changing it.
#
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
stacks/scriberr/.env.example
+4 -3
View File
@@ -19,8 +19,9 @@ SCRIBERR_BIND=0.0.0.0
SCRIBERR_ALLOWED_ORIGINS=http://10.251.50.54:8080,http://scriberr.fv.internal:8080
# ── GPU ──────────────────────────────────────────────────────────────────
# GPU0 is fully committed to the `gen` seat; GPU1 is the one with headroom.
SCRIBERR_GPU_ID=1
# GPU 3 since 2026-09-30 (Prime): an on-demand tenant of the full-size-seat reserve; it steps aside
# (to irv-ml1's A6000) when a full-size seat claims GPU 3. See compose.yaml.
SCRIBERR_GPU_ID=3
# ── Storage (on /tank — NOT the root pool, weights are multi-GB) ─────────
SCRIBERR_DATA_DIR=/tank/scriberr/data
@@ -48,7 +49,7 @@ SCRIBERR_SECURE_COOKIES=false
# tab — keep the configured model on a free local seat.
# SCRIBERR_OPENAI_API_KEY=
# GPU 1 memory budget (2026-09-30). Parakeet slice length in seconds and the
# Memory settings measured on GPU 1 (2026-09-30; still in force on GPU 3). Parakeet slice length in seconds and the
# torch allocator mode; see compose.yaml for the measurements. Defaults apply
# when unset; override only with a re-measured peak.
# SCRIBERR_PARAKEET_CHUNK_SECS=120
stacks/scriberr/README.md
+4
View File
@@ -142,6 +142,10 @@ tab. Keep the configured model on a free local seat.
## Parakeet memory and slicing (measured 2026-09-30)
> **Moved to fv-ml1 GPU 3 at 1322 on 2026-09-30 (Prime).** The GPU 1 budget below no longer binds. Scriberr is
> an on-demand tenant of GPU 3's reserve: it steps aside to irv-ml1's A6000 when a full-size seat claims GPU 3.
> A 20-min file was verified on GPU 3 at a 5,496 MiB peak. The settings below were not changed by the move.
Scriberr shares fv-ml1 GPU 1 with intern-decision (9.0 GiB cap, 9,876 MiB card peak).
GPU 1's nvidia-smi Free is 15,442 MiB, so the budget left for Scriberr is about 5.5 GB
(70 MiB spare at both peaks), so the compose file sets
stacks/scriberr/compose.yaml
+10 -7
View File
@@ -18,11 +18,13 @@
# /tank/scriberr/src/Scriberr on fv-ml1.
#
# ── GPU PINNING ───────────────────────────────────────────────────────────
# Pinned to **GPU1** via explicit device_ids, per the house convention and
# because GPU0 is fully committed to the `gen` seat. GPU1 shares space with
# the `sec` seat, so this stack is a guest there — keep an eye on VRAM.
# Pinned to **GPU 3** (Prime, 2026-09-30 1322) via explicit device_ids. Scriberr holds 0 VRAM
# when idle, so it is an ON-DEMAND tenant of GPU 3's full-size-seat reserve, like Blender: when a
# full-size seat (Flash-Next) claims GPU 3, Scriberr STEPS ASIDE. Its planned landing spot then is
# irv-ml1's A6000, not GPU 1 (GPU 1's headroom now funds intern-decision's 32k-token calls).
# (It was on GPU 1 until 2026-09-30, beside the vLLM seats; history in stacks/scriberr/README.md.)
# NOTE: do NOT add `NVIDIA_VISIBLE_DEVICES=all` (as upstream's compose does).
# It overrides the device_ids reservation and exposes both cards.
# It overrides the device_ids reservation and exposes every card.
#
# All tunables live in .env — edit that, not this file.
@@ -70,7 +72,8 @@ services:
# and takes out the Parakeet + Sortformer backends (WhisperX survives).
# `copy` trades a little disk and time for it actually working.
- UV_LINK_MODE=${SCRIBERR_UV_LINK_MODE:-copy}
# ── GPU 1 memory budget (2026-09-30, Prime: Scriberr shares GPU 1 with
# ── Memory settings, measured while Scriberr shared GPU 1 (2026-09-30; it moved to
# GPU 3 at 1322 the same day, so the 5.5 GB budget no longer binds, but the values stand). Prime: Scriberr shared GPU 1 with
# intern-decision, which holds ~9.7 GB resting / 10.3 GB peak). ──────────
# Parakeet's buffered path cuts audio into slices of this many seconds
# (Scriberr reads it in parakeet_adapter.go for BOTH the "is this long
@@ -94,7 +97,7 @@ services:
reservations:
devices:
- driver: nvidia
device_ids: ["${SCRIBERR_GPU_ID:-1}"]
device_ids: ["${SCRIBERR_GPU_ID:-3}"]
capabilities: [gpu]
healthcheck:
# 127.0.0.1 rather than localhost — the IPv6-first resolution trap has
@@ -121,7 +124,7 @@ services:
- homepage.group=AI - Studios
- homepage.name=Scriberr
- homepage.icon=mdi-microphone-message
- homepage.description=Audio/video transcription + diarization (fv-ml1, GPU1)
- homepage.description=Audio/video transcription + diarization (fv-ml1, GPU3)
- homepage.href=http://10.251.50.54:${SCRIBERR_PORT}
networks: