fix(parakeet): move the seat from the empty GPU 3 to GPU 0
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
This commit is contained in:
@@ -9,10 +9,14 @@
|
||||
# cleanly.
|
||||
PARAKEET_TAG=sherpa-onnx-v4
|
||||
|
||||
# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM
|
||||
# serving seats and sit at 85-98% VRAM, so this is the only one with room.
|
||||
# The container sees whichever card this names as cuda:0 internally.
|
||||
PARAKEET_GPU=3
|
||||
# Which GPU to pin. The container sees whichever card this names as cuda:0.
|
||||
#
|
||||
# ⚠ GPU 0, deliberately, NOT the empty GPU 3. This seat is ~800 MiB and GPU 0 has
|
||||
# the most uncommitted headroom of the three working cards (its seats commit
|
||||
# util 0.88, ~13 GB spare; GPU 1 is at 0.975, GPU 2 at 0.96). Leaving GPU 3
|
||||
# untouched keeps a full 96 GB card available for a real seat — vLLM sizes KV
|
||||
# cache against TOTAL VRAM, so even a 1 GB tenant eats into a big seat's margin.
|
||||
PARAKEET_GPU=0
|
||||
|
||||
# Host port for the FastAPI server (container listens on 8000). 8300 is
|
||||
# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
|
||||
|
||||
+26
-14
@@ -6,24 +6,35 @@ wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
|
||||
|
||||
**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
|
||||
**Port:** 8300 (container 8000)
|
||||
**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
|
||||
**GPU:** **0**, pinned explicitly via `device_ids` — shares the card with two vLLM seats
|
||||
**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
|
||||
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
|
||||
**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
|
||||
|
||||
## Why GPU 3
|
||||
## Why GPU 0 and not the empty card
|
||||
|
||||
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
|
||||
vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
|
||||
empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
|
||||
next to those seats, but it still has to go somewhere that is not fighting
|
||||
them for VRAM.
|
||||
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). This seat holds
|
||||
**~800 MiB — under 1 %** of one card, so the question is not "where does it
|
||||
fit" but "whose headroom can it spend most cheaply".
|
||||
|
||||
⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
|
||||
the fleet convention — **not** `count: all`, which is what the dead on-host
|
||||
stub used and which would have handed this seat all four cards. Inside the
|
||||
container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
|
||||
CUDA execution provider takes by default.
|
||||
| GPU | committed `--gpu-memory-utilization` | spare |
|
||||
|---|---|---|
|
||||
| **0** | 0.40 + 0.48 = **0.88** | ~13 GB ← here |
|
||||
| 1 | 0.52+0.24+0.10+0.055+0.03+0.03 = **0.975** | ~4.3 GB |
|
||||
| 2 | **0.96** | ~1.8 GB |
|
||||
| 3 | — | *kept empty* |
|
||||
|
||||
It was first placed on the empty GPU 3, which was wrong: **vLLM sizes its KV
|
||||
cache as a fraction of TOTAL VRAM, not free VRAM**, so any resident tenant on an
|
||||
otherwise-clean card eats directly into the profiling margin of whatever big seat
|
||||
lands there later — and `flash-next` needs 93 GiB of 96. A 96 GB card at 2 MiB is
|
||||
worth far more than 800 MiB of it. Moved to GPU 0 the same night.
|
||||
|
||||
⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
|
||||
convention — **not** `count: all`, which is what the dead on-host stub used and
|
||||
which would have handed this seat all four cards. Inside the container the pinned
|
||||
card presents as `cuda:0`, which is what sherpa-onnx's CUDA execution provider
|
||||
takes by default.
|
||||
|
||||
## Why not the FastAPI community wrappers
|
||||
|
||||
@@ -70,13 +81,14 @@ The honest check is to watch the card while a transcription runs:
|
||||
```bash
|
||||
# on fv-ml1 — terminal 1
|
||||
watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
|
||||
--format=csv -i 3'
|
||||
--format=csv -i 0'
|
||||
|
||||
# terminal 2 — send real audio, not silence
|
||||
curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
|
||||
```
|
||||
|
||||
A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
|
||||
A process must appear **on GPU 0** for the duration — ~800 MiB alongside the two
|
||||
much larger `VLLM::EngineCore` entries. If it never appears, the
|
||||
CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
|
||||
Confirm with the container's own startup log, which names the providers ORT
|
||||
actually registered:
|
||||
|
||||
@@ -6,16 +6,23 @@
|
||||
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
|
||||
# wrapper we own end-to-end.
|
||||
#
|
||||
# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
|
||||
# card — the other three carry the vLLM serving seats and run 85-98% full, so a
|
||||
# seat placed anywhere else would fight them for VRAM.
|
||||
# HOST: fv-ml1, GPU 0 (relocated from irv-ml1 2026-09-15).
|
||||
#
|
||||
# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
|
||||
# ⚠ GPU 0, NOT the empty GPU 3. This seat holds ~800 MiB — under 1% of a 96 GB
|
||||
# card — so it rides on the card with the most uncommitted headroom rather than
|
||||
# putting the first fingerprint on a pristine one. GPU 0's seats commit
|
||||
# util 0.40 + 0.48 = 0.88, leaving ~13 GB; GPU 1 is at 0.975 and GPU 2 at 0.96,
|
||||
# both too tight. Keeping GPU 3 at 2 MiB means it can still take a full-card seat
|
||||
# (flash-next needs 93 GiB) without a neighbour eating its profiling margin —
|
||||
# vLLM sizes KV cache as a fraction of TOTAL VRAM, not free VRAM, so a resident
|
||||
# tenant on an otherwise-empty card is worth more than its megabytes suggest.
|
||||
#
|
||||
# ⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
|
||||
# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
|
||||
# `count: all` (which is what the dead on-host stub did, and would have let this
|
||||
# tiny ASR seat see all four cards including the three that are full).
|
||||
# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
|
||||
# what sherpa-onnx's CUDAExecutionProvider takes by default.
|
||||
# tiny ASR seat see all four cards). device_ids presents the pinned card as
|
||||
# cuda:0 INSIDE the container, which is what sherpa-onnx's
|
||||
# CUDAExecutionProvider takes by default.
|
||||
#
|
||||
# Model weights (~460 MB int8) download on first run via the entrypoint to
|
||||
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
|
||||
@@ -51,7 +58,7 @@ services:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["${PARAKEET_GPU:-3}"]
|
||||
device_ids: ["${PARAKEET_GPU:-0}"]
|
||||
capabilities: [gpu]
|
||||
networks:
|
||||
- tnet
|
||||
@@ -67,7 +74,7 @@ services:
|
||||
- homepage.group=AI - Audio Tools
|
||||
- homepage.name=Parakeet ASR
|
||||
- homepage.icon=mdi-microphone
|
||||
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
|
||||
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 0)
|
||||
- homepage.href=http://10.251.50.54:${PARAKEET_PORT}
|
||||
|
||||
networks:
|
||||
|
||||
Reference in New Issue
Block a user