vh
caa04801f3
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
84 lines
3.3 KiB
YAML
84 lines
3.3 KiB
YAML
# Parakeet ASR via sherpa-onnx + our own thin FastAPI wrapper.
|
|
#
|
|
# We previously wrapped Shadowfita/parakeet-tdt-0.6b-v2-fastapi but hit two
|
|
# unfixed upstream bugs (open issues #16 + #10) the first time we actually sent
|
|
# a transcription request. Switched to sherpa-onnx — ONNX Runtime + CUDA, a
|
|
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
|
|
# wrapper we own end-to-end.
|
|
#
|
|
# HOST: fv-ml1, GPU 0 (relocated from irv-ml1 2026-09-15).
|
|
#
|
|
# ⚠ GPU 0, NOT the empty GPU 3. This seat holds ~800 MiB — under 1% of a 96 GB
|
|
# card — so it rides on the card with the most uncommitted headroom rather than
|
|
# putting the first fingerprint on a pristine one. GPU 0's seats commit
|
|
# util 0.40 + 0.48 = 0.88, leaving ~13 GB; GPU 1 is at 0.975 and GPU 2 at 0.96,
|
|
# both too tight. Keeping GPU 3 at 2 MiB means it can still take a full-card seat
|
|
# (flash-next needs 93 GiB) without a neighbour eating its profiling margin —
|
|
# vLLM sizes KV cache as a fraction of TOTAL VRAM, not free VRAM, so a resident
|
|
# tenant on an otherwise-empty card is worth more than its megabytes suggest.
|
|
#
|
|
# ⚠ The pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
|
|
# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
|
|
# `count: all` (which is what the dead on-host stub did, and would have let this
|
|
# tiny ASR seat see all four cards). device_ids presents the pinned card as
|
|
# cuda:0 INSIDE the container, which is what sherpa-onnx's
|
|
# CUDAExecutionProvider takes by default.
|
|
#
|
|
# Model weights (~460 MB int8) download on first run via the entrypoint to
|
|
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
|
|
# the download.
|
|
#
|
|
# API:
|
|
# POST /transcribe — multipart file upload, returns {"text": "..."}
|
|
# POST /v1/audio/transcriptions — same body, OpenAI-compatible path alias
|
|
# GET /healthz
|
|
#
|
|
# All tunables live in .env — edit that, not this file.
|
|
|
|
services:
|
|
parakeet:
|
|
image: local/parakeet:${PARAKEET_TAG}
|
|
build:
|
|
context: .
|
|
dockerfile: Dockerfile
|
|
container_name: parakeet
|
|
restart: unless-stopped
|
|
ports:
|
|
- "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
|
|
environment:
|
|
- MODEL_DIR=/models
|
|
- MODEL_URL=${PARAKEET_MODEL_URL}
|
|
- PROVIDER=${PARAKEET_PROVIDER:-cuda}
|
|
- NUM_THREADS=${PARAKEET_NUM_THREADS:-1}
|
|
- LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
|
|
volumes:
|
|
- ${PARAKEET_MODELS_DIR}:/models
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids: ["${PARAKEET_GPU:-0}"]
|
|
capabilities: [gpu]
|
|
networks:
|
|
- tnet
|
|
healthcheck:
|
|
# Image ships wget (apt) but not curl — use wget so the check actually runs.
|
|
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
# First boot may include a ~460 MB model download.
|
|
start_period: 300s
|
|
labels:
|
|
- homepage.group=AI - Audio Tools
|
|
- homepage.name=Parakeet ASR
|
|
- homepage.icon=mdi-microphone
|
|
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 0)
|
|
- homepage.href=http://10.251.50.54:${PARAKEET_PORT}
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|