vh
caa04801f3
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
58 lines
2.6 KiB
Bash
58 lines
2.6 KiB
Bash
# Parakeet ASR stack tunables. Copy to `.env` on fv-ml1 before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# # edit as needed
|
|
# docker compose build
|
|
# docker compose up -d
|
|
|
|
# Image tag. Bump when you change the Dockerfile / app.py so docker caches
|
|
# cleanly.
|
|
PARAKEET_TAG=sherpa-onnx-v4
|
|
|
|
# Which GPU to pin. The container sees whichever card this names as cuda:0.
|
|
#
|
|
# ⚠ GPU 0, deliberately, NOT the empty GPU 3. This seat is ~800 MiB and GPU 0 has
|
|
# the most uncommitted headroom of the three working cards (its seats commit
|
|
# util 0.88, ~13 GB spare; GPU 1 is at 0.975, GPU 2 at 0.96). Leaving GPU 3
|
|
# untouched keeps a full 96 GB card available for a real seat — vLLM sizes KV
|
|
# cache against TOTAL VRAM, so even a 1 GB tenant eats into a big seat's margin.
|
|
PARAKEET_GPU=0
|
|
|
|
# Host port for the FastAPI server (container listens on 8000). 8300 is
|
|
# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
|
|
PARAKEET_PORT=8300
|
|
|
|
# Bind address. 0.0.0.0 exposes on all interfaces. Use 127.0.0.1 to restrict
|
|
# to local-only — but LiteLLM on ana-docker reaches this over the LAN, so it
|
|
# has to be 0.0.0.0 for the gateway alias to work.
|
|
PARAKEET_BIND=0.0.0.0
|
|
|
|
# Host path for the ONNX model files — encoder/decoder/joiner/tokens.txt.
|
|
# Downloaded by the entrypoint on first run if absent. Must exist before
|
|
# first `up` (directory, not files). Regenerable — exclude from restic.
|
|
PARAKEET_MODELS_DIR=/tank/parakeet/models
|
|
|
|
# Which sherpa-onnx release tarball to fetch on first boot.
|
|
# v3 (default, 464 MiB) — 25 European languages
|
|
# v2 — English only, swap the URL below
|
|
# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
|
|
PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
|
|
|
|
# ONNX Runtime execution provider. `cuda` uses the GPU (requires the nvidia
|
|
# container runtime + matching CUDA/cuDNN in the image). `cpu` falls back to
|
|
# CPU.
|
|
#
|
|
# ⚠ ORT's CUDA EP FALLS BACK TO CPU SILENTLY when it cannot initialise — the
|
|
# server still answers 200 and still returns correct text, just slowly. So
|
|
# `PROVIDER=cuda` is a REQUEST, not a guarantee, and the only honest check is
|
|
# to watch `nvidia-smi` during a transcription and confirm a process appears on
|
|
# the pinned card. See README § Verifying the GPU is actually in use.
|
|
PARAKEET_PROVIDER=cuda
|
|
|
|
# CPU threads per recognizer session. Irrelevant when provider=cuda;
|
|
# only matters for provider=cpu.
|
|
PARAKEET_NUM_THREADS=1
|
|
|
|
# Log level: DEBUG | INFO | WARNING | ERROR
|
|
PARAKEET_LOG_LEVEL=INFO
|