vh
b9b14b5baf
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call.
54 lines
2.3 KiB
Bash
54 lines
2.3 KiB
Bash
# Parakeet ASR stack tunables. Copy to `.env` on fv-ml1 before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# # edit as needed
|
|
# docker compose build
|
|
# docker compose up -d
|
|
|
|
# Image tag. Bump when you change the Dockerfile / app.py so docker caches
|
|
# cleanly.
|
|
PARAKEET_TAG=sherpa-onnx-v4
|
|
|
|
# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM
|
|
# serving seats and sit at 85-98% VRAM, so this is the only one with room.
|
|
# The container sees whichever card this names as cuda:0 internally.
|
|
PARAKEET_GPU=3
|
|
|
|
# Host port for the FastAPI server (container listens on 8000). 8300 is
|
|
# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
|
|
PARAKEET_PORT=8300
|
|
|
|
# Bind address. 0.0.0.0 exposes on all interfaces. Use 127.0.0.1 to restrict
|
|
# to local-only — but LiteLLM on ana-docker reaches this over the LAN, so it
|
|
# has to be 0.0.0.0 for the gateway alias to work.
|
|
PARAKEET_BIND=0.0.0.0
|
|
|
|
# Host path for the ONNX model files — encoder/decoder/joiner/tokens.txt.
|
|
# Downloaded by the entrypoint on first run if absent. Must exist before
|
|
# first `up` (directory, not files). Regenerable — exclude from restic.
|
|
PARAKEET_MODELS_DIR=/tank/parakeet/models
|
|
|
|
# Which sherpa-onnx release tarball to fetch on first boot.
|
|
# v3 (default, 464 MiB) — 25 European languages
|
|
# v2 — English only, swap the URL below
|
|
# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
|
|
PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
|
|
|
|
# ONNX Runtime execution provider. `cuda` uses the GPU (requires the nvidia
|
|
# container runtime + matching CUDA/cuDNN in the image). `cpu` falls back to
|
|
# CPU.
|
|
#
|
|
# ⚠ ORT's CUDA EP FALLS BACK TO CPU SILENTLY when it cannot initialise — the
|
|
# server still answers 200 and still returns correct text, just slowly. So
|
|
# `PROVIDER=cuda` is a REQUEST, not a guarantee, and the only honest check is
|
|
# to watch `nvidia-smi` during a transcription and confirm a process appears on
|
|
# the pinned card. See README § Verifying the GPU is actually in use.
|
|
PARAKEET_PROVIDER=cuda
|
|
|
|
# CPU threads per recognizer session. Irrelevant when provider=cuda;
|
|
# only matters for provider=cpu.
|
|
PARAKEET_NUM_THREADS=1
|
|
|
|
# Log level: DEBUG | INFO | WARNING | ERROR
|
|
PARAKEET_LOG_LEVEL=INFO
|