vh
b9b14b5baf
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call.
77 lines
2.8 KiB
YAML
77 lines
2.8 KiB
YAML
# Parakeet ASR via sherpa-onnx + our own thin FastAPI wrapper.
|
|
#
|
|
# We previously wrapped Shadowfita/parakeet-tdt-0.6b-v2-fastapi but hit two
|
|
# unfixed upstream bugs (open issues #16 + #10) the first time we actually sent
|
|
# a transcription request. Switched to sherpa-onnx — ONNX Runtime + CUDA, a
|
|
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
|
|
# wrapper we own end-to-end.
|
|
#
|
|
# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
|
|
# card — the other three carry the vLLM serving seats and run 85-98% full, so a
|
|
# seat placed anywhere else would fight them for VRAM.
|
|
#
|
|
# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
|
|
# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
|
|
# `count: all` (which is what the dead on-host stub did, and would have let this
|
|
# tiny ASR seat see all four cards including the three that are full).
|
|
# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
|
|
# what sherpa-onnx's CUDAExecutionProvider takes by default.
|
|
#
|
|
# Model weights (~460 MB int8) download on first run via the entrypoint to
|
|
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
|
|
# the download.
|
|
#
|
|
# API:
|
|
# POST /transcribe — multipart file upload, returns {"text": "..."}
|
|
# POST /v1/audio/transcriptions — same body, OpenAI-compatible path alias
|
|
# GET /healthz
|
|
#
|
|
# All tunables live in .env — edit that, not this file.
|
|
|
|
services:
|
|
parakeet:
|
|
image: local/parakeet:${PARAKEET_TAG}
|
|
build:
|
|
context: .
|
|
dockerfile: Dockerfile
|
|
container_name: parakeet
|
|
restart: unless-stopped
|
|
ports:
|
|
- "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
|
|
environment:
|
|
- MODEL_DIR=/models
|
|
- MODEL_URL=${PARAKEET_MODEL_URL}
|
|
- PROVIDER=${PARAKEET_PROVIDER:-cuda}
|
|
- NUM_THREADS=${PARAKEET_NUM_THREADS:-1}
|
|
- LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
|
|
volumes:
|
|
- ${PARAKEET_MODELS_DIR}:/models
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
device_ids: ["${PARAKEET_GPU:-3}"]
|
|
capabilities: [gpu]
|
|
networks:
|
|
- tnet
|
|
healthcheck:
|
|
# Image ships wget (apt) but not curl — use wget so the check actually runs.
|
|
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
# First boot may include a ~460 MB model download.
|
|
start_period: 300s
|
|
labels:
|
|
- homepage.group=AI - Audio Tools
|
|
- homepage.name=Parakeet ASR
|
|
- homepage.icon=mdi-microphone
|
|
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
|
|
- homepage.href=http://10.251.50.54:${PARAKEET_PORT}
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|