feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1

Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.

Changes:

- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
  `count: all`, which would have handed a 0.6B ASR seat all four cards);
  join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
  rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.

The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.

Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.

Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.

The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.

Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.
This commit is contained in:
vh
2026-09-15 01:41:41 -07:00
parent c6b6435c52
commit b9b14b5baf
8 changed files with 366 additions and 93 deletions
+29 -6
View File
@@ -6,7 +6,18 @@
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
# wrapper we own end-to-end.
#
# Model weights (~400 MB int8) download on first run via the entrypoint to
# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
# card — the other three carry the vLLM serving seats and run 85-98% full, so a
# seat placed anywhere else would fight them for VRAM.
#
# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
# `count: all` (which is what the dead on-host stub did, and would have let this
# tiny ASR seat see all four cards including the three that are full).
# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
# what sherpa-onnx's CUDAExecutionProvider takes by default.
#
# Model weights (~460 MB int8) download on first run via the entrypoint to
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
# the download.
#
@@ -25,11 +36,9 @@ services:
dockerfile: Dockerfile
container_name: parakeet
restart: unless-stopped
runtime: nvidia
ports:
- "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
environment:
- NVIDIA_VISIBLE_DEVICES=0
- MODEL_DIR=/models
- MODEL_URL=${PARAKEET_MODEL_URL}
- PROVIDER=${PARAKEET_PROVIDER:-cuda}
@@ -37,17 +46,31 @@ services:
- LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
volumes:
- ${PARAKEET_MODELS_DIR}:/models
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["${PARAKEET_GPU:-3}"]
capabilities: [gpu]
networks:
- tnet
healthcheck:
# Image ships wget (apt) but not curl — use wget so the check actually runs.
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"]
interval: 30s
timeout: 10s
retries: 3
# First boot may include a ~400 MB model download.
# First boot may include a ~460 MB model download.
start_period: 300s
labels:
- homepage.group=AI - Audio Tools
- homepage.name=Parakeet ASR
- homepage.icon=mdi-microphone
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (irv-ml1)
- homepage.href=http://irv-ml1.nh3.internal:${PARAKEET_PORT}
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
- homepage.href=http://10.251.50.54:${PARAKEET_PORT}
networks:
tnet:
name: traefik-net
external: true