Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.
Changes:
- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
`count: all`, which would have handed a 0.6B ASR seat all four cards);
join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.
The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.
Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.
Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.
The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.
Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.
The Shadowfita FastAPI wrapper hit two unfixed upstream bugs on the
first real /transcribe call — chunker return-shape mismatch (open
issue #16) and a `torchaudio.tensor` that doesn't exist (open #10).
Rather than babysit someone else's half-tested code, switched to
sherpa-onnx with the prebuilt int8 Parakeet-TDT tarball from k2-fsa,
and wrote our own ~60-line FastAPI wrapper.
Moving parts now owned in-tree:
Dockerfile CUDA 12.8 + cuDNN 9 runtime base, installs
sherpa-onnx==1.12.39+cuda12.cudnn9 + fastapi +
soundfile + libasound2 (sherpa-onnx links to ALSA
at load time even when we never touch a mic).
app.py OfflineRecognizer.from_transducer() once at startup;
/transcribe and /v1/audio/transcriptions both accept
multipart uploads and return {"text": ...}.
entrypoint.sh Idempotent model download to /models on first run
(~400 MB int8 tarball), then exec uvicorn.
Smoke test: 0.wav (bundled in the tarball, The House of the Seven
Gables excerpt) transcribes cleanly in ~1.2s on GPU.
PARAKEET_MODEL_URL in .env lets you swap to the v3 (25-language)
tarball without touching any other files. Wipe *.onnx + tokens.txt
from the models dir and the entrypoint re-downloads.
vh