vh
aa5863c9a3
Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves
large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204,
fp16, both resident, ~5.9 GB VRAM against 20 GB still free.
Sits alongside the existing parakeet stack (:8765) deliberately: parakeet
is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob
concept, so it structurally cannot serve this consumer.
The load-bearing requirement -- segments[].no_speech_prob surviving
response_format=verbose_json -- is VERIFIED on both tiers.
Measured finding worth more than the deployment: no_speech_prob alone is a
WEAK hallucination gate on this stack. Pure silence and pink room tone both
produced the classic Whisper 'Thank you.' hallucination while no_speech_prob
stayed under 0.11 -- a conventional >0.6 threshold passes both through.
avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs
-0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556.
Consumers should gate on a composite, not no_speech_prob alone. Table in
the README.
VAD pinned OFF at the consumer's request (they VAD-gate upstream on the
capture edge). Consequence stated plainly in the README: with VAD off this
service will transcribe silence into text and is not defending itself.
Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off
setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly
marks unstable. A floating tag could rename it on any bump, silently
restoring VAD and moving no_speech_prob semantics under a calibrated gate
with no error and no log line.
Two deployment gotchas recorded: PRELOAD_MODELS only loads models already
cached (it does not download -- use POST /v1/models/{id}), and the bind-
mounted cache needs a hub/ subdir or every /v1/models call 500s with
CacheNotFound while /health still returns 200.
speaches — OpenAI-compatible ASR (faster-whisper) on irv-ml1
Deployed 2026-08-21 for Eyra (meeting recorder) at the eyra-dev request.
- Host/GPU: irv-ml1, RTX A6000 (
device_ids: ["1"]— see the GPU note below). - Port:
:8204→http://10.100.79.3:8204. WG/LAN-internal, no auth (fleet default, agreed with the consumer for v1). - Image:
ghcr.io/speaches-ai/speachespinned by digest — see "Why the digest pin". - Models:
Systran/faster-whisper-large-v3(batch tier) +Systran/faster-distil-whisper-large-v3(low-latency tier). fp16, ~4.2 GB on disk, ~5.9 GB VRAM with both resident.
curl -X POST http://10.100.79.3:8204/v1/audio/transcriptions \
-F "file=@clip.wav;type=audio/wav" \
-F "model=Systran/faster-distil-whisper-large-v3" \
-F "response_format=verbose_json"
Why this exists next to parakeet (:8765)
Two ASR services on one box is deliberate, not drift. Parakeet is a TDT/transducer model
returning a bare {"text": ...}; it has no no_speech_prob concept at all. Eyra's
hallucination gate keys on per-segment no_speech_prob, so parakeet structurally cannot serve
it. Parakeet remains the right pick for plain text-out transcription.
The load-bearing requirement
response_format=verbose_json must return segments[].no_speech_prob intact. Verified
2026-08-21 on both tiers — every segment carries no_speech_prob, avg_logprob,
compression_ratio, temperature, tokens, and word timings.
⚠ Measured: no_speech_prob alone is a WEAK hallucination gate
Verified against synthetic speech / silence / pink-noise room tone, VAD off, distil tier:
| input | no_speech_prob |
avg_logprob |
compression_ratio |
text returned |
|---|---|---|---|---|
| speech (6.6 s) | 0.018 | -0.114 | 1.141 | correct verbatim |
| pure silence | 0.107 | -0.650 | 0.556 | "Thank you." ← hallucinated |
| pink room tone | 0.077 | -0.724 | 0.556 | "Thank you." ← hallucinated |
Both non-speech inputs produced the classic Whisper "Thank you." hallucination, and
no_speech_prob stayed under 0.11 on both — a conventional > 0.6 threshold would pass
them straight through. avg_logprob separates the same cases ~6× more decisively
(< -0.5 catches both), and compression_ratio splits cleanly at 0.556 vs 1.141.
Recommendation to any consumer: gate on a composite, not no_speech_prob alone. The field
is present and directionally correct, but its dynamic range on this stack is too compressed to
carry a threshold by itself.
Why VAD is pinned OFF
_UNSTABLE_VAD_FILTER=False, set explicitly in .env, at the consumer's request. Eyra VAD-gates
upstream on the capture edge and sends only speech segments; a second VAD here would re-chunk the
audio and therefore change what no_speech_prob means per segment, underneath a gate calibrated
against this stack.
Consequence worth stating plainly: with VAD off, this service will happily transcribe silence into text (see the table above). Upstream VAD gating is what prevents that — the service is not defending itself.
Why the digest pin
_UNSTABLE_VAD_FILTER carries a leading underscore and the literal word "unstable" — upstream
reserves the right to rename it. On a floating :latest-cuda, a routine image bump could silently
drop that variable, restore VAD, and move no_speech_prob semantics under a calibrated consumer
gate with no error and no log line. Bumping the pin is a deliberate act that requires re-running
the fidelity + discrimination checks above.
Gotcha: PRELOAD_MODELS does not download
PRELOAD_MODELS only loads models already in the HF cache — it will not fetch them. A first
boot with an empty cache starts healthy, serves /health 200, and exposes an empty
/v1/models. Download explicitly (URL-encode the / in the repo id):
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-whisper-large-v3"
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-distil-whisper-large-v3"
Related: the bind-mounted cache dir must contain a hub/ subdirectory or every
/v1/models call 500s with huggingface_hub.errors.CacheNotFound while /health still returns
200 — a healthy container serving a broken registry. mkdir -p <cache>/hub owned by uid 1000.
GPU note — the docker/native index inversion
device_ids: ["1"] is the A6000, verified empirically (the container reports
NVIDIA RTX A6000). Do not "fix" this to 0 by reading nvidia-smi from a native shell:
native CUDA on irv-ml1 enumerates the A6000 as cuda:0 while the docker view has it at 1.
0 in compose is the RTX 3090, which already hosts parakeet and the TTS stacks.
Residency
STT_MODEL_TTL=-1 keeps both tiers resident (upstream default is 300 s). Deliberate: a
mid-meeting reload would be a multi-second stall on a latency-sensitive draft-caption tier, and
~5.9 GB against 20 GB still free is cheap. Revisit when Eyra's diarization workload lands on
this box — that ask is expected at their diarize milestone and will want GPU embeddings plus
gated pyannote weights. Switching to 300 trades the latency back for VRAM in one .env line.
Deploy
scripts/deploy-stack.sh irv-ml1 speaches # diffs vs live, prompts y/N
# on host: cp .env.example .env; docker compose up -d