vh
aa5863c9a3
Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves
large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204,
fp16, both resident, ~5.9 GB VRAM against 20 GB still free.
Sits alongside the existing parakeet stack (:8765) deliberately: parakeet
is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob
concept, so it structurally cannot serve this consumer.
The load-bearing requirement -- segments[].no_speech_prob surviving
response_format=verbose_json -- is VERIFIED on both tiers.
Measured finding worth more than the deployment: no_speech_prob alone is a
WEAK hallucination gate on this stack. Pure silence and pink room tone both
produced the classic Whisper 'Thank you.' hallucination while no_speech_prob
stayed under 0.11 -- a conventional >0.6 threshold passes both through.
avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs
-0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556.
Consumers should gate on a composite, not no_speech_prob alone. Table in
the README.
VAD pinned OFF at the consumer's request (they VAD-gate upstream on the
capture edge). Consequence stated plainly in the README: with VAD off this
service will transcribe silence into text and is not defending itself.
Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off
setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly
marks unstable. A floating tag could rename it on any bump, silently
restoring VAD and moving no_speech_prob semantics under a calibrated gate
with no error and no log line.
Two deployment gotchas recorded: PRELOAD_MODELS only loads models already
cached (it does not download -- use POST /v1/models/{id}), and the bind-
mounted cache needs a hub/ subdir or every /v1/models call 500s with
CacheNotFound while /health still returns 200.
84 lines
4.2 KiB
YAML
84 lines
4.2 KiB
YAML
# speaches — OpenAI-compatible ASR (faster-whisper / CTranslate2) on irv-ml1.
|
|
#
|
|
# Consumer: Eyra (meeting recorder). Its hallucination gate keys on the
|
|
# per-segment `no_speech_prob` field returned by `response_format=verbose_json`,
|
|
# so that field surviving the API boundary is LOAD-BEARING, not cosmetic.
|
|
#
|
|
# API:
|
|
# POST /v1/audio/transcriptions — multipart; response_format=verbose_json
|
|
# GET /v1/models — registry of available STT models
|
|
# GET /health
|
|
#
|
|
# Coexists with the `parakeet` stack (:8765). Parakeet is a TDT/transducer model
|
|
# returning a bare {"text": ...} — it has no `no_speech_prob` concept at all, so
|
|
# it cannot serve this consumer. Two ASR services on one box is deliberate.
|
|
#
|
|
# All tunables live in .env — edit that, not this file.
|
|
|
|
services:
|
|
speaches:
|
|
image: ${SPEACHES_IMAGE}
|
|
container_name: speaches
|
|
restart: unless-stopped
|
|
ports:
|
|
- "${SPEACHES_BIND:-0.0.0.0}:${SPEACHES_PORT}:8000"
|
|
environment:
|
|
- UVICORN_HOST=0.0.0.0
|
|
- UVICORN_PORT=8000
|
|
- LOG_LEVEL=${SPEACHES_LOG_LEVEL:-info}
|
|
# ── VAD: PINNED OFF at the consumer's explicit request ──────────────
|
|
# Eyra VAD-gates upstream on the capture edge and sends only speech
|
|
# segments. A second VAD here would re-chunk the audio and therefore
|
|
# shift what `no_speech_prob` MEANS per segment, underneath a gate
|
|
# calibrated against this stack.
|
|
# ⚠ The upstream default is True, and the variable name is explicitly
|
|
# marked unstable (leading underscore). That is exactly why
|
|
# SPEACHES_IMAGE is pinned to a digest below — an unpinned bump could
|
|
# silently rename this var, restoring VAD and moving the gate.
|
|
- _UNSTABLE_VAD_FILTER=${SPEACHES_VAD_FILTER:-False}
|
|
# ── Whisper / CTranslate2 ───────────────────────────────────────────
|
|
# float16: irv-ml1 is Ampere (sm_86) — no native FP8/NVFP4, fp16 is the
|
|
# correct compute type here. device_index 0 is the index INSIDE the
|
|
# container, which the device_ids pin below maps to the A6000.
|
|
- WHISPER__INFERENCE_DEVICE=cuda
|
|
- WHISPER__DEVICE_INDEX=0
|
|
- WHISPER__COMPUTE_TYPE=${SPEACHES_COMPUTE_TYPE:-float16}
|
|
- WHISPER__NUM_WORKERS=${SPEACHES_NUM_WORKERS:-1}
|
|
# ── Residency ───────────────────────────────────────────────────────
|
|
# -1 = never unload. Deliberate: a mid-meeting model reload would be a
|
|
# multi-second stall on a latency-sensitive draft-caption tier. Both
|
|
# tiers resident is ~4.6 GB against 26 GB free on the A6000. Set to 300
|
|
# (the upstream default) to trade that latency back for VRAM if the card
|
|
# gets tight — e.g. when Eyra's diarization workload lands here.
|
|
- STT_MODEL_TTL=${SPEACHES_STT_MODEL_TTL:--1}
|
|
- ENABLE_UI=${SPEACHES_ENABLE_UI:-true}
|
|
- PRELOAD_MODELS=${SPEACHES_PRELOAD_MODELS}
|
|
- HF_HOME=/home/ubuntu/.cache/huggingface
|
|
volumes:
|
|
# Model cache persists across recreates so a bounce is not a re-download.
|
|
- ${SPEACHES_CACHE_DIR}:/home/ubuntu/.cache/huggingface
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
# "1" = RTX A6000 (verified empirically — nvidia-smi index 1;
|
|
# note native CUDA on this host enumerates the A6000 as 0, the
|
|
# docker view is what matters here). "0" would be the 3090,
|
|
# which already hosts parakeet + the TTS stacks.
|
|
device_ids: ["${SPEACHES_GPU_ID:-1}"]
|
|
capabilities: [gpu]
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "curl -fsS http://localhost:8000/health || exit 1"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
# First boot downloads ~4.6 GB of CT2 weights for the two preloaded tiers.
|
|
start_period: 600s
|
|
labels:
|
|
- homepage.group=AI - Audio Tools
|
|
- homepage.name=Speaches ASR
|
|
- homepage.icon=mdi-text-to-speech
|
|
- homepage.description=OpenAI-compatible faster-whisper STT, verbose_json (irv-ml1)
|
|
- homepage.href=http://10.100.79.3:${SPEACHES_PORT}
|