feat(speaches): OpenAI-compatible faster-whisper ASR seat on irv-ml1 A6000

Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204, fp16, both resident, ~5.9 GB VRAM against 20 GB still free. Sits alongside the existing parakeet stack (:8765) deliberately: parakeet is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob concept, so it structurally cannot serve this consumer. The load-bearing requirement -- segments[].no_speech_prob surviving response_format=verbose_json -- is VERIFIED on both tiers. Measured finding worth more than the deployment: no_speech_prob alone is a WEAK hallucination gate on this stack. Pure silence and pink room tone both produced the classic Whisper 'Thank you.' hallucination while no_speech_prob stayed under 0.11 -- a conventional >0.6 threshold passes both through. avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs -0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556. Consumers should gate on a composite, not no_speech_prob alone. Table in the README. VAD pinned OFF at the consumer's request (they VAD-gate upstream on the capture edge). Consequence stated plainly in the README: with VAD off this service will transcribe silence into text and is not defending itself. Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly marks unstable. A floating tag could rename it on any bump, silently restoring VAD and moving no_speech_prob semantics under a calibrated gate with no error and no log line. Two deployment gotchas recorded: PRELOAD_MODELS only loads models already cached (it does not download -- use POST /v1/models/{id}), and the bind- mounted cache needs a hub/ subdir or every /v1/models call 500s with CacheNotFound while /health still returns 200.
2026-08-21 14:31:57 -07:00
parent ff5ce212da
commit aa5863c9a3
3 changed files with 232 additions and 0 deletions
@@ -0,0 +1,43 @@
+# speaches — irv-ml1. Copy to .env on the host and adjust.
+
+# ── Image ───────────────────────────────────────────────────────────────────
+# PINNED BY DIGEST, deliberately. The VAD-off setting below rides on an
+# upstream variable explicitly marked unstable (`_UNSTABLE_VAD_FILTER`); an
+# unpinned `:latest-cuda` could rename it on any bump, silently restoring VAD
+# and shifting `no_speech_prob` semantics under Eyra's calibrated gate.
+# Bumping this tag is a deliberate act that REQUIRES re-running the
+# verbose_json fidelity check afterwards. Digest captured 2026-08-21.
+SPEACHES_IMAGE=ghcr.io/speaches-ai/speaches@sha256:6ec12ebf890a17e0d4b242a8ba9e0eb1fb836e60e8a3c857aea9838d541579ac
+
+# ── Network ─────────────────────────────────────────────────────────────────
+SPEACHES_PORT=8204
+SPEACHES_BIND=0.0.0.0
+
+# ── GPU ─────────────────────────────────────────────────────────────────────
+# 1 = RTX A6000 (26 GB free). 0 = RTX 3090, already hosting parakeet + TTS.
+SPEACHES_GPU_ID=1
+
+# ── Models ──────────────────────────────────────────────────────────────────
+# Both tiers preloaded at startup so the first call of a meeting is not a
+# cold download. Repo IDs verified against the HF API 2026-08-21.
+#   large-v3        (~3.1 GB fp16) — batch/accuracy tier, post-meeting job
+#   distil-large-v3 (~1.5 GB fp16) — low-latency draft captioning
+# ⚠ distil-large-v3 is ENGLISH-ONLY. Eyra is English-only for v1 by design.
+#   When multilingual unparks, `Systran/faster-whisper-large-v3-turbo` is the
+#   multilingual replacement for the fast tier — swapping it is a .env edit.
+SPEACHES_PRELOAD_MODELS=["Systran/faster-whisper-large-v3","Systran/faster-distil-whisper-large-v3"]
+SPEACHES_CACHE_DIR=/worktank/speaches/cache
+
+# ── Inference ───────────────────────────────────────────────────────────────
+# VAD OFF — consumer requirement, do not flip without telling eyra-dev.
+SPEACHES_VAD_FILTER=False
+# Ampere sm_86: fp16 is correct here; no FP8/NVFP4 on this hardware.
+SPEACHES_COMPUTE_TYPE=float16
+SPEACHES_NUM_WORKERS=1
+# -1 = never unload (deterministic latency). 300 = upstream default, trades
+# latency for VRAM — switch if the A6000 gets tight.
+SPEACHES_STT_MODEL_TTL=-1
+
+# ── Misc ────────────────────────────────────────────────────────────────────
+SPEACHES_ENABLE_UI=true
+SPEACHES_LOG_LEVEL=info
@@ -0,0 +1,106 @@
+# speaches — OpenAI-compatible ASR (faster-whisper) on irv-ml1
+
+Deployed 2026-08-21 for **Eyra** (meeting recorder) at the `eyra-dev` request.
+
+- **Host/GPU:** irv-ml1, **RTX A6000** (`device_ids: ["1"]` — see the GPU note below).
+- **Port:** `:8204` → `http://10.100.79.3:8204`. WG/LAN-internal, **no auth** (fleet default,
+  agreed with the consumer for v1).
+- **Image:** `ghcr.io/speaches-ai/speaches` **pinned by digest** — see "Why the digest pin".
+- **Models:** `Systran/faster-whisper-large-v3` (batch tier) +
+  `Systran/faster-distil-whisper-large-v3` (low-latency tier). fp16, ~4.2 GB on disk,
+  ~5.9 GB VRAM with both resident.
+
+```bash
+curl -X POST http://10.100.79.3:8204/v1/audio/transcriptions \
+  -F "file=@clip.wav;type=audio/wav" \
+  -F "model=Systran/faster-distil-whisper-large-v3" \
+  -F "response_format=verbose_json"
+```
+
+## Why this exists next to `parakeet` (:8765)
+
+Two ASR services on one box is deliberate, not drift. Parakeet is a TDT/transducer model
+returning a bare `{"text": ...}`; **it has no `no_speech_prob` concept at all.** Eyra's
+hallucination gate keys on per-segment `no_speech_prob`, so parakeet structurally cannot serve
+it. Parakeet remains the right pick for plain text-out transcription.
+
+## The load-bearing requirement
+
+`response_format=verbose_json` must return `segments[].no_speech_prob` intact. **Verified
+2026-08-21** on both tiers — every segment carries `no_speech_prob`, `avg_logprob`,
+`compression_ratio`, `temperature`, `tokens`, and word timings.
+
+### ⚠ Measured: `no_speech_prob` alone is a WEAK hallucination gate
+
+Verified against synthetic speech / silence / pink-noise room tone, VAD off, distil tier:
+
+| input | `no_speech_prob` | `avg_logprob` | `compression_ratio` | text returned |
+|---|---|---|---|---|
+| speech (6.6 s) | **0.018** | **-0.114** | 1.141 | correct verbatim |
+| pure silence | **0.107** | **-0.650** | 0.556 | `"Thank you."` ← hallucinated |
+| pink room tone | **0.077** | **-0.724** | 0.556 | `"Thank you."` ← hallucinated |
+
+Both non-speech inputs produced the classic Whisper `"Thank you."` hallucination, and
+**`no_speech_prob` stayed under 0.11 on both** — a conventional `> 0.6` threshold would pass
+them straight through. `avg_logprob` separates the same cases ~6× more decisively
+(`< -0.5` catches both), and `compression_ratio` splits cleanly at 0.556 vs 1.141.
+
+**Recommendation to any consumer: gate on a composite, not `no_speech_prob` alone.** The field
+is present and directionally correct, but its dynamic range on this stack is too compressed to
+carry a threshold by itself.
+
+## Why VAD is pinned OFF
+
+`_UNSTABLE_VAD_FILTER=False`, set explicitly in `.env`, at the consumer's request. Eyra VAD-gates
+upstream on the capture edge and sends only speech segments; a second VAD here would re-chunk the
+audio and therefore change what `no_speech_prob` *means* per segment, underneath a gate calibrated
+against this stack.
+
+Consequence worth stating plainly: **with VAD off, this service will happily transcribe silence
+into text** (see the table above). Upstream VAD gating is what prevents that — the service is not
+defending itself.
+
+## Why the digest pin
+
+`_UNSTABLE_VAD_FILTER` carries a leading underscore and the literal word "unstable" — upstream
+reserves the right to rename it. On a floating `:latest-cuda`, a routine image bump could silently
+drop that variable, restore VAD, and move `no_speech_prob` semantics under a calibrated consumer
+gate with no error and no log line. **Bumping the pin is a deliberate act that requires re-running
+the fidelity + discrimination checks above.**
+
+## Gotcha: `PRELOAD_MODELS` does not download
+
+`PRELOAD_MODELS` only *loads models already in the HF cache* — it will not fetch them. A first
+boot with an empty cache starts healthy, serves `/health` 200, and exposes an **empty**
+`/v1/models`. Download explicitly (URL-encode the `/` in the repo id):
+
+```bash
+curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-whisper-large-v3"
+curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-distil-whisper-large-v3"
+```
+
+Related: the bind-mounted cache dir must contain a `hub/` subdirectory or **every**
+`/v1/models` call 500s with `huggingface_hub.errors.CacheNotFound` while `/health` still returns
+200 — a healthy container serving a broken registry. `mkdir -p <cache>/hub` owned by uid 1000.
+
+## GPU note — the docker/native index inversion
+
+`device_ids: ["1"]` is the **A6000**, verified empirically (the container reports
+`NVIDIA RTX A6000`). Do not "fix" this to `0` by reading `nvidia-smi` from a native shell:
+native CUDA on irv-ml1 enumerates the A6000 as `cuda:0` while the docker view has it at `1`.
+`0` in compose is the RTX 3090, which already hosts parakeet and the TTS stacks.
+
+## Residency
+
+`STT_MODEL_TTL=-1` keeps both tiers resident (upstream default is 300 s). Deliberate: a
+mid-meeting reload would be a multi-second stall on a latency-sensitive draft-caption tier, and
+~5.9 GB against 20 GB still free is cheap. **Revisit when Eyra's diarization workload lands on
+this box** — that ask is expected at their diarize milestone and will want GPU embeddings plus
+gated pyannote weights. Switching to `300` trades the latency back for VRAM in one `.env` line.
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh irv-ml1 speaches     # diffs vs live, prompts y/N
+# on host: cp .env.example .env; docker compose up -d
+```
@@ -0,0 +1,83 @@
+# speaches — OpenAI-compatible ASR (faster-whisper / CTranslate2) on irv-ml1.
+#
+# Consumer: Eyra (meeting recorder). Its hallucination gate keys on the
+# per-segment `no_speech_prob` field returned by `response_format=verbose_json`,
+# so that field surviving the API boundary is LOAD-BEARING, not cosmetic.
+#
+# API:
+#   POST /v1/audio/transcriptions   — multipart; response_format=verbose_json
+#   GET  /v1/models                 — registry of available STT models
+#   GET  /health
+#
+# Coexists with the `parakeet` stack (:8765). Parakeet is a TDT/transducer model
+# returning a bare {"text": ...} — it has no `no_speech_prob` concept at all, so
+# it cannot serve this consumer. Two ASR services on one box is deliberate.
+#
+# All tunables live in .env — edit that, not this file.
+
+services:
+  speaches:
+    image: ${SPEACHES_IMAGE}
+    container_name: speaches
+    restart: unless-stopped
+    ports:
+      - "${SPEACHES_BIND:-0.0.0.0}:${SPEACHES_PORT}:8000"
+    environment:
+      - UVICORN_HOST=0.0.0.0
+      - UVICORN_PORT=8000
+      - LOG_LEVEL=${SPEACHES_LOG_LEVEL:-info}
+      # ── VAD: PINNED OFF at the consumer's explicit request ──────────────
+      # Eyra VAD-gates upstream on the capture edge and sends only speech
+      # segments. A second VAD here would re-chunk the audio and therefore
+      # shift what `no_speech_prob` MEANS per segment, underneath a gate
+      # calibrated against this stack.
+      # ⚠ The upstream default is True, and the variable name is explicitly
+      # marked unstable (leading underscore). That is exactly why
+      # SPEACHES_IMAGE is pinned to a digest below — an unpinned bump could
+      # silently rename this var, restoring VAD and moving the gate.
+      - _UNSTABLE_VAD_FILTER=${SPEACHES_VAD_FILTER:-False}
+      # ── Whisper / CTranslate2 ───────────────────────────────────────────
+      # float16: irv-ml1 is Ampere (sm_86) — no native FP8/NVFP4, fp16 is the
+      # correct compute type here. device_index 0 is the index INSIDE the
+      # container, which the device_ids pin below maps to the A6000.
+      - WHISPER__INFERENCE_DEVICE=cuda
+      - WHISPER__DEVICE_INDEX=0
+      - WHISPER__COMPUTE_TYPE=${SPEACHES_COMPUTE_TYPE:-float16}
+      - WHISPER__NUM_WORKERS=${SPEACHES_NUM_WORKERS:-1}
+      # ── Residency ───────────────────────────────────────────────────────
+      # -1 = never unload. Deliberate: a mid-meeting model reload would be a
+      # multi-second stall on a latency-sensitive draft-caption tier. Both
+      # tiers resident is ~4.6 GB against 26 GB free on the A6000. Set to 300
+      # (the upstream default) to trade that latency back for VRAM if the card
+      # gets tight — e.g. when Eyra's diarization workload lands here.
+      - STT_MODEL_TTL=${SPEACHES_STT_MODEL_TTL:--1}
+      - ENABLE_UI=${SPEACHES_ENABLE_UI:-true}
+      - PRELOAD_MODELS=${SPEACHES_PRELOAD_MODELS}
+      - HF_HOME=/home/ubuntu/.cache/huggingface
+    volumes:
+      # Model cache persists across recreates so a bounce is not a re-download.
+      - ${SPEACHES_CACHE_DIR}:/home/ubuntu/.cache/huggingface
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              # "1" = RTX A6000 (verified empirically — nvidia-smi index 1;
+              # note native CUDA on this host enumerates the A6000 as 0, the
+              # docker view is what matters here). "0" would be the 3090,
+              # which already hosts parakeet + the TTS stacks.
+              device_ids: ["${SPEACHES_GPU_ID:-1}"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://localhost:8000/health || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      # First boot downloads ~4.6 GB of CT2 weights for the two preloaded tiers.
+      start_period: 600s
+    labels:
+      - homepage.group=AI - Audio Tools
+      - homepage.name=Speaches ASR
+      - homepage.icon=mdi-text-to-speech
+      - homepage.description=OpenAI-compatible faster-whisper STT, verbose_json (irv-ml1)
+      - homepage.href=http://10.100.79.3:${SPEACHES_PORT}